How does a disruptive event change emergent clusters?
By: Aaron Dharna
Tags: Final Report, Project

If one has trained algorithmic models deployed, and if one knows that a disruptive event, E, is coming, it raises the question of how will the disruptive event require the deployed models to change? Similarly, if a system’s current formulation (e.g. parameters in a recommendation engine) has allowed for emergent communities to form that are dependent on said system, how does an emergent community respond to fundamental changes in their environment? To study this question in a data-rich environment, this work pulls data from the Massively Multiplayer Online First Person Shooter videogame Destiny 2, examines how players are interacting with the game via building player-weapon graphs, and applies clustering and community detection techniques on said player-weapon usage graphs to determine playstyle clusters. The key idea of this work is that if the player-base plays the game in methods A, B, and C, then that should be reflected in the data people have generated. Using this insight and clustering techniques to group similar players, does a change in how the game itself functions change how people are interacting with the game, therefore, resulting in new e.g. playstyle clusters? Does the playerbase engage with the game differently after the game was changed?

Studying the impact of large-scale shocks onto dependent trained models is very specific and requires not only a dataset that spans a time of great change but also to have models that are trained before the shock as well as after. Furthermore, many modern learning algorithms require many samples of data to extrapolate a latent pattern. So now we not only need data before and after a large-scale shock, but we need lots of data both before and after the shock. The stacking of each of these requirements means that there are not many publicly available datasets that meet these criteria. Therefore, part of the contribution of this work is the creation of one such dataset.

How did Destiny 2 change in Beyond Light?

As I note here:

On November 10th, 2020 Bungie launched an expansion for their game Destiny 2 entitled Beyond Light. In addition to new story content, the large yearly expansions often come with dramatic tuning of the game world via changes to the weapon systems, abilities, and/or new game modes. Beyond Light follows that trend by not only introducing a new element — Stasis — into Destiny (a first for the game since it launched in 2012), but also making drastic changes to the weapon system.

These weapon changes include removing an entire weapon subclass (a first for the game), changing damage amounts, slowing down handling animations (e.g. how long it takes to switch guns, which massively changes how one can play the game), and making a vast majority of the game’s prior loot unuseable in high-level activities. In addition to all of this, Bungie introduced new abilities into the game that also change how players optimally dual other players (e.g. running at me with a shotgun is now much more dangerous for the shotgunner since the game introduced freezing that can stop aggression easily). All of this taken together results in a game that, at least anecdotally, plays very differently than it did before November 10th.

Related Work

When machine learning models are trained on e.g. MNIST data to recognize digits, they implicitly assume that all digits the model might see in the future come from the same distribution as what is available in MNIST. However, that assumption is easily violated outside of laboratory conditions. Specifically, observing changes in one’s environment is commonplace in the technology industry. For example, any time that Google wants to make a change to their search engine, they will need to make sure that they do not break a dependence on e.g. GPT-3. This degraded ability due to a change comes about because the i.i.d. [3] requirement most trained models rely on has been violated. In other words, if you train on a distribution of data, you expect to also test as well as operate on that distribution. Therefore, if a model sees out-of-distribution data, it should not be surprising that the algorithm fails to operate as previously trained.

This is a necessary question from the perspective of researchers studying e.g. radicalization in the age of the internet. If researchers have their data and analysis pipelines set up to track and capture YouTube data [2] and YouTube changes their underlying recommendation engine, months of research studying how people get indoctrinated (which is a form of community formation, albeit a dark one) in the age of the internet can be invalidated.

Another example of research around large-scale systemic shocks comes from researchers Sakaki, Okazaki, and Matsuo (2010). They built a system to detect the presence of earthquakes in Japan (as earthquakes are unpredictable events) by determining and isolating (i.e. finding the epicenter of) disruptive changes in Twitter. At a broad level, this work can be abstracted to have studied the question of “what happens to our system (e.g. Twitter) in the aftermath of a disruptive event (an event that changes how people use the platform), and can we algorithmically determine what type of event this disruption was (e.g. a disaster, an election, the shutdown of the Twitter itself, etc.) and where it originated?”

Data

The data for this project was pulled via PyDest (think Tweepy for Destiny 2) from open-facing Bungie servers using Snowball sampling (start with an initial subject and have them point you towards other eligible people). Specifically, I scraped data from the Player vs Player competitive arena of the game. For the pre Beyond Light expansion, creating a large dataset was simple. I simply extracted the 250 most-recent competitive PvP match results — Post Game Carnage Report (PGCRs) — for each unique person I came across (excluding duplicate matches). As the player population is millions of players, this sampling method could have continued virtually indefinitely, but I stopped after 22617 players. Naturally, this sampling method is biased, and due to its “word-of-mouth” feature, this method is not a random sample of the population. Furthermore, this sampling is reflective of Bungie’s match-making (aka player recommendation) algorithm they use for building up matches where players get selected for their respective teams one by one.

Post Game Carnage Report from a competitive Destiny 2 match. This (player-facing) report shows basic information such as the number of kills someone got while their team held a zone advantage. When this same PGCR is pulled from Bungie’s servers, it contains much more detail than shown here.

I saved this data into a large JSON/dictionary object indexed by matchIDs and playerIDs. This information can also be flattened into a CSV form to facilitate many different types of research questions (e.g. skill prediction, matchmaking, engagement, measuring team cohesion and its effect on play, etc). From there, I reformatted the data to reflect the underlying graph this work would go on to analyze. See more here.

[That d]ifferent classes and weapon types exist in Destiny implies that there exists different ways of playing the game. Playing the game with shotguns results in vastly different play patterns than using a sniper rifle. Therefore, we’re going to try to tease [latent playstyle structure] out of the graph.

The underlying graph was characterized by the interaction of players with different weapon types where edges are increasingly weighted according to how much a player uses a weapon type. This graph can be seen below. For a less crowded graph, see here where a reduced version of the directly-below exists. For a full breakdown of node abbreviations, see here.