Perceptual Similarity for Measuring Decision-Making Style and Policy Diversity in Games
Authors: Chiu-Chou Lin
Wei-Chen Chiu
I-Chen Wu
Venue: Accepted by TMLR ()

Abstract: Defining and measuring decision-making styles, also known as playstyles, is crucial in gaming, where these styles reflect a broad spectrum of individuality and diversity. However, finding a universally applicable measure for these styles poses a challenge. Building on $\textit{Playstyle Distance}$, the first unsupervised metric to measure playstyle similarity based on game screens and raw actions by identifying comparable states with discrete representations for computing policy distance, we introduce three enhancements to increase accuracy: multiscale analysis with varied state granularity, a perceptual kernel rooted in psychology, and the utilization of the intersection-over-union method for efficient evaluation. These innovations not only advance measurement precision but also offer insights into human cognition of similarity. Across two racing games and seven Atari games, our techniques significantly improve the precision of zero-shot playstyle classification, achieving an accuracy exceeding 90\% with fewer than 512 observation-action pairs—less than half an episode of these games. Furthermore, our experiments with $\textit{2048}$ and $\textit{Go}$ demonstrate the potential of discrete playstyle measures in puzzle and board games. We also develop an algorithm for assessing decision-making diversity using these measures. Our findings improve the measurement of end-to-end game analysis and the evolution of artificial intelligence for diverse playstyles.

Published in Transactions on Machine Learning Research (08/2024)
Perceptual Similarity for Measuring Decision-Making Style and Policy Diversity in Games
Chiu-Chou Lin
@sobscureeBottlook.cco com
Deportment of Computer Science National Yang Ming Chiao Tung University, Hsinchu 30010, Taiwon
Wei-Chen Chiu
walon &ccs ncfu. edu.
Department Computer Science National Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan
I-Chen Wu
icwu@cs. www.:eccs....ccooom edu.
Department of Computer Science
National Yang Ming Chiao Tung University, Hsinchu 30010, Taiwan Research Center for Information Technology Innovation Academia Sinica, Taipei 11529, Taiwan
Reviewed on OpenReview: https: openreveee. net/ forum? id= 30C9AWB444
Abstract
Defining and measuring decision-making styles, also known as playstyles, is crucial in gam- ing, where these styles reflect a broad spectrum of individuality and diversity. However, finding universally applicable measure for these styles poses a challenge. Building on Playstyle Distance, the first unsupervised metric to measure playstyle similarity based on game screens and raw actions by identifying comparable states with discrete representations for computing policy distance, we introduce three enhancements to increase accuracy: mul- tiscale analysis with varied state granularity, a perceptual kernel rooted in psychology, and the utilization of the intersoetion-over-uuieee method for efficient evaluation. These innova- lions not only advance measurement precision but also offer insights into human cognition of similarity. Across two racing games and seven Atari games, our techniques significantly improve the precision of zero-shot playstyle classification, achieving an accuracy exceeding 90% with fewer than 512 observation-action Furthermore, our experiments with 2048 and Go demonstrate the potential of discrete playstyle measures in puzzle and board games. We also develop an algorithm for assessing decision-makinn diversity using these measures. Our findings improve the measurement of end-to-end game analysis and the evolution of artificial intelligence for diverse playstyles.
less than half an episode of these games.
1 Introduction
The pursuit of diversity decision-making one of the intrinsic motivations that drive human behavior, resulting in individuality and creativity (Rheinberg) 2020). This evident in the context of games, where decision- decision-making manifests various playstyles, each reflecting unique strategies and characteristics (Bean) 2016 Yannakakis et al. 2013). Alongside the pursuit of diversity, another central as- focuses on achieving optimal performance. Significant advances in decision-making performance have been witnessed, especially with the development. of Deep Reinforcement Learning (DRL) Russell & Norvig 2020). The effectiveness of DRL was first showcased arcade video games (Mnih et al. 2015). Subsoquent applications to board games emphasized its potential, achieving superhuman skills (Silver et al. 2018). This success expanded into various types of games, from Agent57's superhuman performance
& Groth-Marnnn
pect of decision-makiie
in Atari games to groundbreaking feats in Dota 2 and StarCraft. IT (Badia al.) 2020 Berner et. al. 2019:

---

Published in Transactions on Machine Learning Research (08/2024)
Vinyals et al. 2019 Beyond gaming, DRL applications extend to robot control (Andrychowicz et al. and natural language processing (Ouyang et al. 2022), among others.
2020
Yet, while DRL continues to show promise in diverse applications, understanding and analyzing playstyles with minimal domain-specifie knowledge remains a complex endeavor. Data from diverse sources are essential to improve agent strength and efficiency (Fan & Xiao] 2022: Fan et al. 2023), ; just. as acquiring skills from different styles is vital for agents to generalize across tasks (Eysenbach et al., 2019). Although a robust playstyle measure fosters a spectrum of playing strategics, it also reveals the inherent challenges in mcasuring these styles, particularly in environments without built-in features for style measurement. Consequently, achieving precise playstyle measurement remains formidable task (Tychsen & Canossa 2008
There are several methods to evaluate playstyles or perform player modeling (Yannakaiis el al. 2013 from explormes
heuristic rules design to in-game feature exploration (Tychsen & Canossa 2008 Boutchev & Georgieva, 2018
Mader & Tassin, 2022). Supervised lcarning facilitates the discrimination of playstyles (Brombacher ct al. 2017), while unsupervised clustering offers behavioral insights (Drachen et al., 2009 2013 Ferguson et al. p020 Another avenue involves contrastive learning to identify playstyles and behaviors among players (Mellro-- Young et al. 2021; Agarwal et al. 2021). Through these methods, the notion of playstyle can be gauged using distance or similarity measures across game datasets, addressing dynamic and evolving challenges in different, scenarios. The concept is reflected in the work by Lupu et al. (2021). which specifies policy diversity using the divergence of action distributions but requires parametrized policies.
The recent innovation by Lin al. (2021) introduces the Playstyle Distance measure, which stands out by directly measuring playstyle from game screens and raw action pairs. Unlike common methods that compare latent features or rely on parametrized policies, this approach measures action distribution distances directly from raw gameplay samples, reducing the reliance on game features, predefined style labels or even extensive training sets for learning latent features or policies. Its effectiveness hinges on the critical role of state discretization. By discretizing observations like game screens, we can identify similar and comparable states, allowing for a direct comparison of action distributions of each state, thereby defining the decision-making style based on the expected distance of policies.
While the Playstyle Distance offers an advance in end-to-end and unsupervised playstyle measurement, our research endeavors to elevate this foundation. We introduce techniques to improve playstyle measurement, harnessing the advantage of discrete states. Initially, we leverage multiscale analysis with varied state granu- larity, emulating human judgment of similarity from multiple attributes and viewpoints (Medin et al. 1993) We then derive a perceptual kernel from psychophysics (Fechner) 1966) in psychology to obtain a probabilis- tie similarity value, which is more in harmony with human comprehension than distance values. Moreover, incorporating the concept of the Jaccard inder [Murphy] 1996). we broaden the focus of measurement beyond intersection samples, harnessing all observed game data to improve measurement. accuracy. These techniques not only improve the precision of Playstyle Distance but also provide new aspect to understand similarity through the lens of human cognition. From their fusion emerges the Playstyle Similarity measure.
To underscore our contributions, we propose three improvements for playstyle measurement. With these im- provements, we alleviate the trade-off between using small or large discrete state spaces in discrete playstyle measures and provide a more explainable similarity with probability. Additionally, we achieve over 90% accuracy in zero-shot playstyle classification tasks with fewer than 512 observation-actinn pairs, which is less than half an episode in the examined games, including two racing games and seven Atari games. Further- more, our experiments with 2048 and Go demonstrate the potential of discrete playstyle measures in puzzle and board games. Additionally, we introduce an algorithm to mcasure the diversity of decision-making, which showcases the applicability of our measures for tasks that are challenging to quantitatively evaluate without built-in features. This algorithm is based on the simple idea of comparing the similarity of a new trajectory to previous trajectories using a unified probabiliss bilisticc similarity threshold. These explorations en- hance our understanding of end-to-end game analysis and AI training, and also harmoniously merge playstyle measurement with human cognition.
2

---

Published in Transactions on Machine Learning Research (08/2024)
2
Background and Related Works
In this section, we discuss playstyle in depth, provide historical overview, and highlight the importance of discrete represcntation in the creation of general playstyle measures.
2.1 Playstyle and Measurement
Establishing a universally accepted playstyle measure is a formidable challenge, as perceptions of playstyle are influenced by myriad factors and often harbor subjective muances. Consequently, any playstyle mea- sure should specify its evaluative parameters transparently to ensure that its measurements are persuasive. Historically, tailored metrics, characterized by heuristic rules specific in-game features, often presented the most precision for dedicated case studies. For example, the study by Lample & Chaplot (2017) used measures such as object counts, kills, and deaths in shooting games. However, due to their inherent manual nature, these mcasures are often domain-spocific and limited to specific behaviors.
For video games, methods in player modeling can help us find interesting player behaviors and personality (Yannakakis et al. 2013 Costa Jr & McCrae, 1995). A key problem in these behavior analyses is how define effective imput features and the corresponding target outputs. Even if we detect and taxonomize some behaviors, a real playstyle can be complex combination of several behaviors. For example, the Bartle taxonomy in game character theory, where players can be separated into four types: achievers, explorers, socializers, and killers (Bartle) 1996). is a high-level concept of playstyles and the detailed behaviors of these taxonomics can be different in different genres of games. The true playstyle can be represented as the characteristic of playing behavior sets to fuse into a holistic intention of players (Lin et al., 2021); thus, processing all possible information from raw gameplay should be included in playstyle measurements.
To achieve such wider applicability, some researchers have resorted to supervised learning to identify styles (Brombacher et al. 2017). However, this method requires labeled training data and may encounter difficulties in detecting styles not present in the training set. Unsupervised clustering offers a different angle, emphasizing latent feature distances for classification (Drachen et al. 2009 2013 Ferguson et al., 2020). But this approach may obscure the semantic meaning of the measures, particularly when image data is the primary source that is common video games. A notable approach is the Behavioral Stylometry proposed by MMelroy- Young let al. (2021) using the idea of contrastive learning. This measure, designed for chess, encodes chess moves into game vector, aggregates these vectors to represent style, and then compares this representation against reference set. Central to this method is the contrastive learning technique Generalized End-to- End-to-End (Wan et al. 2018) used to learn latent features to identify the most similar player in the given datascts.
For a more generalized measurement of playstyle, one could consider measuring the similarity of policies. Methods that extend to specify similarity or diversity by comparing the action distribution of two policies have also been explored (Agarwal et al.] [201]: Lupu et al., 2021). Notably, these methods often require a parametrized poliey for comparisons. This limitation is addressed by the Playstyle Distance measure {Lin) et al. 2021). ) Instead of emphasizing latent features parametrizod policies, Playstyle Distance focuses the action distributions of given samples. Raw observations, such game screens, discretized and then used for determining which action samples are comparable. Such a method resonates more with human instinct, echoing the case-by-case assessment we often deploy.
2.2 Framework of Playstyle Distance
To delve deeper into the generality and importance of the Playstyle Distance in playstyle measurement, we cxamine its foundation as follows. A pivotal component of its methodology is the use of the Vector Quantized-VVriiatoona AutoEneoder (VQ-VAE), which specializes in discrete represenlations by mapping continuous encodings to the nearest vectors in a predefined codebook (van den Oord et al.) 2017). Building upon VAE, the Hicrarchical State Discrctization (HSD) in Playstyle Distance ensures a concise and hierarchical state space. This is essential for identifying overlapping discrete states while preserving the feature integrity of observation reconstruction and gameplay details.
3

---

Published in Transactions on Machine Learning Research (08/2024)
Herarchiccl
Herarchical
Embbodiin space
Enbedding 50a00
Espacted Distanes over Intersect on States
Dataset MA
Dataset
Naarest Neghtbor
Embezding
Mecresi Naigtbor
Embediingg Index
Disencto
Discrele Enoodar
Fnowler
Foky
Halry
Actore
Arters
Figure/ Illustration of the Playstyle Distonce computation using hierarchical discrete state encoder ø. The Venn diagram highlights the intersection of discrete states for distance calculation.
Central to this framework is the discrete state encoder, denoted as Φ. Obsorvations o and their associated actions a are mapped to datasets M, ~ Style, The encoder @ translates these observations into a compact state representation 8, formulated as:
Sp:0(0) +
In the initial Playstyle Distance approach, the hierarchical encoder ø has the capability to generate multiscale discrete states. However, the foundational literature employs only a singular state space for computations: the hierarchical structure is used to control the state space and maintain the quality discrete representation learning. Using a state space that is too large too small state space may lead to unstable measurements.
From state 8, action distributions deduced using sampling distríbution:
(o, a) E M, - s} = TM (s)
Here, π represents the policy, depicting action distributions for a given state. Subsequently, the distances between these distributions are determinod using the metric D()x,Ty), where the 2- 2assersstnin distance (W2) serves the standard (Vaserstein, 1969.. Recognized for measuring the 'effort' to transform distribution into another, the Wasserstein distance is apt for policy comparisons, analogous to quantifying the 'effort' to transition between playstyles.
The essence of this measure can be succinctly captured as:
1
(M MR) 2 (MAIM)) + (M
(1)
where dg [My) Eo (D
To summarize, the Playstyle Distance framework presents method contrast playstyles, reducing the nood for predefined heuristies and datasets, thereby increasing the applicability in various games. For graphical representation of the framework, see Figure
3 Discrete Playstyle Measures
In this section, delve into series of discrete playstyle measures derived from Playstyle Distance. We first discuss the limitations of Playstyle Distance. We then expand it into a multiscale approach by leveraging the hierarchical structure of states. Subsequently, we explore converting the action distribution distance inlo a perceptual similarity rooted in cognitive psychology, utilizing a perceptual kernel function. Thirdly, we broaden our scope from merely intersection states the union of states, aiming for more efficient estimation of all observed data. Concluding the section, we integrate these improvements into comprehensive measure term Playstyle Similarity.
4

---

Published in Transactions on Machine Learning Research (08/2024)
Intersection States
Multiscale Granularity
Playstyle
Distance
Multiscale
States
Smaller pistance
Perceptual
Distance
Similarity
Kemel
Perceptual Kernel
Intersection
Langer Distance
Union
Playstyle
Similarity
Union States
(a) Degree Similarity
(b) From Playstyle Distance to Playstyle Similarity
Figure 2: (a) Degree Similarity: This demonstrates how multiple candidate points can share identical distance values from a target point T, emphasizing that as distance increases, the degree of similarity information diminishes. (b) From Playstyle Distance to Playstyle Similarity: This transformation begins by processing observation sample into multiple discrete states of varying granularity. A perceptual kernel then transforms these distance values into probabilistic similarities, using the concept of overlapping regions for intuitive understanding. Lastly, the application of the ontersection-overmnee method refines the similarity based on the Jaccard index, enhancing measurement comprehensiveness across all observed data.
3.1 Effect of Multiscale States
Playstyle Distance hinges on discrete states for performing style measurements. Consequently, it resorts to constrained state space sourced from the HSD model. A sample count threshold is applied to the intersection state to ensure the quality of action distribution, necessitating at least t samples in both datasets under comparison; failing which, the state is excluded from the intersection. This filtering sometimes discards important information and the trade-off between using small or large state spaces also poses a dilemma for singular state space; thus, we propose using multiscale analysis to alleviate these problems.
Human cognition perceives similarity as convergence of multiple attributes leading a holistic understand- ing (Goldstone & Barsalou 1998). Hence, advocate for employing varied granularity of discrete states to augment mcasurement capabilities, analogous to human judgment that varies from broad view to intricate details. The HSD model's 's design inherently possesses large stale space for observational reconstruction and the discernment of gameplay nuances (Lin et al.) 2021). Though the state space may be large, intersections are not void if observations are sufficiently similar or come from identical gameplay. It is worth noting that Lin et al. [2021] were able to identify intersection states even with unprocessed screen pixels in Atari games. In different scenario, when treating each state as the same, we can invariably find the single intersection state. In this context, distance simply gauges the action distribution over the entirety of the game, akin traditional methods deploying post-game action statistics.
Broadly speaking, can enhance the original state encoder function, Φ. evolving it into a state encoder mapping, 2 wherein & is an assemblage of mapping functions, ó E Φ, S4 2(o) + {o(o) e Solø e Φ). Consoquently, the projected state of dataset M is defined as: ((M) + {Ø(o) E Salo E M. e Φ). We can then reinterpret Equation I as:
do (M MB)
do do +
do (MMn) MA) where do (Mx|Mr)
1
Σ
dd (My)
(2)
2

---

Published in Transactions on Machine Learning Research (08/2024)
This reformulated measure demonstrates superior accuracy in playstyle classification tasks in exper- iments, even negating the need for a sample threshold count 1. This improvement is likely due to the integration of hierarchical discrete states of varying granularity, which dilutes the impact of outliers during distance computation and leverages more useful details. Furthermore, the adoption of a multiscale state space elfectively mitigales the trade-off between a compact intersection space and the preservation of intri- cate information details. This balance becomes crucial complex games or those that require vast state space to encode trajectory data.
3.2 Perception of Similarity
One potential shortcoming of the Playstyle Distance stems from the nature of distance itself. While distance is a common measure for determining similarity, a larger distance value conveys primarily that two entities are different, without giving much insight into the degree of their similarity. For example, given a point in 2D space, the candidate points with the same distance to the given point form a circle. As the distance increases, the size of this candidate circle also increases, and the similarity information diluted as illustrated in Figure [a This phenomenon has been observed in human decision-making as the Magnitude Effect, suggesting diminished sensitivity to larger numbers (Kahneman & Tversky 1979). This aligns with the Weebe-FFechner Law in psychophysics, where the relationship between stimulus and perception is logarithmic; as the magnitude of stimuli increases, sensitivity diminishes (Fechner) 1966). Drawing from the concept of similarity, we can infer that a smaller distance provides more definitive information about similarity. As distance grows, the distinction becomes vaguer. Therefore, we argue for a measure that reflects higher sensitivity to smaller distances, emulating human perceptual behaviors.
We propose a probability-based model for similarity. In this model, greater similarity (i.c., smaller distance) corresponds to probability closer to 100%, while lesser similarity (larger distance) approaches 0%. This proposed probability function aligns with the logarithmic human perceptual sensitivity to differences. Specif- ically, we use the exponential kernel to describe the probability of similarity, with the mapping function given by P(d)= = J where is the distance value from the policy distance function D(Tx, Ty) with 2- Wasseessttinn distance. This perceptual relation the only relation under our assumptions from human cognition and probability. We provide a proof using differential equations in Appendix A.I
This exponential transformation can also be found in the radial basis function (Vert et al. 2004 and Bhattacharyya coefficient (Bhattacharyya 1946). The Bhattacharyya coefficient BC((P, Q) measures the similarity between two probability distributions P and Q, and it is related to the overlapping region between these two distributions. It is defined as (C Q) = fx (P(=)Q(z)dz The Bhattacharyya distance, derived from the coefficient, is Q)) = (B((BC(P, Q)), and the inversion is BC(P,Q) = ex)(-Ds(P,Q)).
Thus, we define a new playstyle mcasure
with probability of similarity in Equation
(M) EveUueo ((M)))((MM)) P(D (TM) (M) (s))) IUeea 1Usea (M)))l
D(#x, TT)
where D (πx,
D
(3)
The measure has been simplified by adopting a uniform average distance instead of an expected value. This not only streamlines calculations but also underscores the significance of encoder granularity. In particular, intricate encoder with vast state space may be accorded greater weight, especially if the intricate encoder reveals more intersection states. To match probabilistic framework (Appendix A.1 we rescale . the distances with a constant, D ensuring the expected distance converges to 1. The constant D can be calculated by averaging all observed distance on each discrete state in comparisons. Collectively, our revamped mcasure provides a probabilistic lens to interpret similarity, firmly rooted in cognitive theory and tailored for human comprehension. There more discussion about the role of the distance metric in A Apeedix A.2.1 including the implications of adopting the Bhattacharyya distance metric.

---

Published in Transactions on Machine Learning Research (08/2024)
3.3 Beyond Intersection
Before presenting final measure, it is pertinent to revisit the foundational concept of the Playstyle Distance: the intersection of states. Identifying comparable states before measuring policy similarities encounters challenges when the intersoeting samples are limited. A smaller intersection proportion can result in unstable or insufficient samples for measuring playstyles. Such à small intersection could indicate two scenarios. First, distinct state-visiting distributions might signify different playstyles. In contrast, uncontrollable factors external to playstyles, such as environmental randomness or decisions from other players, may also play role, indicating the necessity for more extensive sampling.
A prudent approach would assess the proportion of intersecting samples relative to the total observed samples. In the realm of collection comparison, the Jaccard index (Murphy 1996), also known Intersection over Union, emerges as prevalent similarity measure. The Jaccard indez can serve as an effective playstyle measure under specific conditions. is particularly apt when game observations clearly delineate playstyle distinctions. For instance, in deterministic environments where states can be distinctly segmented by different actions, the Jaccard inder appears to be a fiuting measure. However, complications arise when cerlain states recur due to game rules or every state visited. The task of distinguishing different playstyles based solely on observations becomes considcrably challenging. This evident in single-state games, such as K-arm bandits [Sutton& & Barlo 2018) where measuring playstyles only from states becomes impractical endeavor.
Despite potential challenges, our empirical findings suggest that the Jaccard inder serves as handy measure, when the state space is large and the randomness in the game is low. The incorporation of the Jaccard inder into a playstyle measure with a multiscale state space is expressed in Equation
UsED
(MA, MB) Use U (MMB)
(4)
3.4 Playstyle Similarity
Throughout our exploration, we have derived and discussed various discrete playstyle measures. Collat- ing these insights, we introduce a comprehensive measure termed as the Playstyle Similarity. Defined as PS (MA. MB), it synthesizes our carlier discussions into a singular mcasure as illustrated below:
P A. MB) (M MB) X PS MB)
ExEU.ee d(Ma)no(Mg) P (TMA (5), Mu (s)))
(5)
IUAEo 6(MA) UB((B)
What makes this measure novel is its unique treatment of intersection states. While the Jaceard inder assigns a uniform weight (of 1) to each intersecting state regardless of the similarity between the action distributions, our approach infuses a more nuanced probability-based weighting. The values range wet and 1, increasing proportionally with similarity. This modification alleviates the potential limitation of using the Jaccard inder for playstyle measurements.
Furthermore, our approach ensures consistent interpretation of zero values. For states not part of the intersection, where the distance between action distributions is maximal (approaching infinity), they can be understood as totally dissimilar. Playstyle Distance cannot directly incorporate with the Jaccard index due to nature as a negative similarity measure. The overview of the transformation from Playstyle Distance to Playstyle Similarity illustrated Figure 2b
4 Experiment Settings
In this section, we explain the specifics of our experiment setup, focusing on the datasets, sources of encoder models, and our playstyle classification methodology.
7

---

Published in Transactions on Machine Learning Research (08/2024)
Style
Styles
Siylee
Style.
Random Subsampling
M,
M,
M,
Mensure Simiunity
Styk.
(a) TORCS
(b) RGSK
(c) Atari
(d) Playstyle classification task
Figure 3: Three game platforms and the illustration of zero-shot playstyle classification tasks.
4.1 Game Platforms, Datasets, and Model Source
Our study encompasses three distinct game platforms, as depicted Figures
3a
and
3c
This racing game features stable, controlled rule-based Al players Yoshida al. 2017) The datasets derived from TORCS include total 25 playstyles based on 5 different target driving speeds and 5 different action noise levels. Each observation consists of sequence of 4 consecutive RGB images with a size of 64 x 64. The action space is 2-dimensional and continuons.
TORCS:
2. RGSK- Racing Game Starter Kit: This racing game, available on the Unity Asset Store (Juliani et al. 2020), showcases human players. From RGSK, we have data from total of 24 players, exhibiting individual playstyles. Human players are told to follow one specific style factor in 4 different style dimensions, including using nitro acceleration or not, driving on road surface or the grass surface, keeping the car in the inner or outer of the track, and passing a corner via drifting or slowing down with brake. Each style dimension includes 6 players. Each observation from this game comprises 4 consccutive RGB images of size 72 X 128, with 27 discrete actions.
Atari games with DRL agents: The dataset spans different Atari games (Bellemare et al. 2013) from this platform. Each game includes 20 AI models, all of which demonstrate varied playstyles. These AT models originate from the DRL framework, Dopamine (Castro et al. 2018). Each obser- vation involves 4 consocutive grayscale images of resolution 84 x 84. The action space is discrete and varies depending on the game.
It crucial to clarify that our research did not involve the training new encoder models. Instead, we leveraged three pretrained encoder models and corresponding datasets for each game, provided by [in et al. The associated resources available in their official release! The game details are listed in Table
2021
Table 1: Game details. The game 2048 and Go are used in Section
for extra evaluation.
Game Platform
Agent Type
Style Count
Observation Size
Action Space
TORCS
Rule-based Agent
25
4, 3, 64, 64)
Continuous 2-dimension
RGSK
Human Player
24
4, 3, 72, 128)
Discrete 27
Atari
DRL Agent
20
4, 1, 84, 84)
Discrete 4 to 18
2048*
RL Agent
10
4, 4
Discrete 4
Go
Human Player
200
18, 19, 19]
Discrete 362
4.2 Playstyle Classification and State Space Levels
Our playstyle classification adheres the zero-shot methodology. As depicted in Figure 3d we start with query datasct N, sampled from playstyle Stylen, We then compare this to multiple reference datascts M, each sampled from different playstyles Style. We perform 100 rounds of random subsampling for each playstyle; our primary performance metric for this task is the accuracy of playstyle classification. If dataset exhibits the highest similarity to a reference dataset M,, it suggests that Style,, = Style,, The reported accuracy represents an average, derived from results obtained using the three discrete encoder models.
http:://pppraatthhcccccca/ppppps
game -attid---a--pllytyyl--eeeee playstyle-netric

---

Published in Transactions on Machine Learning Research (08/2024)
Regarding the discrete state space levels, three tiers have been considered:
1. Space size 1, a basic mapping with state space 1, which maps all observations identically.
2. Space size 200 as suggested by Playstyle Distance.
3. Space size 24644-144 256"x, level trained by HSD for the base hicrarchy, depending the resolution res of convolution features from the game screens.
5 Results in Video Games
In this section, we assess the efficacy of our methods on two racing gamees and seven Atari games. Initially, we demonstrate how a multiscale state space can aid in the selection of proper state spaces and poten- tially enhance the accuracy of Playstyle Distance. Subsequently, we compare several baselines, illustrating that probabilistic values for measuring similarity offer a viable alternative to distance values. Next, we incorporate all observed data to evaluate measures across all platforms. Furthermore, we discuss the contin- uous playstyle spectrum, demonstrating the consistency of measure values under slightly different behaviors. Lastly, we compare potential unsupervised measures beyond discrete playstyle measures, using observation latent features common in generative styles. Overall, our intention in this section is to determine whether new measures can enhance playstyle measurement in video games.
5.1 Multiscale State Space Efficacy
We evaluate the proposed multiscale state space and compare it with the singular state space used by Playstyle Distance in Table We record the mean accuracy and corresponding standard deviation over 100 rounds of random subsampling for each discrete encoder. Each dataset sampled from the given playstyles compriscs 1024 observation-action pairs. For example, TORCS, there are playstyles; we poll on 25 query sets, and each dalaset has 1024 observation-action pairs sampled from its playstyle. We compute this query set with another 25 candidate sets, each having 1024 pairs sampled from their playstyles. When this round of polling is finished, we count whether the most similar candidate set has the same playstyle as the query set. This polling process will run over 100 times for sampling different subsets of actual playstyle datasets. Additionally, we examine the intersecting states' sample threshold count 4, where an intersecting state requires at least t samples in both compared datasets for a stable action distribution estimation.
In our comparisons, conventional methods, such as supervised learning and contrastive learning, fall short for this classification due to the lack of playstyle labels groups in the training datasets, resulting in random model. Thus, focus our comparisons on Playstyle Distance, detailed in Equation 2 For the multiscale version, which incorporates three discrete state spaces {1,220 11,2202.255e 256res) using the label "mix." The results, as shown in Table indicate that using a multiscale discrete state space not only simplifics the selection of a proper state space by using all spaces and potentially yiclds superior results but also obviates the need for a sample threshold count & for intersecting states in some games requiring stable action distribution, such as TORCS, Asterix, and Breakout.
simplify our terminology by
5.2 Probability vs. Distance
In addition to introducing the multiscale discrete state space, another key contribution of our work is the proposal to use probabilistic similarity from a perceptual perspective rather than employing negative distance mcasure of similarity. To clucidate the benefits of this modification, we examine the relationship between accuracy and dataset size of the sampled observation-action pairs. These pairs are evaluated under single discrete state space {220), without employing sample count threshold, to provide clear assessment of the transformation from distance to similarity. Further comparisons with different discrete state spaces can be found in Appendix A.2
We evaluate several measures this comparison:
Playstyle Distance: -d
9

---

Published in Transactions on Machine Learning Research (08/2024)
Table 2: Playstyle accuracy (%) + standard deviation (%) when employing various discrete state spaces in the multiscale version of Playstyle Distance, with sample threshold count for intersecting states. Conventional methods, such as supervised learning and contrastive learning, are not suited for cases lacking playstyle labels in the training datascts, resulting in random model with Style Count accuracy. In contrast, discrete playstyle measures do not suffer from this limitation, they do not require playstyle labels for training the discrete encoders. The accuracy and standard deviation values are averaged from 3 different encoder models. The results show that using a multiscale state space (mix) offers more convenient state space selection and potentially improves the accuracy. Additionally, the sample threshold count t suggested by Lin el al. (2021) can be ignored, which was required in TORCS with 2 space and in Atari games like Asterix and Breakout with 256 space for stable measurement. The detailed statistics for each discrete
encoder are listed in Section A.3.1
22 t-2
220
256 t=2
256 t=1
mix t=2
mix
TORCS
35.1 + 9.1
73.3 + 8.2
66.5 = 7.9
4.3 + 3.1
60.9 + 9.4
77.3 + 7.4
77.5 = 7.9
RGSK
81.0 = 7.2
79.2 7.9
93.7 4.7
5.7 + 2.5
25.6 7.2
78.8 = 7.5
93.5 4.3
Asterix
25.2 + 9.0
99.9 + 0.5
100 + 0
49.6 + 7.7
32.7 + 8.0
100 + 0
100 = 0
Breakout
32.7 + 9.2
99.4 + 1.6
99.9 + 0.6
65.9 + 8.5
29.9 + 9.4
99.8 + 1.1
99.9 = 0.2
MsPac.
100 = 0
99.9 = 0.5
100 = 0
92.8 + 4.0
100 + 0
100 =
100 =
Pong
49.9 9.7
92.1 2.7
92.3 2.6
50.7 + 9.5
52.2
9.9
93.1 3.2
92.4 + 2.6
Qbert
99.9 + 0.5
100
100 +
90.1 + 5.3
91.6 + 4.6
99.9 0.5
100 = 0
Seaquest
82.0 = 7.6
99.7 1.2
99.9 + 0.6
17.1 = 5.2
16.7 - 4.9
99.9 - 0.3
99.9 0.2
Spaceln.
73.1 + 8.5
98.7 = 2.3
99.7 + 1.2
50.4 + 5.7
49.6
8.4
99.9 + 0.5
99.9 = 0.6
Playstyle Intersection Similarity: PSB
Playstyle Inter BD Similarity: PS a variant of PSG that employs the Bhattacharyya distance in place of the 2- Wasserstein distance
Playstyle Inter BC Simdlarity: PSD the Bhattacharyya cocfficient version, which omits the scaling coefficient before the perceplual kernel f
Random: uniform random bascline that a common result from supervised learning or contrastive learning if there is no style label group (like self and others) information in the training data.
Results presented in Figure suggest that probabilistic similarity be good alternative to distance- based similarity, offering improved explainability in terms of measure values. Among the methods evaluated, the 2- 2-Wasserstein distance with a perceptual kernel and the Bhattacharyya coefficient emerge as superior candidates. The intention behind using probabilistic similarity is that provides a consistent measure of similarity across different games (via likelihood). For distance similarity, understanding the property's 's and distribution of distance is essential to interpret the measure value. Besides the explainability of similarity values, the transformed similarity value can be incorporated with the Jaccard index as described in Sec- tion 3.3 The evidence shows that results with probabilistic similarity not worse than distance similarity and are slightly better on TORCS, which includes slightly different, playstyles. The upcoming experiments in Sections 5.3 and 5.4 also support the idea of probabilistic similarity under slightly changed playstyles, such as rule-based based TORCS agents and Atari game agents trained with the same algorithm.
5.3 Full Data Evaluation
Based on the previous evaluations, we further perform comprehensive evaluation of various playstyle measures, including leveraging full data with union operations. The evaluation method mirrors the one presented Section 5.2 but expands the scope beyond racing games and adopts a multiscale state space.
Detailed results for each Atari game have been moved Appendix A.3.2 Instead, leveraging the consistent observation and action space shared across Atari games, we propose unified Atari console evaluation. This
10

---

Published in Transactions on Machine Learning Research (08/2024)
TORCS State Space: {240)
RGSK State Space: (2*0)
100
100
Playstyle Distance
Playstyle Distance
Playstyle Intersection Similarity
80
Playstyle Intersection Similarity
Playstyle inter BD Similanty
Playstyle Inter BD Similarity
Playstyle Inter BC Similarity
Playstyle Inter BC Similarity
Aoursiy
Random
Random
20
2
2
2
2
27
2
2
2
22
2
2
2
2
2
2
2
Datenal Size
Delase Stm
(a) TORCS
(b) RGSK
Figure 4: Comparison of Efficacy: Probabilistic Distance Approaches. The plot illustrates the relation- ship between accuracy (Y-axis) and size of the sampled observation-action pairs (X-axis). The shaded area indicates the range between min and max accuracy among three encoder models.
evaluation views each DRL agent's gameplay individual games distinct playstyles, yielding 7 20 unique playstyles. As for the discrete state space, we include single shared state mapping in addition to two hierarchical discrete encoders from the seven games; thus, there are totally 1 1+ x 2 discrete state encoders. For actions, rather than aligning their semantics across games, we expand the action set to the largest. count Atari games, which 18. This based on the assumption that variations game content can be interpreted as different states.
The platforms span TORCS, RGSK, and Atari games. We compare the following measures:
Playstyle Distance: -d4
Playstyle Intersection. Similarity: PSG
Playstyle Inter BC Similarity; PSCRO
Playstyle Jaccard Index: Ja
Playstyle Similarity: PSJ
Playstyle BC Similarity: PSS the union version of Playstyle Inter BC Similarity
Random: A uniform random baseline.
Results displayed in Figure 5 show that the Playstyle Similarity outperforms its counterparts. Moreover, the Jaccard index has proven to be uscful in practice. Our combined Atari console evaluation further underscores the robustness and adaptability of our measures.
Conclusively, our proposed Playstyle Similarity measure excels across these video game platforms. It is particularly impressive that it can identify playstyles with over 90% accuracy with just 512 obscrvation- action pairs - less than half an episode across all tested games. This suggests the possibility of accurate playstyle prediction even before a game concludes, paving the way for real-time analysis.
5.4 Continuous Playstyle Spectrum in TORCS
This experiment investigates the response of similarity measure values to variations in a continuous playstyle spectrum within the TORCS environment. It particularly focuses on whether these measures can accurately rank playstyles, ensuring precise predictions for the closest playstyle (Top-1 similarity) and maintaining a correctly ordered sequence playstyles based on similarity measures.
Utilizing the TORCS dataset, which inchudes five levels target speeds (60, 65, 70, 75, 80) and five levels of action noise, provides broad spectrum for examining continuous playstyle changes. To illustrate, consider
11

---

Published in Transactions on Machine Learning Research (08/2024)
TORCS State Space: {1, (1,220,256*4)
100
Playstyle Distance
Playstyle Intersection Similarity
80
Playstyle Inter BC similarity
Playstyle Jaccard Index
Accurscy (96)
60
Playstyle Similarity
Playstyle BC Similarity
40
Random
20
2
2
23
2
2
2
(a) TORCS
RGSK State Space: {1,2 256144)
100
Playstyle Distance
Playstyle Intersection Similarity
80
Playstyle Inter BC Similarity
Playstyle Jaccard Index
Accuracy (%)
60
Playstyle Similarity
Playstyle BC Similarity
40
Random
20
0
21
22
2
24
2
2
27
2
29
Dataset Size
(b) RGSK
Atari Console State Space: {1, 7x2 122
100
Playstyle Distance
Playstyle Intersection Similarity
80
Playstyle Inter BC Similarity
Playstyle Jaccard Index
Accurscy (%)
60
Playstyle Similarity
Playstyle Similarity
40
Random
20
2
22
23
24
2
2
27
2
2
Dataset Size
(c) Atari Console
Figure Playstyle Mcasure Evaluation in TORCS, RGSK, and Atari Console. The plots showcase the efficacy of different measures in the context of the 'Full Data Evaluation' subsection. The shaded area indicates the range between min and max accuracy among three encoder models.
12

---

Published in Transactions on Machine Learning Research (08/2024)
Table 3: Corner Case uses Playstyle Similarity (mix): consistent count = 9. The query playstyle is marked as orange, and those columns with consistency measure values are marked in blue. For rows with consistency, marked as red, and for those values with consistency in both columns and rows, marked violet.
Speed60N0
Speed60 (C) Speed65 (C) Speed70 (C)
Speed75 (C)
Speed80 (C)
No (C)
0.0753
0.0650
0.0608
0.0555
0.0472
N1 (C)
0.0590
0.0579
0.0549
0.0513
0.0446
N2
0.0519
0.0533
0.0525
0.0458
0.0440
N3 (C)
0.0482
0.0473
0.0448
0.0435
0.0382
N4 (C)
0.0421
0.0413
0.0402
0.0349
0.0337
five playstyles labeled A through E, with A' being a variant closely aligned with A. We anticipate that the similarity between A' and A would be the highest, progressively decreasing towards E. This expectation the stage for consistency test, wherein similarity mcasure M should validate the order M(A,, A) ( B) > M(A',C) > M(A'DD) M(A', (Corner Case).
We assess the following measures:
1. Playstyle Distance with 2 state space (baseline)
2. Playstyle Distance using mixed state space
3. Playstyle Interscction Similarity with mixed state space
Playstyle Jaccard Index with a mixed state space
Playstyle Similarity with a mixed state space
With 100 rounds of random subsampling (each dataset consisting of 512 observation-action pairs) and using the first Hierarchical State Discretization (HSD) model, we examine the consistency of similarity values as playstyle shifts. docrease in similarity consistent with playstyle changes marked with a (C), indicating measure reliability.
For instance, choosing Speed60NO as target playstyle, observe how similarity measures adjust across row column in response to increasing speed or action noise levels. A practical demonstration reveals how the measure values consistently decrease across increasing speed levels and noise intensities, illustrating the measure's ability to capture playstyle divergence accurately. Table 3 presents one such example with Playstyle Similarity (mix). For all measure values, please refer to our Appendix A.4
Similar analyses exlend Center Case scenario with Speed70N2 the target playstyle, emphasizing the necessity for similarity measures to respect two key sequences for consistency:
M(C', > M(C", B) M(C', A) M(C',C) > M(C", D) M(C",E)
These sequences confirm the measure's capacity to accurately reflect. the gradual divergence of playstyles from a central reference point. The examples of Center Case can be found in Appendix A.4
Table f summarizes the consistency count across all evaluated measures, showcasing the reliability and effectiveness of each similarity measure in maintaining continuous playstyle spectrum.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Table 4: Consistency count of two continuous playstyle spectrum cases.
Corner Case
Center Case
Playstyle Distance (220)
2
Playstyle Distance (mixed)
4
2
Playstyle Intersection Similarity (mixed)
8
2
Playstyle Jaccard Index (mixod)
5
1
Playstyle Similarity (mixed)
9
3
5.5 Comparison of Potential Unsupervised Similarity Measures
With previous experiments, we have verified the effectiveness of these discrete playstyle measures. Although there few methods for unsupervised playstyle measurement before Playstyle Distance, there some measures popular in generative styles with latent feature similarity. For example, the two most popular measures for research in generative adversarial nelksrks (GAN) with styles (Karras et al., 2019) are the Inception Score and Fréchet Inception Distance (FID) (Salimans et al., 2016 Heusel et al., 2017). These are based on using an image classification model, Inception (Szegedy et al., 2015). for scoring the generated images. The Inception Score is based on the prediction distribution of the classilier to delect real or generated images, which diverges from the target of playstyle measurement as all observations are real images from game environments. FID measures the 2- 2-Wasserstein distance on the latent features of images and could be incorporated into playstyle measurement if we assume a playstyle features unique observation distribution. However, calculating the W2 distance for high-dimensional continuous latent distributions is computal ionally intensive and docs not yicld good results in the game TORCS (Lin et al. 2021). The major complexity arises from the calculation related to the covariance matrix. A lime-feasible alternative is using the similarity of the mean vector these latent features. We can first average those latent features of observations into a mcan vector to represent the playstyle and compare the similarity to candidate vectors obtained from the same process, which is analogous to the method in Behavior Stylometric (Mellroy- Youllr al., 2021). These latent features used in the following experiments are the continuous latent features before vector quantization to the 220 state space in the HSD models with 500 dimensions. The discrete version of these latents with information loss has alrcady shown effectiveness in playstyle measurements, so we compare using continuous latents with two popular similarity measures for playstyle measurements: Euclidean Distance (L,2 distance) and Cosine Similarity, Cosine Similarity is especially common in latent similarity applications (Chung et al. 2022 Hellroy-Youung et al. 2021
We assess the following measures:
Playstyle Distance with a 220 state space
Playstyle Jaccard Index with a mixed state space
3. Playstyle Similarity with mixed state space
Playstyle BC Similarity with a mixed state space
5. Random, the uniform random baseline
6. Euclidean Distance, using L2 distance as the similarity measure for observation latent features
Cosine Similarity, using Cosine Similarity the similarity measure for observation latent features
We first check the results for TORCS and RGSK in Figure 0 It. is clear that Euclidean Distance and Cosine Similarity do not provide good predictions, which is not surprising since the observations in the TORCS practice track are monotonous, and the playstyles are not directly correlated to observations. In contrast, playstyles in RGSK, such as nitro acceleration or preferred road surface, have a high correlation to visual features and even perform better than Playstyle Similarity under few samples. When we further examine
14

---

Published in Transactions on Machine Learning Research (08/2024)
100
100
Playstyle Distance (2"h)
Playstyle Distance (2**)
Playslyle Jaccard Index
80
Playstyle Jaccard Index
Playstyle similarity
Playstyle Similarity
Aoons PN
Playstyle Similerity
Playstyle Similarity
Random
Aouraty
Random
Euclidean Distance
Euclidean Distance
Cosine Similarity
Casine limilariity
20
21
2
2
2
2.
2'
2
2
2
2
2"
2
2
2
Datanal Size
Detasel Som
(a) TORCS
(b) RGSK
Figure 6: Accuracy comparison of potential unsupervised similarity measures on (a) TORCS and (b) RGSK. The shaded area indicates the range between the min and max accuracy among three encoder models. Ob- servation latent-based methods like Euclidean Distance and Cosine Similarity perform nearly at random levels in TORCS and perform much better in RGSK due to different playstyles and game properties. The playstyles in RGSK have many visual features that reflect styles, such as the blue fire from the nitro accel- cration system and the driving surface position on the track.
these measures Atari games in Figure Euclidean Distance and Cosine Similarity usually share similar performance, and the results depend on the games. In all cases, Playstyle Similarity and its BC variant are superior to these potential candidates with datasets over 256 in size, regardless of the playstyle and games. The measure values based on action distributions are more explainable than observation latent features. Furthermore, observations are sometimes not controllable by the players performing playstyles but are con- trolled by game mechanisms, other players, or even sample bias. Discrete playstyle measures can defend against these influences since observations are used for comparison conditions rather than direct measure- ment computations. If these influencing factors make observations different, discrete playstyle measures act conservatively, requiring more samples deeming them incomparable, while latent similarity approaches will directly take them into calculation without considering the influencing factors. Overall, these results show that our Playstyle Similarity and its BC variant are more general and effective in unsupervised playstyle measurement compared to other existing or potential measures.
6 Playstyle Measures Under High Uncertainty Games
To further assess the efficacy of our playstyle measures in games characterized by high uncertainty, we conducted experiments with the puzzle game 2048 and the board game Go (Figure
6.1 2048: High Randomness Puzzle Game
Known for its single-player format and high degree of randomness, 2048 presents challenges in generating identical trajectories. We trained a reinforcement learning agent (Szubert & Jaskowski 2014] over 10 million episodes, creating 10 distinct players by saving the model every million episodes. For each player, we collected 1000 episodes, using the first 500 the reference datasct and the remaining 500 as scparate query datasets. This resulted in total of 5000 query datasets for the experiment. The setup aimed test the accuracy of playstyle measures in scenarios marked by high randomness and similar behaviors across small query datascts, simulating conditions that could challenge discrete playstyle measures. In this case, we directly utilized the x 4 full board as the discrete states, which is equivalent to raw game screens in 2048 in terms of state information. The experimental settings used for this analysis are listed in Appendix A.6
The results in Table demonstrate that measures incorporating the Jaccard index negatively impact accu- racy. In games with high randomness and large observation space, is crucial to evaluate whether discrete states can identify key style factors while maintaining manageable state space find comparable samples.
15

---

Published in Transactions on Machine Learning Research (08/2024)
Alari Asterix
Alari Breakaut
100
100
Playstyle Distance (2
Playstyle Distance (2
Playstyle Jaccard Incex
Playstyle Jaccard Index
Playstyle similarity
Playstyle Similarity
A000 Ao0
Playstyle BC Similariy
Random
fooraey
Playstyle BC Similarily
Euclidean Distance
Euclidean Stance
40
Cosine Similan
40
CosimE Sin larity
21
2
2
2"
2
29
2'
2
2
21
2
2
2
2
2
2
2
Oxteaa
Detumet
(a) Atari Asterix
(b) Atari Breakout
MsPacman
Pong
100
100
Playstyle (2
Playstyle Distance (2*t)
80
Playstyle Jaccard Incee
80
Playstyle Jaccard Index
Playstyle Similarity
Playstyle Similarity
Playstylc Similority
Playstylc BC Similarity
Random
Random
Euclidean Distance
Euclidean Distance
Cosine Similaria
Cosine Similarity
20
20
2
2
Dstaset Sizt
Detaset S
(c) Atari MsPacman
(d) Atari Pong Atari Seaquest
Atari Obert
100
100
Playstyle Distance (2
Playstyle Distance (22*)
Playstyle Jaccard Inciex
Playstyle Jaccard Index
Playstyle Similarity
Playstyle Similarity
kcurasey 183
60
Playstyle BC Similerity
Rourssy (N)
60
Playstyle BC Similarity
Random
Random
Euclidean Distance
Euclidean Distance
Similarity
sirilaritty
20
0
0
21
2
2
24
2
23
2
(c) Atari Qbert
(I)
Atari Seaquest
Atari Space Invaders
Atarl Console
100
100
Playstyle Distance (2?0)
Maystyyl Distance (2
Playstyle Jaccard Incex
Playstyle Jaccard Index
Playstyle Similarity
Playstyle Similarity
Playstyle BC Similarity
Playstyle BC Similarity
Random
Euclidean Distance
Euclidean Distance
40
Cosine Similarity
40
Cosine Sim larky
21
2
2
24
2
26
27
29
29
21
22
23
24
20
20
27
28
29
Dataset Size
Detaset 30
(g) Atari Space Invaders
(h) Atari Console
Figure 7: Accuracy comparison of potential unsupervised similarity measures Atari games. The shaded area indicates the range between the min and max accuracy among three encoder models. Euclidean Distance and Cosine Similarity show nearly the same performance across different games, sometimes better and sometimes worse than Playstyle Distance.
16

---

Published in Transactions on Machine Learning Research (08/2024)
2
8
32
256
4
8
4
2
2
(a) 2048
(b) Go
Figure 8: Game screens of 2048 and Go.
Table 5: Accuracy of game 2048 model identification.
Accuracy
Playstyle Distance
98.90%
Playstyle Intersection Similarity
98.52%
Playstyle Inter BC Similarity
98.90%
Playstyle Jaccard Index
49.22%
Playstyle Similarity
71.26%
Playstyle BC Similarity
71.26%
6.2 Go: Two-Player Board Game
In addition to the inherent randomness of game environments, the inclusion of other players introduces further uncertainty in measuring playstyles. Consequently, we undertook a human playstyle identification task using the two-player board game, Go, which is known for its high game tree and state space complexity challenge discrete playstyle measures.
(Van Den Herik et al. 2002). This complexity
We implemented a variant of the HSD encoder (Lin et al. 2021) to obtain a discrete state encoder for this task. The major difference from the original HSD is that the reconstruction objective is replaced by predicting the win value, with prediction policy and value being the standard objectives Alpha Zero series algorithms (Silver al.) (218)). The details about this discrete encoder are described in Section A.7 The Go dataset used in this study was sourced from Fox Go (Fox Go 2024a[b) and provided by the team of the MiniZero framework (Wu et al. 2024). It includes 45,000 games from players with 9 Dan Go skill for training the encoder, with corresponding actions but without player information or style labels. Another dataset includes 200 human players with Go skill ranging from Dan to 9 Dan, each contributing 100 games to the query datasets and 100 games to the candidate datasets. The discrete state space used for training the encoder includes (4*,16", 16", 2563613.
Results in Table demonstrate the accuracy comparisons of the player identification task with different discrete playstyle measures. There two playstyle scenarios in our evaluation. For the first 10 moves case, it is common and straightforward in board games that preferred opening is kind of playstyle and using small state space can achieve 97.0% accuracy ({4*)). However, when we do not specifically focus the playstyles in the opening, we may want to use all game moves to capture any possible playstyles, and such a small state space negatively impacts the accuracy. Some boards cannot be compared in this kind of playstyle, but the small state space cannot provide information to separate these cases. Instead, large state spaces like [16*] or (256361) can handle this kind of problem. If we do not know which state space can give the best result, our multiscale state space is a good choice that leverages all state spaces. Even if it may be influenced by very bad state space like (48) in the full games case, the damage is limited. Also,
17

---

Published in Transactions on Machine Learning Research (08/2024)
Table 6: Accuracy of Go 200 player identification with M games as the query set and also M games as the candidate set. The full comparison with different numbers of query and candidate sets listed in Section A..7] The discrete encoder is trained from variant of HSD (Lin et al. 2021), and the available state spaces this encoder (4%, 16 256361). ). The measures with "mix" notation imply using all three available state spaces simultaneously with our multiscale modification.
Only First 10 Moves
M-1
M-5
M-10
M-25
M-50
M-75
M-100
Playstyle Distance (48)
1.5%
11.5%
33.5%
71.0%
87.0%
93.0%
97.0%
Playstyle Distance (16*).
2.0%
10.0%
41.5%
74.0%
85.0%
92.5%
95.5%
Playstyle Distance (256361)
2.0%
8.0%
38.0%
75.0%
87.0%
92.0%
94.5%
Playstyle Distance (mix)
1.5%
14.0%
46.5%
75.5%
87.5%
94.5%
96.5%
Playstyle Inter. Similarity (mix)
2.5%
15.0%
42.0%
68.5%
84.0%
94.5%
94.5%
Playstyle Inter BC Similarity (mix)
2.5%
14.0%
42.5%
72.0%
89.0%
96.5%
95.5%
Playstyle Jaccard Index (mix)
3.0%
9.5%
21.5%
49.0%
78.0%
88.0%
95.0%
Playstyle Similarity (mix)
3.0%
21.5%
40.0%
65.0%
87.5%
94.0%
97.0%
Playstyle BC Similarity (mix)
3.0%
24.0%
45.5%
70.5%
88.0%
96.5%
97.0%
Full Game Moves
M=1
M=5
M=10
M=25
M=50
M=75
M=100
Playstyle Distance (48)
2.5%
5.5%
11.5%
20.0%
31.0%
39.5%
50.0%
Playstyle Distance (16$)
2.0%
10.0%
44.5%
74.0%
86.0%
93.5%
96.5%
Playstyle Distance (256361)
2.0%
8.0%
39.5%
75.5%
88.5%
93.5%
96.5%
Playstyle Distance (mix)
2.0%
17.0%
33.0%
56.5%
76.0%
83.0%
90.0%
Playstyle Inter. Similarity (mix)
3.0%
19.0%
38.0%
66.0%
86.0%
95.0%
95.0%
Playstyle Inter BC Similarity (mix)
2.5%
16.5%
37.5%
66.0%
90.0%
96.0%
97.0%
Playstyle Jaccard Index (mix)
3.0%
5.0%
8.5%
22.5%
51.0%
66.0%
80.5%
Playstyle Similarity (mix)
4.5%
15.5%
29.5%
56.5%
81.5%
90.5%
94.0%
Playstyle BC Similarity (mix)
4.0%
22.0%
45.5%
73.0%
92.0%
97.5%
97.5%
discrete playstyles incorporating the Jaccard index (Playstyle Similarity and its variants) can further cover this weakness.
Additionally, the Bhattacharyya coefficient is tend be more effective than using perceptual kernel with scaled W2 distance when there sufficient samples in Go, especially in the full game moves case. It is possibly due to its distribution property that rapidly decreases similarity with few overlapping outcomes, and our Go result is an example. For slightly different playstyles like TORCS with continuous actions, using the scaled W2 metric can provide better accuracy. Our findings suggest that discrete playstyle measures can achiove significant accuracy with sufficient samples dircctly from the measure definition oven in a complex multi-agent board game, without any predefined style labels.
7 Diversity Measurement in DRL
With these discrete playstyle measures, we can design an algorithm to quantify the diversity among DRL models, which is challenging to measure and quantify formally environments with high-dimensionall ob- servations. Algorithm provides simple method to quantify diversity in decision-making by mcasuring the similarity between new trajectory and observed trajectories. If new trajectory is not similar enough to any observed trajectories, count it different one.
We conducted experiments to evaluate this algorithm using the Atari DRL agent datasct. Each DRL algorithm (DQN, C51, Rainbow, IQN, (Mnih el al. 2015 Bellemare et al. 2017 Hessel el al. 2018 Dabney et al. 2018)) includes models, each contributing trajectories, resulting in 25 trajectories per algorithm. Using Algorithm [] we asscssed the diversity of trajectorics produced by each algorithm. Results Table averaged across three discrete encoder models with similarity threshold of t 0.2, show the capacity of models trained under the same DRL algorithm to generate diverse game episodes within 25 attempts. The
18

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm Measuring Policy Diversity
Input: Policy Environment &, Similarity mcasure M
Input: Similarity threshold t, Number of trajectories N
1: Initialize $ (store trajectories) and diverse trajectory count d = 0
2: for 1 to N do
Generate a trajectory Ti ~ n, Set is diverse true for each T in S do
Compute similarity M(G, if M ( Ty) 2 t then
7:
8:
is diverse = false
9.
break
10:
end if end for
11:
12:
if is diverse then
13:
d d= end if
14:
15:
Store ni in S
16:
end for
Output: Return d (diverse trajectory count) and N (total trajectories)
Table 7: Averaged diverse trajectory count of diffcrent DRL algorithms across 7 Atari games in 25 episodes.
DRL Algorithm
Asterix
Breakout
MsPacman
Pong
Qbert
Seaquest
SpaceInvaders
DQN
6.00
6.00
5.33
4.00
6.00
11.00
25.00
C51
6.00
7.00
7.00
6.00
7.00
21.00
25.00
Rainbow
8.67
5.33
8.00
5.00
5.00
24.00
25.00
IQN
25.00
14.00
10.00
9.00
10.00
25.00
25.00
IQN algorithm displays higher diversity across games, consistent with its risk sampling feature. For more details and use cases, please refer to Appendix A.5)
8 Conclusion and Future Works
In this research, we introduced three techniques Lo enhance discrete playstyle measures: adopting a multiscale state space, using perceptual similarity rooted in human cognition, and applying the Jaccord inder observed data. These advanccments have been incorporated into playstyle measurement for the first time and collectively give rise to our playstyle measure, Ploystyle Similarity. This measure stands out in terms of accuracy and explainability, requiring minimal predefined rules and data. Notably, the integration of multiscale state space expands the measure's applicability, particularly for games that have a trade-off between small and large state spaces for game details. Furthermore, our literature review and theoretical proof about human perception bridge the gap between distance similarity and human cognition playstyle. In addition to the common accuracy cvaluations our experiments, we also conducted scries of statistical tests using McNemar's test (McNemar) 1947) to report some results in the main paper with p-values in Section A.8 These tests help to determine whether two results have statistical significance rather than showing differences due to sampling uncertainty.
The Playslyle Similarily measure oflers new potential for end-to-end game analysis and AI training with specific playstyles, such as diversity analysis or human-like behaviors (Fuji et al. 2013). As an example, we propose an algorithm to quantify the diversity among DRL models, and the playstyle classification tasks on human playstyles in RGSK and Go also support future applications for human- agents. These insights emphasize that AT development extend beyond simple measures like scores win rates in games with
19

---

Published in Transactions on Machine Learning Research (08/2024)
high-dimensional observations, encompassing more behavioral patterns. Additionally, the quantification of policy diversity becomes more tangible.
In conclusion, despite our validation efforts across several platforms, many games remain unexplored. Fur- thermore, some playstyles can be shared across different games, existing in similar scenarios for game rec- ommendation systems (Fear) 2023)- Beyond gaming, playstyle measures can provide more behavioral infor- mation for other decision-making topics, such as Al safety (Amodei al. 2016) and the interactions among language model agents (OpenAl 2024 Park et al. 2023
Broader Impact Statement
We advise caution when using these measures for analyzing policy diversity due to potential sensitivities to discretization techniques and inherent explainability challenges in neural networks. Additionally, employing unsupervised methods could raise privacy concerns, as they facilitate targeted advertising or fraudulent activities without the need for costly playstyle labeling.
Acknowledgments
We would like to express our sincere gratitude to Chen (2023) for generously sharing the Go playstyle dataset based on the MiniZero Framework (Wu et al. 2024). which was instrumental training our discrete encoder and conducting the playstyle tests. Additionally, we appreciate the valuable feedback and insights provided by the reviewers, which greatly helped in refining and improving this work. Finally, would like to thank Ubitus K.K. for motivating the research topics related to video game playstyles in our early studies.
References
Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Marc G. Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement lcarning. In International Conference Learning Represenlations (ICLR), 2021.
Dario Amodei, Chris Olah, Jacob Steinhardi, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety, CoRR, htss/606.06655 2016.
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. The International Journal of Roboties Research, 2020.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhao- han Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In Inter- national Conference on Machine Learning (ICML), 2020.
Richard Bartle. Hearts, clubs, diamonds, spades: Players who suit muds. Journal of MUD research, 1996.
Anthony Bean and Gary Groth-Marnat. Video gamers and personality: A five-factor model to understand game playing style. Psychology of Popular Media Culture, 2016.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 2013.
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforccment learning. In International Conference on Machine Learning (ICML), 2017.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ily Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, htt//1912.06668 2019.
20

---

Published in Transactions on Machine Learning Research (08/2024)
Anil Bhattacharyya. On measure of divergence between two multinomial populations. Sankhya: the indian journal of statistics, 1946.
Boyan Bontchev and Olga Georgieva. Playing style recognition through an adaptive video game. Compulers in Human Behavior, 2018
Patrick Brombacher, Johannes Masino, Michael Frey, and Frank Gauterin. Driving event detection and driving style classification using artíficial neural networks. In IEEE International Conference on Industrial Technology (ICIT), 2017.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kunar, and Mare G. Bellemare. Dopamine: A research framework for deep reinforcement learning. CoRR, abs:1111.006110, 2018.
Chun-Jung Chen. Detecting individual decision-making style in the game of go. Master's thesis, National Yang Ming Chiao Tung University, 2023. URL https://hdl handlanndlleaaae wwt. ntt.2/e//n///111//99111
Inseop Chung, Daesik Kim, and Nojum Kwak. Maximizing cosine similarity between spatial features for unsu- pervised domain adaptation in semantic segmentation. In IEEE/CVP Winter Conference on Applications of Computer Vision (WACV). IEEE, 2022.
Paul T Costa Jr and Robert R McCrae. Domains and facets: Hierarchical personality assessment using the revised neo personality inventory. Journal of personalily assessment, 1995.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile nelworks for dist ional reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), International Conference on Machine Learning (ICML), 2018.
Anders Drachen, Alessandro Canossa, and Georgios N. Yannakakis. Player modeling using self-organization in tomb raider: Underworld. In IEEE Conference on Computational Intelligence and Games (CIG), 2009.
Anders Drachen, Christian Thurau, Julian Togelius, Georgios N. Yannakakis, and Christian Bauckhage. Game data mining. In Game Amalylics, Marimizzng the Value of Player Dalu. Springer, 2013.
Benjamin Eysenbach, Abhishek Gupla, Julian Ibarz, and Sergey Levine. Diversily all you need: Learning skills without a reward function. In International Conference on Learning Representations (ICLR), 2019.
Jiajun Fan and Changnan Xiao. Generalized data distribution iteration. In International Conference on Machine Learning, (ICML), 2022.
Jiajun Fan, Yuzheng Zhuang, Yuecheng Liu, Jianye HAO, Bin Wang, Jiangcheng Zhu, Hao Wang, and Shu-
Tao Xia. Learnable behavior control: Breaking atari human world records via sample-efficient behavior selection. In International Conference on Learning Representations (ICLR), 2023.
Andrew Fear. Playstyle analysis for game recommendations, 6 2023. US Patent Application US20230177583AL.
Gustav Theodor Fechner. Elements of Psychophysics. Holt, Rinehart and Winston, 1966. Originally pub- lished in 1860.
Mark Ferguson, Sam Devlin, Daniel Kudenko, and James Alfred Walker. Player style clustering without game variables. In Georgios N. Yannakakis, Autonios Liapis, Penny Kyburz, Vanessa Volz, Foaad Khosmood, and Phil Lopes (eds.), International Conference on the Foundations of Digital Games (FDG), 2020.
Ronald Aylmer Fisher. Statistical methods for rescarch workers. In Breakthroughs in statistics: Methodology and distribution, pp. 66-70. Springer, 1970.
Fox Go. Fox Go, 2024a. URL https://wwa http:/:///wwwwww...oooo
Accessed: 2024-07-07.
Fox
Go. privacyguidellnesv2.htt Accessed: 2024-07-07.
Fox Go Privacy Guidelines, 2024b.
URL https://edu.. http::///du..ooq.... com/ www.ccomccccoecc complex/
21

---

Published in Transactions on Machine Learning Research (08/2024)
Nobuto Fujii, Yuichi Sato, Hironori Wakama, Koji Kazai, and Haruhiro Katayose. Evaluating human- like behaviors of video-game agents autonomously acquired with biological constraints. In Advances in Computer Entertainment Technology (ACE), 2013.
Robert L. Goldstone and Lawrence W. Barsalou. Reuniting perception and conception. Cognilion, 1998.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence (AAAT), 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. In Advances in Newral Information Processing Systems (NewrIPS), 2017.
Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents. CoRR, abs/1809.022 2020.
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. Econometrica, 1979.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Tn IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Guillaume Lample and Devendra Singh Chaplot. Playing FPS games with deep reinforcement learning. In Satinder Singh and Shaul Markovitch (eds.), AAAI Conference on Artificial Intelligence (AAAI), 2017.
Chin-Chou Lin, Wei-Chen Chiu, and I-Chen Wu. An unsupervised video game playstyle metric via state discretization. In Conference on Uncertainty in Artificial Intelligence (UAT), 2021.
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob N. Foerster. Trajectory diversity for zero-shot coor- dination. In International Conference on Machine Learning (ICML), 2021.
Stéphanie Mader and Eloise Tassin. Playstyles in tetris: Beyond player skill, score, and competition. Tn International Conference on Entertainment Computing (IFIP ICEC), 2022.
Reid Mcllroy-Younr Yu Wang, Siddhartha Sen, Jon M. Kleinberg, and Ashton Anderson. Detecting individ- ual decision-making style: Exploring behavioral stylometry in chess. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 1947.
Douglas L. Medin, Robert L. Goldstone, and Dedre Gentner. Respects for similarity. Psychological Review, 1993.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
Allan H. Murphy. The finley affair: signal event in the history of forecast verification. Weather and Forecasting, 1996.
OpenAl. Chatgpt: A large language model, 2024. URL httpttt//ccat..opteet ope.aa..cmmaaccmmmm.mmmmm.. Accessed: 2024-06-19 19.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
22

---

Published in Transactions on Machine Learning Research (08/2024)
Joon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. Tn ACM Symposium on User Interface Software and Technology (UIST). ACM, 2023.
Falko Rheinberg. Intrinsic motivation and flow. Molivalion Science, 2020.
Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach (4th Edition). Pearson, 2020.
Tim Salimans, Jan J. Goodfellow; Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
David Silver, Thomas Hubert, Julian Schrilwieser, Ioannis Anlonoglou, Malthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, second edition, 2018.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanct, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent. Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9. IEEE, 2015.
Marcin Grzegorz Szubert and Wojeiech Jaskowski. Temporal difference learning of n-tuple networks for the game 2048. In IEEE Conference Computational Intelligence and Games (CIG), 2014.
Anders Tychsen and Alessandro Canossa. Defining personas in games using metrics. In International Academic Conference on the Future of Game Design and Technology (FuturePlay). 2008
H Jaap Van Den Herik, Jos WHM Uiterwijk, and Jack Van Rijswijck. Games solved: Now and in the future. Artificial Intelligence, 134, 2002.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Leonid Nisonovich Vaserstein. Markov processes denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii, 1969.
Jean-Philippe Vert, Koji Tsuda, and Bernhard Schölkopf. A primer on kernel methods. Kernel methods in computational biology, 2004.
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Mamuel Kroiss, Ivo Danihelka, Aja Huang, L. Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezh- nevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Caglar Gulcehre, Ziyun Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent rein- forcement learning. Nature, 2019.
Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez-Moreno. Generalized end-to-ond loss for speaker verification. In International Conference on Acouslies Speech and Signal Processing (ICASSP), 2018
Ti-Rong Wu, Hung Guei, Pei-Chiun Peng, Po-Wei Huang, Ting Han Wei, Chung-Chin Shih, and Yun-Jui Tsai. Minizero: Comparative analysis of alphazero and muzero on go, othello, and atari games. IEEE Transactions on Games, 2024.
Georgios N. Yannakakis, Pieter Spronck, Daniele Loiacono, and Elisabeth André. Player modeling. In Artificial and Computational Intelligence in Games. Schloss Dagstuhl Zeibniz-eentrum für Informatik, 2013.
Naoto Yoshida, Alexander Khimulya, Billy Zhou, and Arun Kumar. Gym-TORCS. https://github. com/ ggo-nama-kuue kun/ htee//yymmeereee torca 2017.
23

---

Published in Transactions on Machine Learning Research (08/2024)
A Appendix
A.1 A Proof of the Perceptual Kernel
In the main paper, we claim that P(d) ed is the if and only if kernel function (as discussed in Section 3.2 We provide a proof of this claim using differential equations.
We make some assumptions about the perceptual kernel. First, P(d) function that maps the distance d between two given action distributions a probability value describing their similarity. Since distance is continuous random variable, P(d).
probability density function f(d) to describe the mapping function
We might intuitively think of P(d) as equal to f(d), even though the probability density value is not the same as the probability value. Thus, we use a cumulative distribution function F(d) to describe P(d). We redefine a real-valued random variable D as the distance variable, where d E D, and a random variable X, where a E X, and + X X() -d. The probability of similarity, denoted as P(X S x), is derived from the distance value d. Thus, (x) P(X I 2), and from the assumptions of similarity, - Fx(x) 0, and Fx(0) = 1. Also, Fx (x) be described with probability density function fx(x)) follows:
Fx(x) = J fx(t)dt
(6)
The corresponding equation to Fp(d) = P(d) becomes:
Fb(d)= = /xa fD(()dt
(7)
Additionally, we adopt another assumption from the field of psychophysics known as the Weber Pechner law Fechner [9666.. Fechner's 's law states that the relationship between stimulus S and perception is logarithmic and can be described as differential equation:
ds S
dp k
(8)
Here, 1e is a constant depending on the sense and type of stimulus.
By integrating the equation, we obtain:
p p= kms
(9)
Where C' is a constant of integration, and it is defined in Fechner's law assumiig that the perceived stimulus becomes zero at some threshold stimulus Sp, where = 0, and = Sg. Thus, can be calculated as follows:
In So
(10)
Combining Equation and Equation 10. Fechner's law Fechner 1966 is:
S So
p k In
(11)
Now, we discuss the roles of p and S in our similarity scenario to construct. a probability density function (d). We assume that similarity weakens as the distance increases, and there is a finite maximal similarity when the distance is O. Additionally, the density value and distance are always non-nogative.
We first assume that p is the density value of similarity and consider Equation 11 There two cases:
24

---

Published in Transactions on Machine Learning Research (08/2024)
1. If S represents distance and k is positive, this is incorrect since p approaches as 8 - 0.
2. If change the growth direction of distance that & is negative, this is still incorrect since p still approaches S O.
Considering invert Equation as follows:
S = So exp t
(12)
Now, we assume that S is the density value of similarity and consider Equation 12 There are two cases:
1. If p represents distance and is positive, this is incorrect since approaches p - oo, although there is finite maximal value So when p = 0.
2. If change the growth direction of distance that h is negative, this meets desire since S approaches 0 p oc, there is finite maximal value So when p 0, and the density value is always non-negalive.
Finally, we can simplify the equations by assuming, for the sake of simplicity, that So equals This assumption is based the intuition that the trend of decreasing similarity and increasing distance is similar around distance 0 in various scenarios:
fD(d) = exp ?
(13)
Returning Equation 7 FD(d) can be described as follows:
( fD(t)dt
Fd
f exp (G)dt
(14)
(lim & exp ( + -+k'')( d 10000
+C))
-kexpp )
Considering that the sum of density values must. be over (-00, we rewrite Equation as follows:
lim Fx(x)
fx(t)dt
(15)
fx(1)di
The corresponding equation to (old) P(d) becomes:
25

---

Published in Transactions on Machine Learning Research (08/2024)
lim Fb(d) = [ fo(t)dt
d--00
In(l)dl
(n(()dtt + S/- fn(l)dt
(16)
-00
0+ Lo Sn(()dt
=1
(D) =1
Combining Equation 14 and Equation 16
0 (
Fb(0) -k kxpp
=
=1
k = -1
(17)
FD(d) = exp(
(d) = edà
P(d) f
Therefore, we have verified the claim that P(d) to derive the equations from Equation 12 17
ed If there is case where So + it is straightforward
Besides, the expected value of distance is can be obtained by the equations as follows:
E
D
<f
000
0+
xfn(
J
I dr
(18)
6
-| -11
-1-0-1+- C')
- -0-1
lim
+ CC)
(+000
ct
0 (-1)
This concept of expected value is used to scaling the distance value in different scenarios, as described in Section 3.2 with the notation D M
A.2 Perceptual Similarity Under Different State Spaces
There are various methods for generating discrete representations, and the effectiveness of perceptual sim- ilarity may vary under these representations, especially when combined with our proposed multiscale state space. In this section, explore the impact of different. state space choices perceptual similarity.
26

---

Published in Transactions on Machine Learning Research (08/2024)
A.2.1 Bhattacharyya distance implementation
In this paper, we also provide some variants of Playstyle Similarity, which use Bhattacharyya distance or coefficient as an alternative to the 2-Wasserstein metric to assess the difference in playstyle from different perspective. Bhattacharyya distance is related the overlapping region between two distributions, and it is defined through Bhattacharyya cocfficient BC. The value range of BC is [0. 1], and the corresponding distance DR is Dn = -ln(BC). For discrete probability distribution, it is simple lo compute the Bhat- tacharyya coefficient: Q)( = Ezex ((()))((). However, it is more challenging to calculate for continuous probability distributions, as in the case of actions in racing games like TORCS, since it involves the integration of probability density functions: BC(P, Q) = Leex v vp(z)q(x). Thus, we adopt the formula- tion of multivariate normal distributions of Bhattacharyya distance (DB) (Bhattacharyya 1946) as follows, where p; = (m, E,):
In( detE {11 - 122)" (MI - -12) + 2 det22
1
Dn(PI,P2)
DR,
8
(19)
where == Σ1 + E2
2
Additionally, we clip the maximum Bhattacharyya distance to 10 to prevent extremlly large value from affecting the average scaling (2t = 0.00004539992 = 0%). The small value for dealing with singular matrices in matrix determinant calculation is set to 1e-8.
Recalling our earlier discussion, we mentioned that the Wasserstein distance can be likened to the 'effort' required to transition between different playstyle action distributions (as described in Section 2.2 The Bhattacharyya distance, in contrast, isn't about this 'effort". Instead, it gauges the likelihood that two playstyles will result in the same action. This is due to its relation to the overlapping regions between two distributions. Thus, while the Playstyle Similarity is built on the idea of the effort needed to change playstyles, the Playstyle BC Similarity (or its variant Playstyle BD Similarity) is built on the frequency of identical actions. This distinction might relate to different roles within a game. For instance, a player in the game might be more concerned with the effort required to shift playstyles, while an observer might focus more on the actions they witness. Think of it this way: players exert effort, like moving their fingers to press buttons or manipulate a joystick or even a mental effort to change their belief of playing. The observer, on the other hand, sees only the outcome of these actions, without much insight into the effort involved.
A.2.2 Multiscale State Space with HSD
20 256 generated Figure presents the results of experiments conducted with a multiscale state space 21, from HSD models, as described in Section [.2] The results indicate improvement in accuracy for TORCS, while there is no clear improvement in RGSK. Notably, in RGSK, the accuracy of the perceptual kernel with sample sizes 25 to 2 decreases, suggesting that detailed information for distinguishing these styles has a negative effect. To further investigate, we conducted two ablation studies to understand the effeetiveness of the proposed measures for playstyle similarity. The first study focuses on using only the base hierarchy of HSD with a very large state space (256res), while the second study explores the use of a single-state state space {1} to assess the measures. Figure g illustrates that the measurement is unstable when there are few intersecting samples in a very large state space. However, the negative effect of detailed information is mitigated when considering intersection over union. Figure g also shows that even with single state space, the action statistics of the dataset can offer some information to differentiate playstyle, especially in RGSK, where Lin et al. (022)) made their human players follow some playstyles closely related to the keyboard actions, such as using the nitro system or braking in the racing game.
A.2.3 Discrete Representation from Downsampling
There are several existing methods for generating discrete representations. One conventional method for image data is downsampling to a lower resolution. While the downsampling parameters often require tuning for effective processing, it is a straightforward method that does not require training a neural network model. Previous work by Lin et al. (2021) attempted to use resouuton downsampling as a discretization method, but they encountered challenges due to the lack of intersection states in their settings.
27

---

Published in Transactions on Machine Learning Research (08/2024)
TORCS State Space: (1)
RGSK State Space: (1)
100
100
Playstyle
Playstyle
Playstyle Intersection Similarity
Playstyle Intersection Similarity
Playstyle Inter BD Similanity
Playstyle Inter BD Similarity
Ao0r (N)
Playstyle inter BC Similarity
Loorney 133
Playstyle Inter BC Similarity
Random
Random
40
40
2
2
2
2
2
2
21
2
2
2
21
2
2
2
2
2
22
2
Detimel
Deteset Sine
(a) TORCS with state space.
(b) RGSK with state space.
TORCS State Space: {2
RGSK State Spece: (2
100
100
Playstyle Distance
Playstyle Distance
80
Playstyic Intersection Similarity
80
Playstyle Intersection Similarity
Playstyle Inter BD Similarity
Playstyle Inter BD Similarity
Playstyle Inter BC Similarity
Playstyle Inter BC Similarity
Random
Accuncy
Random
20
20
2
29
2
Dstaset Si2e
Dososct Size
(c) TORCS with 220 state space.
(d) RGSK with 220 state space.
Space: (2564))
(256244)
100
100
Playstyle Distance
Maystyle Distance
Playstyle Intersection Similarity
Playstyle Intersection Similarity
Playstyle BD Similarity
Playstyle Inter BD Similarity
Reurse 188
60
Playstyle Inter Similarity
[oursee
60
Playstyle Inter BC Similarity
Random
Random
20
0
2
2
2
2
2
2
2
200
(e) TORCS with 256 state space.
(I) RGSK with 25614
256144 state space.
TORCS State Space: 22, 256
RGSK State Space: 2 256
100
100
Playstyle
Playsiyle
Playstyic Intersection Similarity
Playstyle Intersection Similarity
Playstyle Inter BD Similarity
Playstyle Inter BD Similarity
Playstyle Inter BC Similarity
Playstyle Inter BC Similarity Random
ACc00o Ph)
Random
40
40
22
2
25
26
27
23
20
2
23
2
26
27
20
29
200
Ditaset
Dasaset Size
(g) TORCS with multiscale state space.
(h) RGSK with multiscale state space.
Figure 9: Comparison of Efficacy: Using difffrrnt state spaces in discrete playstyle measurement. for TORCS and RGSK, including single state space {1}, {220) state space, the base hierarchy of HSD with state space [256tes) and multiscale state space 20 256res]. The shaded area indicates the range between the mini- mum and maximum accuracy among three encoder models.
28

---

Published in Transactions on Machine Learning Research (08/2024)
In experiments, explore the use of downsampling create discrete representations with different state spaces. In TORCS, we map original game screen observations to three levels of state space:
1. 1: Basic mapping with state space 1, which maps all observations identically.
2. 168x8; Downsampling from 4 x [64,64.3] 256-intensity observations to x [8,8,1] 16-intensity ob- servations.
3. 16xx8x4; Downsampling from 4 x [64,64,3] 256-intensity observations to 4 x [8,8,]] 16-intensity observations.
For RGSK, we similarly map original gamn screen observations to three levels of state space:
Basic mapping with state space 1, which maps all observations identically.
16 Dowusampling from 4 [72,188,3 256-intensity observations to x [9,1611] 16-intensity observations.
160x16x4, Downsampling from [72,128.3] 256-intensity observations to [9,16,1] 16-intensity observations.
The results in Figure 10 show that downsampling be viable discretization method in some cases, but overall, the measurement is either unstable shows significant difference compared in these measures. These results highlight the importance of having discrete representations with high quality, providing proper granularity for playstyle features.
A.3 More Results of Video Game Evaluation
In this supplementary section, we delve deeper into the results to provide comprehensive analysis of measures evaluations under different state spaces.
A.3.1 Statistics of Each Discrete Encoder with Multiscale State Space
In this subscction, we provide the mean and standard deviation of the accuracy from Table 2 in Table g There are 3 available discrete encoders trained using the HSD method in our experiments. Their mean and standard deviation values are calculated based on 100 rounds of random subsampling, with each dataset in subsampling consisting of 1024 obsorvation-action pairs. We employ sampling without replacement (there are no duplicated observation-aation pairs in the two comparing datasets) except for Atari games, where some games have fewer than 2 1024 pairs for sampling double the size of the dataset.
A.3.2 Individual Atari game results with multiscale state space
Figure shows the relationship between playstyle classification accuracy and sampled dataset size for the seven Atari games. Playstyle Simvilarity (PS,) and its variant Playstyle BC Similarity (PSY3C) have nearly the same performance, and Playstyle Jaccard Inder (Je) can have decent result. This evidence justifies that some playstyles, especially in a deterministic environment, can be differentiated solely with observations, which explains why the work by Eysenbach et al. 2009 considers states only for diversity.
A.3.3 Atari game results with a smaller state space
Figure 12 shows the relationship between playstyle classification accuracy and sampled dataset size for the seven Atari games and the combined version (Atari Console). These results show that measures with intersection over union still perform well in Atari games even with a smaller state space. Although it seems that Playstyle Jaccard Inder long as all states are visited in the sampled dataset, as described Section 3.3 This potential problem is discussed in more detail in Section A.3.4 where even with a state space of 200 the Playstyle Jaccard Inder may not perform well.
decent and easy measure, we know that it theoretically does not work as
29

---

Published in Transactions on Machine Learning Research (08/2024)
TORCS Space: (1699)
State (16%10)
100
100
Playstyle Distance
Playstyle Distance
Playstyle Intersection Similarity
Playstyle Intersection Similarity
Playstyle BD Similarity
Playstyle BD Similarity
Reursa 1N)
60
Playstyle Inter Similarity
60
Playstyle Inter BC Similarity
Random
Random
0
2
2
2
2
2
2
2
2
2
2
2
2
(a) TORCS with 16 state space.
(b) RGSK with 169x1 1G state space.
TORCS State Space: (16ttten)
RGSK State Space: [1692400)
100
100
Playstyle
Playstyle
Playstylc Intersection Similarity
Playstyle Intersection Similarity
Playstyle inter BD Similanty
Playstyle Inter BD Similarity
Playstyle Inter BC Similarity
Playstyle Inter BC Similarity
Acurecy TM
Random
Random
40
40
20
22
2
25
2.
27
23
20
27
23
2
2
20
29
2
Dstaset Size
Docaset Size
(c)
TORCS with 16 x68xx state space.
(d) RGSK with 16911 state space.
TORCS State Space: (1, 16 16126,26
RGSK State Space: (1, 16225, 16setssay
100
100
Playstyle Distance
Mlasttyle Distance
80
Playstyle Intersection Similarity
80
Playstyle Intersection Similarity
Playstyle Inter BD Similarity
Playstyle Inter BD Similarity
Rooring 183
Playstyle Inter BC Similarity
Playstyle Inter BC Similarity
Random
Random
20
20
2
2
Detamal
(e) TORCS with multiscale state space.
(I) RGSK with multiscale state space.
Figure 10: Evaluation of discrete representations using downsampling, considering intersection states in TORCS with state spaces {163x8), (168x8x4), and {1, 168x8 168x8x4), and in RGSK with state spaces [169116), [169x16x4), and (1, 11x16 [60xx
30

---

Published in Transactions on Machine Learning Research (08/2024)
Table 8: Playstyle accuracy (%) = standard deviation (%) when employing various discrete state spaces in the multiscale version of Playstyle Distance, with sample threshold count t for intersecting states.
220 1-2
220 t=1
256 t-2
256 t=1
mix t-2
mix t=1
TORCS
73.3 8.2
66.5 = 7.9
4.3 3.1
60.9 + 9.4
77.3 - 7.4
77.5 + 7.9
Encoder1
70.3 9.1
63.8 8.5
5.8 4.3
55.0 + 9.9
74.2 8.1
75.9 + 8.3
Encoder2
71.6 9.1
3.6 3.5
60.1 = 9.8
77.0 = 7.1
75.5 + 7.9
Encoder3
77.9 6.6
7.7
3.4 1.4
67.6 + 8.6
80.8 7.1
81.1 + 7.6
RGSK
79.2 7.9
93.7 + 4.7
5.7 + 2.5
25.6 + 7.2
78.8 + 7.5
93.5 + 4.3
Encoder1
80.4 7.5
93.0 + 5.0
5.6 2.4
23.4 7.6
81.2 7.4
93.4 + 4.6
Encoder2
80.2 8.1
97.4 2.9
5.0
20.5 + 6.9
78.2 + 7.8
96.9 + 3.1
Encoder3
76.8 8.1
90.7 + 6.1
6.3 2.9
32.9 + 7.3
77.2 + 7.4
90.3 + 5.3
Asterix
99.9 0.5
100 + 0
49.6 + 7.7
32.7 + 8.0
100 + 0
100 + 0
Encoder1
99.9 0.7
100 + 0
48.7 = 7.3
48.0 + 6.9
100 + 0
100 + 0
Encoder2
99.9 0.7
100 + 0
50.0 = 7.2
24.5 + 8.3
100 1
100 + 0
Encoder3
100
100 =
50.2 = 8.5
26.7 8.8
100
100 + 0
Breakout
99.4 + 1.6
99.9 + 0.6
65.9 = 8.5
29.9 + 9.4
99.8 + 1T1
99.9 + 0.2
Encoder1
98.9 2.4
99.8 + 1.1
67.5 = 8.8
32.7 - 9.6
99.8 = 1.2
99.9 + 0.5
Encoder2
99.4 1.8
99.9 + 0.9
70.9 + 7.8
30.8 + 9.5
99.8 + 1.1
100 + 0
Encoder3
99.9 0.5
59.2 + 8.9
26.4 + 9.0
99.8 + 1.0
100 + 0
MsPac.
99.9 + 0.5
100 + 0
92.8 + 4.0
100 = 0
100 + 0
100 + 0
Encoder1
99.6 + 1.4
100 + 0
93.6 + 4.1
100 = 0
100 + 0
100 + 0
Encoder2
100
100 =
92.8 = 4.3
100 = 0
100 = 0
100 + 0
Encoder3
100
100
92.1 = 3.7
100
100 0
100 + 0
Pong
92.1 + 2.7
92.3 + 2.6
50.7 + 9.5
52.2 + 9.9
93.1 + 3.2
92.4 + 2.6
Encoder1
92.0 + 2.7
92.6 2.5
52.2 = 9.4
52.6 9.0
93.7 - 3.4
92.5 + 2.8
Encoder2
92.4 2.9
92.2 + 2.7
48.3 + 9.1
51.4 + 10.4
93.1 + 3.1
92.4 + 2.6
Encoder3
92.0 2.6
92.2 + 2.5
51.5 + 10.0
52.6 + 10.2
92.5 + 3.1
92.5 + 2.5
Qbert
100 + 0
100 + 0
90.1 + 5.3
91.6 + 4.6
99.9 + 0.5
100 +
Encoder1
100 0
100 + 0
95.4 = 3.8
99.9 + 0.7
99.9 - 0.5
100 +
Encoder2
100 0
100 +
82.1 = 6.7
82.8 8.0
99.9 + 0.5
100 +
Encoder3
100 0
100 = 0
92.8 = 5.5
92.2 5.1
99.9 = 0.5
100 0
Seaquest
99.7 + 1.2
99.9 + 0.6
17.1 = 5.2
16.7 + 4.9
99.9 - 0.3
99.9 + 0.2
Encoder1
99.9 + 0.9
99.9 + 0.5
17.1 = 5.3
15.8 + 4.5
100 + 0
100 + 0
Encoder2
99.9 + 0.7
100 + 0
17.2 + 5.3
17.3 + 5.3
99.9 + 0.9
99.9 + 0.5
Encoder3
99.7 1.2
99.6 + 1.4
17.2 + 5.0
17.0 + 5.0
100 + 0
100 + 0
SpaceIn.
98.7 + 2.3
99.7 + 1.2
50.4 + 5.7
49.6 + 8.4
99.9 + 0.5
99.9 + 0.6
Encoder1
98.8 + 2.5
99.9 + 0.9
51.2 - 8.4
50.2 + 7.8
99.8 + 1.0
99.9 + 0.7
Encoder2
99.6 + 1.4
99.9 - 0.9
50.3 - 0.1
50.2 + 8.7
100 - 0
99.9 + 0.5
Encoder3
97.9 + 3.0
99.3 + 1.9
49.9 + 8.8
48.6 + 8.8
99.9 + 0.5
99.9 + 0.7
31

---

Published in Transactions on Machine Learning Research (08/2024)
Asterix State Space: 21, 256421
Breakout State Space: (1. (29, 256121
100
100
do
-dg
PSg
PSg
PS g
Aoorb0 1N)
Nouray 180
P5%
P5
40
40
P5gec
Readom
21
2
2
2
2
2
2'
2
2
2
2
2
2
2'
2
2
Oxtomet
Dateset
(a) Asterix
(b) Breakout
MsPacraan State Space: (1,220,256 2
Pong State Space: (1,229.25444
100
100
de
80
P53
80
P5gec
P5
AccURNY Ma
PS
PS
PS
Random
Random
20
20
2
2
24
Dstaset Size
Decaset Size
(c) MsPacman
(d) Pong
Obart Space: 220, 256
Seaquest State Space: 256
100
100
do
-dg
PSg
PS
P5g
PS
60
Ja
60
PS)
P5
PSyac
P5
Random
Random
0
23
2
24
2
2
2
(e) Qbert
(f) Seaquest
Spacelnvaders State Space: (1, 249, 256422)
100
80
PSg
PS
60
PS
40
pS
Random
20
21
2
22
27
2
2
Delamat 5
(g) Spaceluvaders
Figure 11: Playstyle Measure Evaluation in Atari games. The plots showcase the efficacy of different measures in the context of the 'Full Data Evaluation' subsection. The shaded area indicates the range between min and max accuracy among three encoder models.
32

---

Published in Transactions on Machine Learning Research (08/2024)
Asterix State Space: (2
Breakout State Space: (2
100
100
-do
-d
PSg
PSg
Aoorto 1N)
LourNN 180
P5
P5
40
40
P5g
Random
21
2
2
2
2'
2
2
2
2
2
2
2
2"
2'
2
2
Oxtomet
Dateset
(a) Asterix
(b) Breakout
MsPacman State Space: (2
Pong State Space: (2
100
100
de
80
P53
80
P5gec
P5
AccURNY Ma
PS
PS
Psyde
PS
Random
Random
20
20
2
2
24
2
Dstaset Size
Decaset Size
(c) MsPacman
(d) Pong
Qbert Space: (2
Seaquest Space: (2
100
100
do
-dg
PSg
PS
PSg
PS
60
Ja
60
PS)
P5
PSyac
P5
Random
Random
0
2a
2
24
2
2
(e) Qbert
(f) Seaquest
Spaceinvaders State Space: (2*)
Atari Console State Space: (7x22")
100
100
do
PSg
80
PS8
P5:8C
PS
KoUrsey PM
PS
P5
PSync
40
PS
Random
Random
20
20
2
23
24
22
2'
2
2
21
2
2
2
2
Detamel Saw
Decaset Sine
(g) SpaceInvaders
(h) Atari Console
Figure 12: Playstyle Measure Evaluation in Atari games. The plots showcase the efficacy of different measures with 200 state space from HSD models. The shaded area indicates the range between min and max accuracy among three encoder models.
33

---

Published in Transactions on Machine Learning Research (08/2024)
TORCS State Space: (2#0)
RGSK State Space: (2
100
100
Playstyle Distance
Playstyle Distance
Playstyle Intersection Similarity
B0
Playstyle Intersection Similarity
Playstylc inter BC Similarity
Playstyle Inter BC Similarity
Playstyle Jaccard Index
Playstyle Jaccard Index
Playstylc Similarity
Playstyle Similarity
Playstylc BC Similarity
Playstyle BC Simi antt
Randorn
40
Random
20
2
2
2
2
27
2
2
2
2
2
2
2
2
2
2
Detanel 5.zm
Oacavet Stne
(a) TORCS
(b) RGSK
Figure 13: Playstyle Measure Evaluation in two racing games.. The plots showcase the efficacy of different measures with a 2 state space from HSD models. The shaded area indicates the range between min and max accuracy among three encoder models.
A.3.4 TORCS and RGSK with a smaller state space
In this section, we conducted experiments using a reduced state space of 220 for the two racing games, TORCS and RGSK, without employing the multiscale technique.
Figure B illustrates that the Playstyle Jaccard Indez performs the poorest in TORCS and exhibits slightly inferior performance Playstyle Similarity and Playstyle BC Similarily RGSK. This observation provides valuable insights into the suitability of the Playstyle Jaccard Inder for precise measurements, particularly in scenarios involving randomness (c.g., TORCS players employing different action noises) where observations exhibit only slight variations (e-g., stable rule-based AI controllers in TORCS with slightly different target). Further investigation may be warranted to understand the reasons behind these performance differences.
A.4 Continuous Playstyle Spectrum in TORCS
This experiment investigates the response of similarity measure values to variations in continuous playstyle spectrum within the TORCS environment. It particularly focuses on whether these mcasures can accurately rank playstyles, ensuring precise predictions for the closest playstyle (Top-1 similarity) and maintaining a correctly ordered sequence playstyles based on similarity measures.
Utilizing the TORCS dataset, which includes five levels target speeds (60, 65, 70, 75, 80) and five levels of action noise, provides a broad spectrum for examining continuous playstyle changes. To illustrate, consider five playstyles labeled A through E, with A' being variant closely aligned with A. We anticipate that the similarity between A' and A would be the highest, progressively decreasing towards E. This expectation sets the stage for our consistency test, wherein a similarity measure M should validate the order M M(A', A) > M(A' B) > M(A',C) > M(A',D) D) > M(A', (,, F) (Corner Case).
We assess the following measures:
1. Playstyle Distance with 220 state space (baseline)
2. Playstyle Distance using mixed state space
Playstyle Intersection Similarity with mixed state space
4. Playstyle Jaccard Index with a mixed state space
5. Playstyle Similarity with mixed state space
With 100 rounds of random subsampling (each dataset consisting of 512 observation-action pairs) and using the first Hierarchical State Discretization (HSD) model, we examine the consistency of similarity values
34

---

Published in Transactions on Machine Learning Research (08/2024)
Table 9: Consistency count of two continuous playstyle spectrum cases.
Corner Case
Center Case
Playstyle Distance (220)
2
Playstyle Distance (mixed)
4
2
Playstyle Intersection Similarity (mixed)
8
2
Playstyle Jaccard Index (mixod)
5
1
Playstyle Similarity (mixed)
9
3
Table 10: Corner Case uses Playstyle Distance (220): consistent count 3
Speed60NO
Speed60 (C)
Speed65
Speed70
Speed75
Speed80
NO (C)
-0.0044 (0.0012)
-0.0048 (0.0010)
-0.0055 (0.0010) -0.0069 (0.0011) -0.0100 (0.0014)
N1 (C)
-0.0054 (0.0011)
-0.0064 (0.0015)
-0.0069 (0.0016)
-0.0086 (0.0013)
-0.0108 (0.0016)
N2
-0.0064 (0.0010)
-0.0058 (0.0010)
-0.0068 (0.0013)
-0.0086 (0.0010)
-0.0097 (0.0015)
N3
-0.0073 (0.0011)
-0.0078 (0.0013)
-0.0071 (0.0010) -0.0100 (0.0017) -0.0118 (0.0016)
N4
-0.0092 (0.0013) -0.0083 (0.0011) -0.0089 (0.0013) -0.0110 (0.0015) -0.0132 (0.0017)
playstyle shifts. A decrease in similarity consistent with playstyle changes is marked with a (C), indicating measure reliability.
For instance, choosing Speed60NO as target playstyle, we observe how similarity measures adjust across a row or column in response to increasing speed or action noise levels. A practical demonstration reveals how the measure values consistently decrease across increasing speed levels and noise intensities, illustrating the measure's ability to capture playstyle divergence accurately.
Similar analyses extend to Center Case scenario with Speed70N2 as the target playstyle, emphasizing the necessity for similarity measures to respect two key sequences for consistency:
1. M(C',C) M(C,BB) MCC,,, A)
2. M(C C) > M(C D) M(C
These sequences confirm the measure's capacity accurately reflect the gradual divergence of playstyles from central reference point. The detailed findings, presented in Tables 19 underscore the nuanced performance of cach evaluated measure. Additionally, we present the standard deviation of the measure values, denoted as (std), alongside the measure values themselves. The query playstyle is marked as orange, and those columns with consistency in measure values are marked in blue. For rows with consistency, marked as red, and for those values with consistency in both columns and rows, marked violet.
Table 11: Corner Case uses Playstyle Distance (mix): consistent count
Speed60NO
Speed60 (C)
Speed65
Speed70
Speed75 (C)
Speed80
NO (C)
-0.0018 (0.0005)
-0.0022 (0.0005)
-0.0028 (0.0006) -0.0037 (0.0006)
-0.0060 (0.0007)
N1 (C)
-0.0024 (0.0005)
-0.0028 (0.0007)
-0.0036 (0.0008)
-0.0043 (0.0007)
-0.0058 (0.0008)
N2
-0.0028 (0.0005)
-0.0027 (0.0006)
-0.0031 (0.0006)
-0.0045 (0.0007)
-0.0056 (0.0008)
N3 (C)
-0.0030 (0.0005)
-0.0033 (0.0007)
-0.0035 (0.0006)
-0.0049 (0.0007)
-0.0060 (0.0009)
N4
-0.0038 (0.0006)
-0.0037 (0.0006)
-0.0040 (0.0006)
-0.0055 (0.0008)
-0.0071 (0.0011)
35

---

Published in Transactions on Machine Learning Research (08/2024)
Table 12: Corner Case uses Playstyle Intersection Similarity (mix): consistent count = 8
Speed60NO
Speed60 (C)
Speed65 (C)
Speed70 (C)
Speed75 (C)
Speed80 (C)
NO (C)
0.8014 (0.0206)
0.7502 (0.0187)
0.7170 (0.0233)
0.6538 (0.0231)
0.5829 (0.0246)
N1 (C)
0.6927 (0.0234)
0.6865 (0.0240)
0.6646 (0.0218)
0.6254 (0.0265)
0,5508 (0.0250)
N2
0.6260 (0.0266)
0.6499 (0.0257)
0.6354 (0.0299)
0.5709 (0.0254)
0.5450 (0.0284)
N3
0.5813 (0.0282)
0.5857 (0.0250)
0.5721 (0.0302)
0.5507 (0.0276)
0.4825 (0.0321)
N4 (C)
0.5420 (0.0268) 0.5390 (0.0298) 0.5322 (0.0310) 0.4708 (0.0288)
0.4544 (0.0328)
Table 13: Corner Case uses Playstyle Jaccard Index (mix): consistent count =
Speed60NO
Speed60 (C)
Speed65
Speed70 (C)
Speed75 (C)
Speed80 (C)
NO (C)
0.0938 (0.0059)
0.0863 (0.0042)
0.0841 (0.0044)
0.0840 (0.0048)
0.0816 (0.0042)
N1
0.0845 (0.0044)
0.0853 (0.0045)
0.0833 (0.0040)
0.0821 (0.0043)
0.0815 (0.0040)
N2
0.0827 (0.0045)
0.0816 (0.0037)
0.0830 (0.0047)
0.0812 (0.0043)
0.0799 (0.0042)
N3
0.0821 (0.0040)
0.0820 (0.0048)
0.0790 (0.0042)
0.0806 (0.0040)
0.0773 (0.0049)
N4
0.0780 (0.0046) 0.0762 (0.0045) 0.0757 (0.0038) 0.0750 (0.0042) 0.0760 (0.0041)
Table 14: Corner Case uses Playstyle Similarity (mix): consistent count 9
Speed6ONO
Speed60 (C)
Speed65 (C)
Speed70 (C)
Speed75 (C)
Speed80 (C)
NO (C)
0.0753 (0.0046)
0.0650 (0.0035)
0.0608 (0.0034)
0.0555 (0.0026)
0.0472 (0.0035)
N1 (C)
0.0590 (0.0035)
0.0579 (0.0033)
0.0549 (0.0034)
0.0513 (0.0035)
0.0446 (0.0031)
N2
0.0519 (0.0036)
0.0533 (0.0033)
0.0525 (0.0033)
0.0458 (0.0032)
0.0440 (0.0034)
N3 (C)
0.0482 (0.0033)
0.0473 (0.0034)
0.0448 (0.0031)
0.0435 (0.0032)
0.0382 (0.0030)
N4 (C)
0.0421 (0.0034) 0.0413 (0.0031)
0.0402 (0.0035) 0.0349 (0.0030) 0.0337 (0.0032)
Table 15:
Center Case uses Playstyle Distance (229); consistent count = 2
Speed70N2
Speed60
Speed65
Speed70
Speed75
Speed80
NO
-0.0068 (0.0013)
-0.0068 (0.0012)
-0.0065 (0.0011) -0.0068 (0.0012)
-0.0097 (0.0017)
N1
-0.0073 (0.0011)
-0.0069 (0.0013)
-0.0073 (0.0014)
-0.0084 (0.0014)
-0.0095 (0.0014)
N2 (C)
-0.0079 (0.0010)
-0.0075 (0.0015)
-0.0069 (0.0013)
-0.0087 (0.0014)
-0.0095 (0.0016)
N3 (C)
-0.0085 (0.0012)
-0.0080 (0.0013)
-0.0075 (0.0011)
-0.0090 (0.0014)
-0.0103 (0.0013)
N4
-0.0105 (0.0013)
-0.0093 (0.0014)
-0.0094 (0.0013)
-0.0112 (0.0015)
-0.0127 (0.0016)
36

---

Published in Transactions on Machine Learning Research (08/2024)
Table 16: Center Case uses Playstyle Distance (mix): consistent count = 2
Speed70N2
Speed60
Speed65
Speed70
Speed75
Speed80
NO
-0.0033
(0.0006)
-0.0029 (0.0006)
-0.0029 (0.0006) -0.0033 (0.0005)
-0.0046 (0.0007)
N1
-0.0035 (0.0006)
-0.0032 (0.0007)
-0.0034 (0.0007)
-0.0036 (0.0007)
-0.0046 (0.0006)
N2 (C)
-0.0038 (0.0005)
-0.0035 (0.0008)
-0.0031 (0.0007)
-0.0037 (0.0006)
-0.0047 (0.0009)
N3 (C)
-0.0039 (0.0006)
-0.0036 (0.0006)
-0.0034 (0.0005)
-0.0040 (0.0007)
-0.0047 (0.0007)
N4
-0.0046 (0.0006)
-0.0037 (0.0005) -0.0040 (0.0006) -0.0046 (0.0006) -0.0060 (0.0009)
Table 17: Center Case uses Playstyle Intersection Similarity (mix): consistent count = 2
Speed70N2
Speed60
Speed65
Speed70
Speed75
Speed80
NO (C)
0.6774 (0.0241)
0.6944 (0.0206)
0.6995 (0.0233)
0.6625 (0.0246) 0.6286 (0.0265)
N1
0.6295 (0.0252)
0.6609 (0.0259)
0.6603 (0.0247)
0.6456 (0.0253)
0.6031 (0.0281)
N2 (C)
0.5962 (0.0323)
0.6376 (0.0308)
0.7100 (0.0231)
0.6086 (0.0291)
0.6034 (0.0273)
N3
0.5891 (0.0272)
0.5844 (0.0277)
0.5999 (0.0261)
0.5764 (0.0297)
0.5486 (0.0256)
N4
0.5129 (0.0306) 0.5664 (0.0274) 0.5503 (0.0278) 0.5135 (0.0297) 0.4841 (0.0253)
Table 18: Center Case uses Playstyle Jaccard Index (mix): consistent count =
Speed70N2
Speed60
Speed65
Speed70
Speed75
Speed80
NO
0.0823 (0.0046)
0.0829 (0.0047)
0.0827 (0.0046) 0.0844 (0.0043)
0.0831 (0.0051)
N1
0.0836 (0.0050)
0.0820 (0.0048)
0.0846 (0.0043)
0.0850 (0.0045)
0.0855 (0.0050)
N2 (C)
0.0821 (0.0050)
0.0825 (0.0043)
0.0930 (0.0049)
0.0845 (0.0042)
0.0842 (0.0045)
N3
0.0806 (0.0042)
0.0827 (0.0044)
0.0795 (0.0049)
0.0839 (0.0043)
0.0780 (0.0045)
N4
0.0802 (0.0046) 0.0792 (0.0046)
0.0799 (0.0048) 0.0782 (0.0047) 0.0804 (0.0050)
Table 19:
Center Case uses Playstyle Similarity (mix): consistent count = 3
Speed70N2
Speed60
Speed65
Speed70
Speed75
Speed80
NO (C)
0.0554 (0.0040)
0.0573 (0.0039)
0.0579 (0.0043)
0.0568 (0.0034)
0.0522 (0.0037)
N1 (C)
0.0527 (0.0036)
0.0555 (0.0041)
0.0560 (0.0035)
0.0549 (0.0038)
0.0518 (0.0037)
N2 (C)
0.0486 (0.0038)
0.0522 (0.004)
0.0652 (0.0043)
0.0512 (0.0035)
0.0498 (0.0035)
N3
0.0486 (0.0033)
0.0487 (0.0035)
0.0479 (0.0037)
0.0485 (0.0038)
0.0438 (0.0030)
N4
0.0417 (0.0033)
0.0452 (0.0033)
0.0444 (0.0037)
0.0407 (0.0035)
0.0399 (0.0035)
37

---

Published in Transactions on Machine Learning Research (08/2024)
Table 20: Averaged diverse trajectory count of different DRL algorithms across
Atari games in 25 episodes.
DRL Algorithm
Asterix
Breakout
MsPacman
Pong
Qbert
Seaquest
Spaceluvaders
DQN
6.00
6.00
5.33
4.00
6.00
11.00
25.00
C51
6.00
7.00
7.00
6.00
7.00
21.00
25.00
Rainbow
8.67
5.33
8.00
5.00
5.00
24.00
25.00
IQN
25.00
14.00
10.00
9.00
10.00
25.00
25.00
A.5
More Experiments Diversity Measurement in DRL
This section introduces additional experiments demonstrating the application of our method for quantifying docision-making diversity, detailed in Algorithm [ We analyze the Atari DRL agent dataset from the main paper, which includes 5 models per DRL algorithm, each contributing 5 trajectories, totaling 25 trajectories per algorithm Mnih et al. 2015 Bellemare et al. 2017 Hessel et al. 2018 Dabney et al. 2018). Using
Algorithm [ we assess the diversity of trajectories produced by cach algorithm. Results in Table 20 averaged across three discrete encoder models with a similarity threshold of = 0.2, show the capacity of models trained under the same DRL algorithm to generate diverse game episodes within 25 attempts. The TQN algorithm displays higher diversity across games, consistent with its risk sampling feature.
We also apply various levels of stochasticity using the Dopamine framework to illustrate another use case of this measure. Our focus is on the first TQN model from Dopamine, which is expected to adapt to wide array of playstyles due to its risk functions and robust performance capabilities. To encourage diversity, we employ the Boltzmann distribution, commonly known as softmax, varying the temperature (z) to infiuence decision- decision-making:
A(ss)
((s) Softmax
(20)
This approach, inspired by Fan & Xiao 2022 uses the advantage function (A) crucial in action selection in reinforcement learning, where a higher advantage indicates a preferable action.
We explore four levels of randomness (a E {0.0001, 0.001, 0.01, anticipating increased diversity with greater randomness. We test similarity thresholds (1) of 0.5, 0.2, and 0.05 across seven Atari games with
100 trajectories cach. Results illustrated through shaded curves in Figures from Figure 14 to Figure 16 demonst,rtinng the elficacy of our diversity measure.
Notably, games like Seaquest (Figure 17b) exhibit high diversity even at lower randomness levels, indicating intrinsic complexity in terms of playstyles. In contrast, Qbert (Figure 17a)) becomes more monotonous since the goal is to achieve a higher score the puzzle game. This observation suggests another application for our measure: identifying the complexity of game content. The time complexity of Algorithm [ is O(N²), given the number of trajectories N. Future rescarch could investigate more efficient methods, perhaps leveraging approximations or advanced data structures for quicker similarity checks.
The algorithm we introduce for diversity quantification shifts our understanding of gaming from the subjective to the quantitative. While various methodologies for measuring diversity exist in different domains, our approach is particularly apt to video game playing. In addition, recognizing and quantifying this diversity can inform the development of more adaptive DRL. models, thereby addressing specific challenges gaming and artificial intelligence. This new measure contributes to our progress toward models that are not only efficient but also demonstrate variety of adaptable strategies, opening up vast avenues for future research.
A.6 Game 2048 Experiment Details
We employed temporal difference learning to train our 2048 agents, utilizing training code available on GitHuty We set the learning rate (a) to 0.01 and maintained all other default settings. As this study does
httpp::/ggthuu..o//pppggggg
38

---

Published in Transactions on Machine Learning Research (08/2024)
Asterix = 0.5)
Breakout (t 0.5]
100
100
Overe Tibecc Cont
=
Dowe Thaecary Cunnt
0.01
z 0.1
0.1
40
40
0
0
20
40
60
80
100
20
40
60
80
100
Trajectory Court
Trajedory Count
(a) Asterix
(b) Breakout
MsPacman 0.5)
Pong
100
100
z= 0.0001
z= 0.000001
80
= 0000
80
2 0.001
Dveee Tabcooy cont
z-
Dieene T-tecury Count
0.00
z- 0.1
0.1
20
20
0
60 Trajectrry
0
(c)
MsPacman
(d) Pong
Obert =t= 0.5)
Seaquest (1 0.5)
100
100
z 0.00001
0.00001
Divess Traectry Coutt
Dhene Traledoy Coutt
z= 0.1
0.1
60
60
40
0
20
60
100
20
40
60
80
100
Coun:
Truaddery Count
Qbert
(f) Seaquest
Spaceinvaders (t= 0.5)
100
0.0001
80
0 0.001
z- 0.01
2 0.1
40
20
0
20
40 Tajectory Count
80
100
(g) Spacelnvaders
Figure 14: Diversity Measurement of IQN Models in Seven Atari Games (t = 0.5). The shaded area indicates the range between min and max accuracy among three encoder models.
39

---

Published in Transactions on Machine Learning Research (08/2024)
Asterix (t= 0.2)
Breakout (t 0.2]
100
100
Overe Taeeca Cont
=
Dowe Tiaecary Cuntt
0.01
z 0.1
0.1
40
40
0
0
20
40
60
80
100
20
40
60
80
100
Trajectory Court
Trajecoryy Count
(a) Asterix
(b) Breakout
MsPacman 0.2)
Pong
100
100
z= 0.0001
2= 0.000001
80
= 0000
80
2 0.001
Dveee Tabcooy cont
z-
Dieene T-secury Count
0.00
z- 0.1
0.1
20
20
0
60 Trajectory
0
(c)
MsPacman
(d) Pong
Obert =t= 0.2)
Seaquest (1 0.2)
100
100
z 0.00001
0.00001
Divess Traectry Coutt
Dhene Traledoy Coutt
z= 0.1
0.1
60
60
40
0
0
20
60
100
20
40
60
80
100
Coun:
Truaddery Count
Qbert
(f) Seaquest
Spaceinvaders (t= 0.2)
100
0.0001
80
0 0.001
0.01
2 0.1
40
20
0
20
40 Tajectory Count
80
100
(g) Spacelnvaders
Figure 15: Diversity Measurement of IQN Models in Seven Atari Games (t 0.2). The shaded area indicates the range between min and max accuracy among three encoder models.
40

---

Published in Transactions on Machine Learning Research (08/2024)
Asterix (t = 0.051
Breakout (t = 0.05)
100
100
Overe Fieeca Cont
=
Dowe Thaecary Cunnt
0.01
z 0.1
0.1
40
40
00
0
20
40
60
80
100
20
40
60
80
100
Trajectory Court
Trajedory Count
(a) Asterix
(b) Breakout
MsPacman (t 0.051
(t-
100
100
0100001
z= 0.000001
80
= 0001
80
0.001
Dveee Tabcooy Conn:
z-
Dieene T-secury Count
0.00
z- 0.1
0.1
20
20
0
60
100
00
Trajectory
Trajedory
(c)
MsPacman
(d) Pong
Obert [t= 0.05)
Seaquest (t= 0.05)
100
100
0.0001
0.0000
Divoss Traeddry Doutt
Dhene Traledoy Coutt
0.1
0.1
60
60
40
0
0
20
60
100
20
40
60
80
100
Coun:
Truaddery Count
Qbert
(f) Seaquest
Spacelnvaders (t- 0.05)
100
0.0001
80
0 0.001
0.01
2 0.1
40
20
0
20
40 Tajectory Count
80
100
(g) Spacelnvaders
Figure 16: Diversity Measurement of IQN Models in Seven Atari Games (t 0.05). The shaded area indicates the range between min and max accuracy among three encoder models.
41

---

Published in Transactions on Machine Learning Research (08/2024)
PLAVER 0000
ROUND:
220
OXYGEN
RETI-ISiON
(a) Qbert: A 2D puzzle game where your goal is to change every cube a pyramid to a target color. To do this, control the on-screen character, Q*bert, and make jumps on top of the cube, avoiding obstacles and enemies.
(b) Seaquest: A 2D survival shooting game. The player sails submarine to shoot at sharks and enemy submarines to rescue divers swimming in the water.
Figure 17: Game screens of Qbert and Seaquest.
not focus optimizing discrete representations, directly the full board as the discrete states for playstyle analysis. The potential state space for the 4 x 4 version of 2048 is calculated to be 1616.
A.7 Go Experiment Details
Our Go experiment follows the standard 19x19 rules with common observations, including the latest eight board views and player color, resulting in 18 channels (Silver et al. 2018 Wu et al. 2024 The action space includes 362 discrete actions (361 move actions and pass).
Figure 18 shows the variant HSD we used. The primary difference from the implementation in Playstyle Distance is that the reconstruction head for the auloencoder is replaced by value head. Additionally, we use three levels of hierarchy with different numbers of embedding candidates: 256 candidate embeddings at the base level, 16 candidate embeddings the next level, and candidate embeddings the final level. The smallest state space is 4 65536, which ensures sufficient intersection states even with only few game records. The datasets used for training the encoder and playstyle identification are private datasets provided by the MiniZero framework team (Wu et al.) 2024). If you need these datasets to reproduce our results on Go, please contact them directly for authorization.
We train the encoder with batch size 1024 over 100 iterations, each iteration including 1000 network updates with the Adam optimizer. The lcarning rate starts 0.00025 and lincarly docays 0 according the iteration number. The coefficient 3 in the vector quantization process is the commonly suggested 0.25 (van den Oord et al. 2017 Lin et al. 2021). The loss function for the policy head is cross-entropy, and the loss for the value head is mean square error, with the loss coefficients of these two heads both set to 1.
Tables 29 list the detailed accuracies for different query and candidate sizes using only the first 10 moves of games and using the full game moves as mentioned Section 6.2 For the first 10 moves case, it is common and straightforward in board games that a preferred opening a kind of playstyle and using a small state space can achieve 97.0% accuracy ([15), Table 21). However, when we do not specifically focus the playstyles in the opening, may want to all game moves capture any possible playstyles, and
42

---

Published in Transactions on Machine Learning Research (08/2024)
Observation [18, 19, 19]
Kernel Size Padding 1, ReLU, Hidden=256
CNN Layer BN
CNN Layer BN + ReLU
Kernel Size 3, Padding 1, ReLU, Hidden=256
Resiciaal Block
Layer
x3
Kernel Size 3, Padding 1, TanH, Hidden=32
Sum RetU
CNN Layer
Stop Gradiont
StepiGradient
FC Layer #ReLU (256)
FC Layer RReLU (128)
VQL (32.19,19)
Weighted Sum
Layer (32"8)
Layer (16*8)
vQ, [32.8]
VQ [16,8)
Weighted Sum
Policy FC
Layer (362)
Layer
FC Layer RR (256)
Layer RReUU [126]
Layer
Layer (2566
(32*19*19)
Figure 18: The neural network architecture of the HSD encoder for Go.
Table 21: Accuracy of 200 human Go player identification with the Playstyle Distance (4%)
First 10 Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
1.5%
2.5%
3.0%
4.0%
4.0%
7.5%
8.5%
Candidate
3.0%
11.5%
20.5%
29.0%
40.0%
51.0%
50.0%
Candidate 10
7.5%
24.5%
33.5%
49.0%
57.0%
64.5%
72.0%
Candidate 25
10.5%
32.0%
49.5%
71.0%
79.0%
87.5%
89.5%
Candidate 50
17.5%
44.0%
59.5%
76.0%
87.0%
92.0%
95.5%
Candidate 75
17.5%
48.5%
65.0%
80.5%
88.5%
93.0%
96.5%
Candidate 100
19.5%
48.5%
66.5%
81.0%
89.0%
94.0%
97.0%
Full Game Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
2.5%
3.0%
5.5%
3.0%
5.5%
3.5%
3.0%
Candidate
5.0%
5.5%
8.0%
6.0%
10.0%
11.5%
10.0%
Candidate 10
7.0%
9.5%
11.5%
10.5%
11.0%
11.0%
10.5%
Candidate 25
9.5%
13.5%
16.0%
20.0%
20.5%
20.5%
21.5%
Candidate 50
10.0%
22.5%
30.0%
32.0%
31.0%
31.5%
33.0%
Candidate 75
13.5%
30.0%
38.0%
42.0%
43.0%
39.5%
39.5%
Candidate 100
16.5%
33.0%
40.0%
47.0%
50.0%
46.0%
50.0%
such a small state space negatively impacts the accuracy. Some boards cannot be compared in this kind of playstyle but the small state space cannot provide information to separate these cases. Instead, large state spaces like {16*} {256361} can handle this kind of problem. If we do not know which state space can give the best result, our multiscale state space is good choice that leverages all state spaces. Even if it may be influenced by a very bad stale space like {45} in the full games case, the damage is limited. Also, discrete playstyles incorporating the Jaccard index (Playstyle Similarity and its variants) can further cover this weakness. Additionally, the Bhattacharyya cocfficient is more effective when a playstyle plays very diversely state, and our Go result is an example. For slightly different playstyles like TORCS with continuous actions, using the scaled W2 metrie provide better accuracy.
43

---

Published in Transactions on Machine Learning Research (08/2024)
Table 22: Accuracy of 200 human Go player identification with the Playstyle Distance (16%).
First 10 Moves
Query
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
2.0%
3.5%
4.0%
5.0%
3.0%
5.5%
5.5%
Candidate
4.5%
10.0%
13.5%
27.5%
38.0%
47.0%
52.5%
Candidate 10
5.0%
23.0%
41.5%
62.5%
65.0%
72.5%
79.0%
Candidate 25
11.5%
43.0%
58.5%
74.0%
83.5%
86.5%
90.5%
Candidate 50
18.0%
48.0%
66.0%
78.5%
85.0%
91.5%
94.0%
Candidate 75
21.0%
54.5%
67.0%
81.5%
90.0%
92.5%
95.0%
Candidate 100
20.5%
53.5%
69.5%
82.0%
91.0%
93.0%
95.5%
Full Game Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
2.0%
3.5%
4.0%
5.5%
4.0%
5.5%
5.0%
Candidate
4.5%
10.0%
14.0%
28.5%
36.5%
46.5%
49.5%
Candidate 10
5.0%
23.5%
44.5%
61.0%
67.5%
71.0%
73.5%
Candidate 25
11.5%
43.0%
59.0%
74.0%
80.0%
84.5%
88.5%
Candidate 50
18.0%
49.5%
66.0%
79.0%
86.0%
91.5%
94.5%
Candidate 75
20.0%
53.0%
70.0%
82.0%
91.5%
93.5%
95.5%
Candidate 100
20.0%
53.0%
72.0%
83.5%
91.0%
95.5%
96.5%
Table 23: Accuracy of 200 human Go player identification with the Playstyle Distance (256361)
First 10 Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
2.0%
4.5%
5.5%
5.5%
5.0%
6.0%
6.0%
Candidate
5.0%
8.0%
10.0%
23.5%
35.0%
42.5%
50.0%
Candidate 10
5.5%
24.5%
38.0%
61.5%
65.0%
72.5%
76.0%
Candidate 25
11.5%
43.0%
58.5%
75.0%
80.0%
87.5%
90.0%
Candidate 50
17.0%
50.0%
63.5%
80.0%
87.0%
91.5%
93.5%
Candidate 75
20.0%
54.5%
68.0%
83.0%
89.0%
92.0%
95.0%
Candidate 100
20.5%
55.0%
67.0%
82.0%
89.5%
92.0%
94.5%
Full Game Moves
Query
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
2.0%
4.5%
5.5%
5.5%
5.0%
6.0%
5.5%
Candidate
5.0%
8.0%
10.0%
23.5%
35.5%
42.5%
49.5%
Candidate 10
5.5%
25.0%
39.5%
62.0%
65.0%
73.0%
76.5%
Candidate 25
11.5%
44.0%
60.0%
75.5%
80.5%
88.5%
90.5%
Candidate 50
17.0%
50.5%
65.0%
81.5%
88.5%
93.0%
94.5%
Candidate 75
20.0%
56.5%
70.0%
84.5%
90.5%
93.5%
96.5%
Candidate 100
21.0%
57.0%
69.5%
84.0%
91.0%
94.0%
96.5%
44

---

Published in Transactions on Machine Learning Research (08/2024)
Table 24: Accuracy of 200 human Go player identification with the Playstyle Distance (mix).
First 10 Moves
Query
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
1.5%
3.0%
2.5%
5.5%
6.5%
7.5%
9.5%
Candidate
4.5%
14.0%
27.5%
42.0%
50.0%
59.0%
65.5%
Candidate 10
7.5%
30.5%
46.5%
59.5%
69.0%
77.5%
79.5%
Candidate 25
16.5%
45.0%
62.0%
75.5%
86.5%
91.5%
93.0%
Candidate 50
20.0%
49.5%
70.0%
80.5%
87.5%
92.5%
96.5%
Candidate 75
21.0%
54.5%
72.5%
84.5%
90.5%
94.5%
97.0%
Candidate 100
25.0%
57.5%
72.5%
85.0%
91.5%
93.5%
96.5%
Full Game Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
2.0%
3.5%
7.5%
7.5%
11.0%
10.5%
10.0%
Candidate
6.5%
17.0%
20.5%
30.5%
34.0%
41.0%
41.0%
Candidate 10
11.0%
24.5%
33.0%
39.0%
43.0%
49.0%
53.0%
Candidate 25
13.5%
36.5%
46.0%
56.5%
62.0%
63.0%
69.5%
Candidate 50
17.0%
45.0%
56.0%
70.5%
76.0%
77.5%
81.0%
Candidate 75
20.0%
49.0%
65.0%
75.5%
79.5%
83.0%
86.5%
Candidate 100
22.0%
53.0%
65.5%
75.0%
81.0%
85.0%
90.0%
Table 25: Accuracy of 200 human Go player identification with the Playstyle Intersection Similarity (mix).
First 10 Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
2.5%
3.0%
3.5%
5.5%
5.5%
8.0%
1.15%
Candidate
5.0%
15.0%
28.5%
39.0%
46.5%
54.0%
57.5%
Candidate 10
8.5%
29.0%
42.0%
54.0%
60.0%
71.0%
76.5%
Candidate 25
14.5%
41.5%
56.0%
68.5%
75.0%
85.5%
91.5%
Candidate 50
18.0%
48.0%
63.0%
75.5%
84.0%
89.0%
95.0%
Candidate 75
21.5%
51.5%
65.0%
79.0%
88.0%
94.5%
95.0%
Candidate 100
23.5%
56.5%
66.0%
80.5%
89.0%
93.0%
94.5%
Full Game Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
3.0%
3.0%
7.0%
7.5%
13.0%
16.0%
20.0%
Candidate
7.0%
19.0%
27.0%
33.0%
44.0%
52.0%
55.5%
Candidate 10
12.5%
27.0%
38.0%
46.0%
56.5%
66.0%
72.0%
Candidate 25
19.0%
40.0%
53.5%
66.0%
80.0%
84.5%
85.5%
Candidate 50
18.5%
47.0%
61.0%
76.0%
86.0%
90.5%
94.5%
Candidate 75
21.0%
50.5%
64.0%
79.5%
91.0%
95.0%
96.0%
Candidate 100
26.5%
55.5%
67.5%
81.0%
92.0%
93.5%
95.0%
45

---

Published in Transactions on Machine Learning Research (08/2024)
Table 26: Accuracy of 200 human Go player identification with the Playstyle Int BC Similarity (mix).
First 10 Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
2.5%
2.5%
3.0%
6.5%
6.5%
9.5%
12.0%
Candidate
5.5%
14.0%
26.0%
40.5%
48.5%
58.5%
61.5%
Candidate 10
9.0%
30.0%
42.5%
53.5%
66.5%
76.0%
78.0%
Candidate 25
16.0%
46.0%
60.5%
72.0%
81.5%
89.5%
93.5%
Candidate 50
19.5%
55.0%
67.5%
81.0%
89.0%
93.0%
95.5%
Candidate 75
23.0%
57.5%
72.5%
85.0%
91.5%
96.5%
96.0%
Candidate 100
28.5%
61.0%
76.5%
86.0%
92.5%
94.5%
95.5%
Full Game Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
2.5%
4.0%
7.5%
8.5%
14.0%
15.0%
18.5%
Candidate
7.5%
16.5%
26.0%
34.5%
41.5%
49.0%
51.0%
Candidate 10
12.5%
27.0%
37.5%
45.5%
58.0%
64.0%
70.5%
Candidate 25
17.0%
39.5%
55.0%
66.0%
79.5%
83.5%
87.5%
Candidate 50
19.5%
48.5%
65.0%
78.0%
90.0%
92.5%
95.0%
Candidate 75
24.5%
60.5%
72.5%
83.0%
92.5%
96.0%
97.5%
Candidate 100
30.0%
63.0%
76.5%
85.0%
94.5%
95.5%
97.0%
Table 27: Accuracy of 200 human Go player identification with the Playstyle Jaccard Index (mix).
First 10 Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
3.0%
7.5%
9.5%
11.0%
14.5%
16.0%
19.0%
Candidate
7.5%
9.5%
15.0%
20.5%
25.0%
29.0%
32.5%
Candidate 10
11.5%
13.5%
21.5%
35.0%
42.5%
48.0%
48.5%
Candidate 25
14.5%
28.0%
37.0%
49.0%
67.0%
72.0%
76.5%
Candidate 50
17.0%
34.5%
42.5%
60.5%
78.0%
80.5%
86.0%
Candidate 75
18.5%
40.5%
53.0%
69.0%
80.5%
88.0%
90.0%
Candidate 100
22.0%
46.0%
53.5%
72.0%
83.5%
90.5%
95.0%
Full Game Moves
Query
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
3.0%
4.0%
3.0%
3.5%
3.5%
2.5%
3.0%
Candidate
3.0%
5.0%
3.0%
5.0%
6.0%
7.0%
10.0%
Candidate 10
4.0%
6.5%
8.5%
10.5%
15.5%
12.0%
14.0%
Candidate 25
9.0%
12.5%
15.5%
22.5%
23.5%
24.5%
25.0%
Candidate 50
8.5%
18.5%
25.5%
40.0%
51.0%
53.5%
51.0%
Candidate 75
13.0%
21.5%
29.0%
46.0%
59.0%
66.0%
68.5%
Candidate 100
16.0%
23.0%
30.0%
45.0%
65.5%
72.0%
80.5%
46

---

Published in Transactions on Machine Learning Research (08/2024)
Table 28: Accuracy of 200 human Go player identification with the Playstyle Similarity (mix).
First 10 Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
3.0%
9.0%
12.5%
18.0%
21.0%
24.5%
28.5%
Candidate
11.0%
21.5%
28.0%
38.0%
49.0%
54.0%
55.5%
Candidate 10
16.0%
28.5%
40.0%
52.0%
62.0%
66.0%
70.0%
Candidate 25
23.0%
42.0%
58.5%
65.0%
81.5%
86.0%
89.0%
Candidate 50
24.0%
48.5%
61.0%
73.5%
87.5%
89.5%
92.5%
Candidate 75
27.0%
57.0%
68.5%
80.0%
88.5%
94.0%
95.5%
Candidate 100
30.0%
55.5%
69.5%
79.5%
90.0%
95.0%
97.0%
Full Game Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
4.5%
9.0%
11.5%
11.5%
10.5%
13.5%
10.0%
Candidate
8.0%
15.5%
17.5%
21.0%
28.5%
29.5%
31.0%
Candidate 10
12.0%
23.0%
29.5%
39.0%
44.5%
44.0%
46.0%
Candidate 25
16.0%
31.5%
44.5%
56.5%
66.5%
69.5%
72.0%
Candidate 50
19.0%
38.5%
58.5%
73.0%
81.5%
83.5%
85.0%
Candidate 75
23.5%
46.5%
56.5%
76.5%
87.0%
90.5%
93.0%
Candidate 100
27.0%
46.5%
55.5%
77.5%
88.5%
93.5%
94.0%
Table 29; Accuracy of 200 human Go player identification with the Playstyle BC Similarity (mix).
First 10 Moves
Query 1
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
3.0%
9.0%
13.0%
20.0%
23.5%
28.0%
33.0%
Candidate
12.0%
24.0%
36.0%
43.0%
49.5%
57.0%
58.5%
Candidate 10
18.5%
32.5%
45.5%
55.0%
66.0%
72.0%
75.0%
Candidate 25
22.0%
45.5%
63.5%
70.5%
83.0%
91.0%
93.0%
Candidate 50
27.0%
55.5%
68.0%
76.0%
88.0%
92.5%
95.5%
Candidate 75
29.5%
63.5%
74.5%
83.5%
90.0%
96.5%
97.0%
Candidate 100
31.5%
64.5%
74.5%
83.0%
92.0%
95.5%
97.0%
Full Game Moves
Query
Query 5
Query 10
Query 25
Query 50
Query 75
Query 100
Candidate
4.0%
9.5%
13.5%
21.0%
26.0%
29.0%
33.5%
Candidate
13.5%
22.0%
33.0%
42.0%
54.0%
57.5%
61.5%
Candidate 10
18.5%
32.0%
45.5%
54.5%
66.5%
73.5%
76.5%
Candidate 25
23.0%
46.5%
61.0%
73.0%
83.5%
91.5%
92.5%
Candidate 50
28.0%
54.5%
70.5%
79.5%
92.0%
94.5%
96.0%
Candidate 75
30.0%
62.5%
74.5%
85.5%
93.0%
97.5%
97.5%
Candidate 100
35.0%
62.0%
78.0%
87.5%
95.0%
97.0%
97.5%
47

---

Published in Transactions on Machine Learning Research (08/2024)
A.8 Statistical Significance of Experiment Results
In this appendix, present the stalistical lests conducted to validate some of the claims made in our main paper. While accuracy is a common metric for evaluating the performance of classification models in machine learning, it is essential to establish that the observed differences are not due to sampling uncertainty. To this end, we employed McNemar's test [McNemar] 1947) for statistical significance. This non-parametric test particularly suitable for our paired classification scenarios with binary outcomes correct or wrong predictions allowing us compare whether the prediction correctness of two measures is statistically significant.
We applied McNemar's test to several comparisons in our study:
Playstyle Distance with multiscale state space can have better accuracy than using any state space alone on the TORCS playstyle dataset.
2. When using the same state space (2 intersection samples, probabilistic similarity is not worse Ihan distance similarity RGSK and slightly better on TORCS.
Jaccard index can hurt accuracy in some extreme cases, such as in 2048 with the raw board as discrete states.
4. For Go playstyle with full game moves, Playstyle BC Similarity has better accuracy than Playstyle Similarity.
The following subsections provide detailed results of these tests, including the p-values, which indicate whether the observed improvements are statistically significant. Each subsection corresponds to a specific comparison, outlining the methodology and presenting the findings. For defining statistical significance, refer to p-value 0.05 the default threshold suggested by Fisher (1970), since there is standard p-value threshold for classification lasks in machine learning.
A.8.1 Multiscale State Space on TORCS
In this subscction, we examine the performance of the multiscale state space in improving accuracy on the TORCS playstyle dataset. We conducted McNemar's test to determine if the multiscale approach offers statistically significant improvements over using individual state spaces alone. The corresponding section in the main paper is Scction 5.1
Since we use random subsampling in our experiments, we reran another 100 rounds of random subsampling, recording the contingency table of different state space settings with the same samples. In other words, the dataset sampled in each subsampling round was used in all state space settings rather than resampling for different settings. The accuracies are slightly different from those in Table 2 but the effectiveness of improvement. with the multiscale state space remains consistent in Table 30 The mix version (multiscale) has better accuracy under the first two encoders and is slightly worse than 22 t-2 with a p-value of 0.179. However, using the same (=2 on the multiscale state space still shows clear accuracy improvement. Overall, using t = 1 or t = 2 is not much different in TORCS; thus, simpler setting with t = 1 (no filtering) for filtering intersection states is what we recommend.
A.8.2 Using Perceptual Similarity as an Alternative to Distance Similarity
Here, investigate the efficacy of using perceptual similarity ity measures. We applied McNemar's test to compare the performance of these two approaches on the TORCS and RGSK datasets, particularly focusing on whether probabilistic similarity can offer equal or better performance. The corresponding section in the main paper is Section 5.2
alternative to distance-based similar-
We reran 100 rounds of random subsampling with a datasct size of 1024 and used the same samples for comparing two variants of probabilistic similarity. The results in Table 31 show clear improvements in accu- racy over Playstyle Distance with both Playstyle Intersection Similarity and Playstyle Inter BC Similarity.
48

---

Published in Transactions on Machine Learning Research (08/2024)
Table 30: Statistical test on multiscale state space in TORCS.
Discrete State Space
p-value
Accuracy Change
+ mix t-1
Encoder1
1.947e-142
37.3%
73.8%
Encoder2
1.021e-191
34.4%
78.0%
Encoder3
4.166e1766
36.6%
77.9%
220 t=2 + mix t=1
Encoderl
0.002
70.7%
73.8%
Encoder2
4.876c-8
72.2%
78.0%
Encoder3
0.179
77.9% 79.2%
220 + mix 4.285e-34
Encoderl
62.8%
73.8%
Encoder2
3.227c-29 29
68.2%
78.0%
Encoder3
2.080e-36
67.4%
77.9%
256 t=2 + mix t=1
Encoderl
<1c-323
8.3% + 73.8%
Encoder2
<le3223
4.9%
78.0%
Encoder3
< 1e-323
4.4% + 77.9%
256 (=1 + mix [=1
Encoder1
1.577e-46
54.9% 73.8%
Encoder2
5.728e-44
60.8%
78.0%
Encoder3
1.716c-16
67.2% 77.9%
mix t=2 + mix t=1
Encoder1
0.966
73.8% 73.8%
Encoder2
0.318
77.0%
78.0%
Encoder3
1.360e-7
82.7% 77.9%
49

---

Published in Transactions on Machine Learning Research (08/2024)
Table 31: Statistical test on perceptual similarity in TORCS and RGSK.
TORCS
p-value
Accuracy Change
Playstyle Distance + Playstyle Intersection Similarity
Encoderl
2.186e-53
62.4%
79.1%
Encoder2
2.180c-46
69.4% + 84.3%
Encoder3
2.655e-70
67.0% + 85.1%
Playstyle Distance - Playstyle Inter BC Similarity
Encoder1
2.232c-137
62.4%
92.8%
Encoder2
3.040e-28
69.4% + 82.3%
Encoder3
1.222e-91
67.0% + 90.7%
RGSK
p-value
Accuracy Change
Playstyle Distance - Playstyle Intersection Similarity
Encoder1
7.093e-23
92.5% + 98.0%
Encoder2
0.001
97.1% + 98.2%
Encoder3
3.974e-44
90.3%
99.0%
Playstyle Distance - Playstyle Inter BC Similarity
Encoderl
3.589e-19
92.5% + 97.7%
Encoder2
9.41le-8
97.1% + 98.8%
Encoder3
5.327e-35
90.3% + 98.1%
Table 32: Statistical test on the game 2048 model identification.
p-value
Accuracy Change
Playstyle Distance + Playstyle Intersection Similarity
0.012
98.90% + 98.52%
Playstyle Distance + Playstyle Inter BC Similarity
1.0
98.90% + 98.90%
Playstyle Distance + Playstyle Jaccard Index
< 1e-323
98.90% - 49.22%
Playstyle Distance + Playstyle Similarity
1.072c-295
98.90%
- 71.26%
Playstyle Distance - Playstyle BC Similarity
1.072e-295
98.90%
71.26%
It is also clearer with tables than plots in Figure g that perceptual similarity can provide better accuracy on RGSK.
A.8.3 The Potential Limitation of the Jaccard Index for Playstyle
In this part, explore the potential limitations of using the Jaccard index in playstyle measurement. Specifically, we assess its impact on accuracy in the game 2048, using McNemar's test to highlight cases where the Jaccard index may negatively affect performance. The corresponding section in the main paper is Section 6.1
The results in Table 32 show that playstyle measures including the Jaccard index have clear accuracy reduction compared to the Playstyle Distance baseline, with the p-values demonstrating statistical signifi- cance.
A.8.4 The Recommendation of Using Playstyle BC Similarity Go
We conclude with an analysis of the Go playstyle dataset in Table 33 demonstrating the benefits of using Playstyle BC Similarity over Playstyle Similarity. McNemar's test was applied to confirm the statistical significance of the observed improvements in accuracy with the BC variant are not due random sampling in the full game moves case. The corresponding section in the main paper Section 6.2 With small samples, like using only game record as dataset, it is likely that these two measures have the performance.
50

---

Published in Transactions on Machine Learning Research (08/2024)
Table 33: Statistical test on the 200 player identification with M games as the query set and also M games as the candidate set. We compare the efficacy between Playstyle Similarity and Playstyle BC Similarity the full game moves case.
Playstyle Similarity (mix) - Playstyle BC Similarity (mix)
p-value
Accuracy Change
M=1
0.564
4.5% + 4.0%
M=5
0.005
15.5% + 22.0%
M=10
1.333e-5
29.5% + 45.5%
M=25
2.553e-7
56.5% + 73.0%
M=50
2.566c-4
81.5% + 92.0%
M=75
0.002
90.5%
+ 97.5%
M=100
0.052
94.0%
97.5%
Conversely, when we have sufficient samples, like 100 games as dataset, the p-value is slightly over 0.05, implying a 5.2% probability that Playstyle Similarity and Playstyle BC Similarity have nearly the same performance. In other sample sizes, we have enough confidence in claiming Playstyle BC Similarity has better performance than Playstyle Similarity due the corresponding p-values being less than 0.05 and showing accuracy improvements.
51


Reviews:
[{"review_id": "2QZMWVVD5M", "reviewer": "Reviewer_ycj8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# strengths\n- Evaluations include a wide variety of games (two racing games, Atari games, 2048, and Go) and sources of policies (human and bot-generated, including policies trained using reinforcement learning for varying amounts of time).\n- Tests check both deterministic (or nearly so) and non-deterministic environments.\n- Strong evaluation methodology: ablates the core algorithm to show the importance (or lack thereof) of algorithm additions.\n- Good performance depends on few trajectories (~256-512 for racing games; ~64-128 for Atari).\n\n\n# weaknesses\n- In general I found myself losing the narrative of the paper and having to go back to get the big picture. The submission would benefit from some overview of the main outcomes and conclusions. There is some of this for the methodological changes, but the narrative threads get lost in the evaluations.\n- The experiments did not include results from multiple seeds (in at least some cases) and lack any reporting of confidence intervals, or variability of results. No statistical tests of differences among conditions were reported. While many differences are large from visual inspection (some of the figures), many are not so large as to be obviously different. Tables do not report any variability. These points should be addressed to clarify the magnitude of improvements from the new techniques.\n- The Go state description is difficult to follow. It would benefit from a figure and revised text discussion.\n- Most discussion of the assessment of diversity using the metric is in the appendix. The submission would benefit from being longer to push this into the main text.\n- Many changes to the baseline algorithm have only modest effects on classification accuracy. For example, the racing game results only need to add intersection. The paper would benefit from more clearly summarizing in which games (or policy sources) the changes helped and which they did not. At least some modifications were clearly helpful, but I found myself doing a lot of work to piece together the big picture of which were the minimal changes to get the biggest improvements (and when).\n- Minor stylistic note: the paper sometimes references a variable like \"N\" ($Style_n$, $N$ first moves in Go, MxM games query) without explaining this will be varied in the experiments. That was confusing and took a while to sort out. It would help to make that clearer up front.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission concerns the question of quantifying the similarity of behaviors defined as trajectories of state-action sequences generated by some policy (human or synthetic). The technical work extends a prior trajectory-based metric that identifies similar trajectories using a cache of prior examples. The new technical extensions are:\n1. Reweighting samples using an exponential weighting scheme to more closely match human similarity perception scaling properties.\n2. Adjustments to the base metric to use a hierarchy of different state granularities and to use the Jaccard index (instead of a simple intersection).\n\nThe experiments provide evidence to support these contributions by comparing the baseline metric definition to extensions that incorporate the state hierarchy, perceptual reweighting, and variations thereof on the task of classifying a new trajectory to match prior playstyles (that is, to trajectories generated by the same policy). This shows success in racing games using either human or AI-generated traces and success in Atari with AI-generated traces. Additional experiments assess the robustness of the metric (and Jaccard index) when used on games with high stochasticity due to the game environment (2048) or due to opponent behavior (Go).\n\nThe submission contributes new knowledge of how to define similarity metrics on trajectories that more accurately identify the generating policy, specifically:\n1. That the Jaccard index is a good proxy for many games but fails under high stochasticity.\n2. That perceptual reweighting (inverse exponential scaling) can improve accuracy in many cases.\n3. That a single sample threshold is typically sufficient for filtering trajectories, where prior evaluations suggested a higher threshold may be necessary for good performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# critical\n- Add results on multiple trials for the experiments and statistical testing to support any claims made about differences among methods. This should include quantification of variability and effect sizes of ablations / different algorithms.\n- Provide a narrative overview of the experiments and their results that clearly articulates how important different components of the proposed metric are and under which conditions.\n\n\n# strengthen\n- Provide a figure to illustrate the Go state encoding scheme (even if only in the appendix). Try to clarify the text to make it easier for readers to understand.\n- Section 4.2\n\t- It would help to have a table that summarized the experiment configuration parameters. Particularly include at least the number of styles for the game and the state and action space size.\n- Equation 3\n\t- Is $D(\\pi_X, \\pi_Y)$ here $B_D$?\n- Section 5.1\n\t- This section would benefit from including results on Atari games as well. The evidence in Table 1 does not support the notion that a sample threshold is necessary (see Table 1 below) and the Atari games may clarify why that was the case for the TORCS and RGSK data.\n- Table 1\n\t- t=2 is worse in all cases for the intersection calculation. So even the baselines did not need a sample count threshold. This weakens the claim that the multi-scale hierarchy has this benefit (made in Section 5.1).\n\t- On RGSK the $2^{20}$ state space is nearly the same as the mix model. Why is that true?\n\t\t- Is this related to the analysis of RGSK state space sizes in Appendix A.2.2?\n- Table 3\n\t- On first 32 moves: Why does \"Inter BC Similarity\" do so much worse than \"Intersection Similarity\", but not with fewer first moves?\n- Figure 3\n\t- What are the shaded areas? \n\t\t- Please include this in the captions to help readers.\n\t\t- Figure 5 in the appendix indicates 3 encoder models and a min-max range. If this is the case for other figures please add that information.\n- Figure 4\n\t- The figure is very difficult to read due to being very small. It would help to make this figure larger.", "broader_impact_concerns": "The broader impact statement already addresses potential ethical implications for privacy by being able to target advertising using behavior alone. As a technique for analyzing user behavior to identify (known) styles or users the technique has limited additional ethical concerns beyond these uses to limit user behavioral tracking with other methods.\n\nIn theory the technique can foster improved reinforcement learning algorithms using notions of similarity among policies. But this downstream application is generic and does not merit any additional mention.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Viiw5033Cs", "reviewer": "Reviewer_rjfA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper covers a topic that is relevant to the reinforcement learning community.\n- The experiments are quite extensive for the datasets and setups considered.\n- The overall writing style is clear (some typos remain).\n- The proposed Playstyle Similarity seems indeed to improve over Playstyle Distance, both by extending the concept and by displaying better performance.\n\nWeaknesses:\n- It is unclear to me what are the impact or implications of having alignment with human perception for the proposed exponential kernel. The kernel is applied to quantities over which humans do not have specific perception or intuition. Moreover, there are no experiments provided to support the human perception angle. These points weaken the interest of perceptual alignment for the present application.\n- The method introduced in Algorithm 1 for measuring policy diversity functions similarly enough to zero shot classification and nearest neighbor rule, having limited interest to the reader. I would argue that it could be deferred to the Appendix, where its experimental results are introduced.\n- The downside of complete or extensive experiments in the way they are currently presented is that it is difficult to interpret and draw conclusions from the multitude of plots and tables. Moreover, the results' interpretation seems a bit superficial. Otherwise stated, it is not clear how one would parametrize the proposed method for usage on a new dataset or task.\n- Additionally, the results' presentation could further be improved. Please find a few suggestions in Requested Changes. \n- The presentation of experimental setups as bullet points (p. 8-9) takes significant space. There is an opportunity to inline these and move some of the Appendix content to the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an improvement of the Playstyle Distance for comparing policy style and diversity for deep reinforcement learning (DRL). More precisely, a Playstyle Similarity is introduced, with a three-fold contribution: (i) introducing multiscale analysis with varied state granularity, (ii) using a similarity measure that aligns to human perception instead of a distance metric, and (iii) considering all states visited by the compared policies, not just the common ones. An algorithm is proposed for measuring policy diversity. Experiments are performed on multiple tasks: TORCS, RGSK, Atari games (7 tasks), Go, and 2048, using pretrained policies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be great to include formal definitions of distance, similarity, and kernel functions.\n- Fig. 1 introducing the Playstyle Distance framework could benefit from step numbers on the arrows, showing the pipeline from input to output. The same figure could also be extended to compare the original Playstyle Distance and the modifications introduced in this paper. Currently, there is no figure representing the proposed Playstyle Similarity.\n- Results' presentation: the efficacy comparison experiment is spread over three figures and two pages; it seems these figures could be fit in half a page. The reader would want to easily compare similar plots, especially when they are part of the same experiment. Using subplots, shared labels and potentially shared axes would improve readability. This could be applied to other experiments in the paper.\n- Results' presentation: numerical results in Tab. 5-14 could arguably use rescaling, i.e., extracting a $10^{-3}$ factor in the table headers.\n- Results' presentation: please highlight best results in bold in tables where it makes sense.", "broader_impact_concerns": "The paper contains a brief impact statement. I have no unaddressed concerns on this topic.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "75675MaKw8", "reviewer": "Reviewer_293a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The primary strength of the paper is the large number of experiments demonstrating the results of the paper in terms of the authors' approach being able to accurately identify players within roughly 512 samples. This represents a large amount of work. \n\nThe paper has a number of weaknesses currently holding it back. \n\nFirst, there is the lack of motivation for this work. The authors propose one possible application in terms of measuring policy diversity, and suggest others such as achieving human-like behaviour. However, there are no experiments run to determine if this new metric would have benefits to these application areas. The authors' claim that \"understanding and analyzing playstyles remain a complex endeavor\" but all of their examples are in terms of specific application areas, rather than why understanding and analyzing play styles itself is so complex. I would have preferred an argument about why we need to better understand similarity measures or why we need more accurate ones.\n\nSecond, there's a lack of engagement with prior work. Measuring player similarity or difference is a major part of the player modelling research field [1]. This area is focused on modelling human players, including finding similar and different players to a reference player. The types of metrics present in the paper, not over an embedded representation but directly on the game's internal state and action representation, are commonly found in this area [2,3]. \n\nThird, the evaluation setup is unclear. It's unclear what accuracy means here. I would guess that this is an accuracy calculated by finding the best match for some test sequence to some reference sequences, but this is not spelled out in the paper. Similarly, it's unclear if \"Dataset Size\" mentioned throughout the paper's graphs refers to the size of the reference sequences, the test sequences, or both. The \"SL\" and \"CL\" baselines are not described at all and appear to be random chance. The baselines in the paper that appear to be ablations of the authors' metric are also not clearly defined. I would suggest cutting the diversity algorithm to give the space to clarify these points. \n\nFourth, the results do not support the authors' claims. From my reading, the results seem to indicate that at low amounts of reference (or test?) data all metrics are the same and that at sufficiently large sizes they're the same. The benefit seems to be if one is in a situation where one only has access to datatsets of between 512 and 1024 samples. This does not to me seem like a \"pivotal advance\". They also do not \"shed light on understanding similarity through the lens of human cognition\", while human cognition is part of the inspiration for one part of the metric, there's no evaluation that supports this claim. I'm also not certain that this can count as zero-shot when the work relies on a learned or authored encoding for the domain. Overall, the results seem overstated. \n\n1. Yannakakis, Georgios N., et al. \"Player modeling.\" (2013).\n2. Zhang, Xiang, Hans-Frederick Brown, and Anil Shankar. \"Data-driven personas: Constructing archetypal users with clickstreams and user telemetry.\" Proceedings of the 2016 CHI conference on human factors in computing systems. 2016.\n3. Rivera-Villicana, Jessica, et al. \"Informing a BDI player model for an interactive narrative.\" Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play. 2018.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission covers a new metric for play style similarity based on observation and action sequences. The authors run a battery of experiments across three environments (TORCS, RGSK, and Atari), and demonstrate the ability to accurately identify players via matching reference observation and action sequences to withheld sequences. They then run additional tests in 2048 and Go, and propose a potential application for their metric.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I do not feel that the current draft is in a state for acceptance. I would suggest the following changes: \n\n1. Rework the introduction to focus on why studying similarity/difference in play styles is important and to remove the unsupported claims\n2. Rework the related work to acknowledge existing similar metrics\n3. Run a new evaluation with existing similar metrics as baselines that clearly identifies why the authors' proposed metric offers something distinct, either in terms of application areas or deeper understanding.\n\nI believe these would be needed to represent sufficient contributions for acceptance, particularly 3.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While some of the reviewers aren't certain that the paper makes a distinctly novel or significant contribution, it meets TMLR's criteria that _the paper's claims are well supported and the findings are of interest to TMLR's audience_, and so is worthy of publication in TMLR.
Decision: Accept (poster)