Reviewerly/exHarmony huggingface dataset:
Sample Efficient Reinforcement Learning with Double Importance Sampling Weight Clipping

Proximal Policy Optimization (PPO) is a stable on-policy policy gradient (PG) method thanks to its clipped importance sampling (IS) weight objective of policy improvement. However, on-policy PG methods usually suffer from poor sample efficiency. In contrast, off-policy methods have demonstrated better sample efficiency by making more effective use of all collected samples during training. In this work, we aim to develop methods that inherit both the stability of on-policy PG methods and the data efficiency of off-policy methods. To this end, we present GeDISC, an off-policy algorithm that improves sample efficiency by reusing off-policy samples drawn from prior policies. Besides, we propose double IS weight clipping to control the high instability caused by off-policy data. We take the recently proposed generalized clipping mechanism for off-policy data as the first clipping to bound the policy update from the current policy and meanwhile we extend the standard clipping mechanism in PPO as the second clipping to prevent high variance and bias brought by extremely old samples. Extensive experiments on continuous and discrete control tasks show that the proposed new algorithm outperforms PPO and other SOTA PPO-based off-policy algorithms.
Playing Pong without a TV Screen: an Alternative controller game

This work addresses a practice-based research method for game development, that resulted from an intervention challenge on the iconic Pong (Atari, 1972). As a critical and tangible path centered on Player-Game interaction, a new game experience was developed - HexPong (2022) - through an alternative controller and playful game design approach to strengthen the emotional experience of fun.The playtesting phase was operationalized in two specific moments. During a period of 5 days, a group of 32 adult-aged stakeholders (N=31, 96.8%) – of which 50.3% were students (N=11) or working-students (N=8) – played the game. Through directed observation and survey by SAM questionnaire after the game's testing session, the results obtained frame the discussion around the relevance of game design practice-based research for learning game development in university context.
Machine Learning Prediction of Just Dance Exergame Enjoyment from Mobile Sensor Data

Many young adults do not exercise enough, choosing instead to spend time on electronic media (e.g., smartphone, Internet). Exergames, which gamify physical activity, have been shown to be effective at increasing physical activity in an enjoyable way. For exergames to remain effective, sustained user engagement is key. However, sustaining long-term engagement in games (including exergames) is a challenging research problem – 95% of all new game players stop playing within 3 months, and 85% of new players stop after just one day. We posit that if detected early, waning player exergame enjoyment can be countered by recommending new, more enjoyable games before the player quits playing. In this paper, we investigate machine learning to predict user enjoyment of the Just Dance exergame by analyzing data gathered from the player’s smartphone. Specifically, "ground truth" scores for the players’ enjoyment obtained from the Immersive Experience Questionnaire (IEQ) E-scores are inferred from user behaviors such as increased excitement and gameplay frequency. These, in turn, are predicted by data gathered by the phone’s sensors – accelerometer, gyroscope and game features. Analysis of data from a user study shows the Naive Bayes classification algorithm achieves the best results, achieving 75% accuracy for binary classification (enjoying vs. not enjoying the exergame) of enjoyment E-scores. The most predictive features were the energy in the 0.5 to 3 GHz range, windowed energy in the 0.5 to 3 Hz range and radio spectral peak using a Discrete Cosine Transform (DCT). Our results are preliminary but encouraging and we plan to improve on our results by collecting more data and utilizing state-of-the-art neural networks approaches.
Procedural generation of rollercoasters

The "RollerCoaster Tycoon" video game involves creating rollercoasters that optimize for various in-game metrics, while also being constrained by the need to ensure a feasible structure in terms of physical and spatial bounds. Creating these procedurally is thus a challenge. In this work, we explore multiple approaches to rollercoaster generation, including Markov chains and machine learning and reinforcement learning algorithms. We show that we can achieve relatively good tracks in terms of the game’s measurement of success, and that reinforcement learning may give more control over other factors of potential interest. A focus on multiple measures allows our work to extend to other factors that also mimic actual player constructions.
PyTAG: Challenges and Opportunities for Reinforcement Learning in Tabletop Games

In recent years, Game AI research has made important breakthroughs using Reinforcement Learning (RL). Despite this, RL for modern tabletop games has gained little to no attention, even when they offer a range of unique challenges compared to video games. To bridge this gap, we introduce PyTAG, a Python API for interacting with the Tabletop Games framework (TAG). TAG contains a growing set of more than 20 modern tabletop games, with a common API for AI agents. We present techniques for training RL agents in these games and introduce baseline results after training Proximal Policy Optimisation algorithms on a subset of games. Finally, we discuss the unique challenges complex modern tabletop games provide, now open to RL research through PyTAG.
Cloud security in a bioanalytical world: considerations for use of third-party cloud services for bioanalysis

While using the cloud environment for various functions has become commonplace, relatively little attention has been given to considerations for the use of third-party cloud services for regulated bioanalytical workflow and data management. Little guidance has been provided as to how to utilize the cloud to support bioanalytical activities. It can be intimidating when considering how to go about using cloud services for data acquisition, but there are some general ideas to keep in mind when evaluating ways to accommodate regulated bioanalysis online. Determining how to incorporate the use of cloud storage with data that are generated from regulated bioanalytical analysis is an important step in maintaining the security of the data.
Segment-wise Level Generation using Iterative Constrained Extension

Generating game levels via constraint satisfaction can become slow as levels grow large. In this work, we explore constraint-based generation of large levels by iteratively extending smaller intermediate levels. In each iteration, a new segment is created by generating a new, larger, intermediate level. The tiles and solution path at the beginning of the new intermediate level are constrained to match the previous intermediate level. We evaluated our approach using three games and two low-level solvers. We found that generating a level using more segments improves performance up to a point, and then decreases performance. We also saw a slight but noticeable decrease in the tile-based differences between levels generated as the number of segments increases, and saw that later segments usually took more time to generate than earlier segments.
Preference-conditioned Pixel-based AI Agent For Game Testing

The game industry is challenged to cope with increasing growth in demand and game complexity while maintaining acceptable quality standards for released games. Classic approaches solely depending on human efforts for quality assurance and game testing do not scale effectively in terms of time and cost. Game-testing AI agents that learn by interaction with the environment have the potential to mitigate these challenges with good scalability properties on time and costs. However, most recent work in this direction depends on game state information for the agent’s state representation, which limits generalization across different game scenarios. Moreover, game test engineers usually prefer exploring a game in a specific style, such as exploring the golden path, yet, current game testing AI agents do not provide an explicit way to satisfy such a preference. This paper addresses these limitations by proposing an agent design that mainly depends on pixel-based state observations while exploring the environment conditioned on a user’s preference specified by demonstration trajectories. In addition, we propose an imitation learning method that couples self-supervised and supervised learning objectives to enhance the quality of imitation behaviors. Our agent significantly outperforms state-of-the-art pixel-based game testing agents over exploration coverage and test execution quality when evaluated on a complex open-world environment resembling many aspects of real AAA games.
A Deductive Agent Hierarchy: Strategic Reasoning in Codenames

The game of Codenames has attracted attention recently as it combines natural language and cooperation in a unique way. Codenames is a clue-giving game, where one teammate gives a clue to their partner, who then has to guess from a set of board words, given the clue. To date, the development of computational agents for Codenames has focused on the language aspects of the game. All clue-giving and guessing AI strategies explored thus far are static, basing their decisions only on the current state, without utilizing information obtained from previous clues and guesses. In this paper we present a preliminary exploration of the strategic aspects of agents for Codenames. We describe a deductive agent hierarchy, where agents utilize teammate models to make their decisions. Existing AI strategies for Codenames can be seen as occupying the bottom level of this hierarchy, and we describe general strategies for higher level agents. To explore these strategic ideas we also introduce a novel abstraction of Codenames called Codenums, which which can be seen as Codenames with numbers instead of words. We experimentally evaluate level 1 hierarchical agents in Codenums, demonstrating the potential for higher level strategic reasoning to influence agent design in Codenames.
Interpretation Using Classified Gradient-Based Saliency Maps for Two-Player Board Games

In this study, we propose a gradient-based saliency map generation method to improve the explainability of saliency maps in two-player board games, such as chess and shogi (Japanese chess). In the proposed method, our saliency maps are generated by masking gradients of unnecessary squares to reduce ambiguity and classify the gradient values to express the positive and negative aspects of each piece on a board. The saliency maps can clearly illustrate the roles of each piece in the game. We experimentally evaluate the saliency maps by comparing them against those obtained through conventional methods using chess and shogi board configurations, demonstrating their capacity to effectively identify vital pieces and squares. Thus, we can understand which pieces are important from the visualized saliency maps. This also provides useful insights for human players to learn the games.
Policy Diversity for Cooperative Agents

Standard cooperative multi-agent reinforcement learning (MARL) methods aim to find the optimal team cooperative policy to complete a task. However there may exist multiple different ways of cooperating, which usually are very needed by domain experts. Therefore, identifying a set of significantly different policies can alleviate the task complexity for them. Unfortunately, there is a general lack of effective policy diversity approaches specifically designed for the multi-agent domain. In this work, we propose a method called Moment-Matching Policy Diversity to alleviate this problem. This method can generate different team policies to varying degrees by formalizing the difference between team policies as the difference in actions of selected agents in different policies. Theoretically, we show that our method is a simple way to implement a constrained optimization problem that regularizes the difference between two trajectory distributions by using the maximum mean discrepancy. The effectiveness of our approach is demonstrated on a challenging team-based shooter.
A Cross-Modality Transfer Reinforcement Learning Blind Agent on DareFightingICE

This demo paper presents a deep reinforcement learning blind agent (blind agent) that uses only sound as the input on the DareFightingICE platform at the DareFightingICE Competition in IEEE CoG 2022 and 2023. A past study used reinforcement learning to train a blind agent and used it as the sample for the AI track of DareFightingICE Competition. However, this approach suffered from significant time expenses due to training the blind agent to recognize sounds and act in an end-to-end manner. To address this challenge, we propose a novel approach for blind agents where sound recognition and decision-making tasks are trained separately.
Visual Attention Assisted Games

In this work, we propose a committee of attention models developed for improving the deep reinforcement learning frequently used for games. The game environment is manifested with spatial and temporal attention mechanisms so as to focus on important regions while playing the games. We propose that when an agent’s visual attention space is streamlined and strategic temporally coherent representations are used, generalisation will be faster than other traditional architectures. The proposed spatial attention mechanism’s output enables direct analysis of the information recognised by the agent to choose its actions, allowing for a more straightforward interpretation of the provided state. We evaluate several techniques across a variety of games to reinforce our argument. Extensive experimental results on five Atari 2600 games demonstrate that an agent that makes use of this framework is capable of outperforming state-of-the-art models on ATARI tasks while being interpretable.
Tug &amp; Swing: Traveling in Virtual Environments with Point-Tugging and Arm-Swinging

In virtual reality games, players experience virtual worlds that often exceed the size of the actual tracking area. Small, stationary setups are especially problematic as they fail the requirements of most locomotion techniques. Although gesture-based approaches, such as walking-in-place, work in minimal playspaces, they often lead to unintentional movements or poor precision. Hence, developers often revert to less realistic virtual navigation concepts. We present a novel locomotion technique for VR games addressing this problem. Depending on the desired movement, players dynamically switch between two modes: point-tugging and arm-swinging. Tugging the controllers at chest height is used for precise maneuvering; swinging them at waist level triggers an accelerated forward motion for long travels. A within- participant study reveals that our concept avoids cybersickness while enhancing physical activity and presence.
Learning Computational Efficient Bots with Costly Features

Deep reinforcement learning (DRL) techniques have become increasingly used in various fields for decision-making processes. However, a challenge that often arises is the trade-off between both the computational efficiency of the decision-making process and the ability of the learned agent to solve a particular task. This is particularly critical in real-time settings such as video games where the agent needs to take relevant decisions at a very high frequency, with a very limited inference timeIn this work, we propose a generic offline learning approach where the computation cost of the input features is taken into account. We derive the Budgeted Decision Transformer as an extension of the Decision Transformer that incorporates cost constraints to limit its cost at inference. As a result, the model can dynamically choose the best input features at each timestep. We demonstrate the effectiveness of our method on several tasks, including D4RL benchmarks and complex 3D environments similar to those found in video games, and show that it can achieve similar performance while using significantly fewer computational resources compared to classical approaches.
Investigating Cases and Clusters-Based Reuse Policies for Card-Playing Agents

Computer games continue to present challenging experimental fields for developing Artificial Intelligence (AI) models. With Case-Based Reasoning and Clustering, this work proposes novel cases and clusters-based reuse criteria for implementing card-playing agents with diversified playing skills. Using the game of Truco, a common game in South America, we detail how game actions are reused from past cases selected as query answers for given game problems. In doing so, the majority rule, the probability-based lottery, the probability of victory, and the number of points won reuse policies are used to select a cluster of cases. Then these policies are also used to select game actions from the cases within the selected cluster. Investigating the combined exploration of reuse policies, experiments of different natures evaluate the performance of implemented Truco bots disputing matches against each other. Players in these tournaments are equipped with varied policies and use case bases constructed differently.
Practical PCG Through Large Language Models

Large Language Models (LLMs) have proven to be useful tools in various domains outside of the field of their inception, which was natural language processing. In this study, we provide practical directions on how to use LLMs to generate 2D-game rooms for an under-development game, named Metavoidal. Our technique can harness the power of GPT-3 by Human-in-the-loop fine-tuning which allows our method to create 37% Playable-Novel levels from as scarce data as only 60 hand-designed rooms under a scenario of the non-trivial game, with respect to (Procedural Content Generation) PCG, that has a good amount of local and global constraints.
Balancing Wargames through Predicting Unit Point Costs

In tactical wargames, such as Warhammer 40K, two or more players control asymmetrical armies that include multiple units of different types and strengths. In these type of games, unit are assigned point costs, which are used to ensure that all players will control armies of similar strength. Players are provided with a total budget of points they can spend to purchase units that will be part of their army lists. Calculating the point value of individual units is a tedious manual process, which often requires long play-testing sessions and iterations of adjustments. In this paper, we propose an automated way of predicting these point costs using a linear regression approach. We use a multi-unit, turn-based, non-balanced game that has three asymmetric armies. We use Monte Carlo Tree Search agents to simulate the players, using different heuristics to emulate playing strategies. We present six different variants of our unit-point prediction algorithm, and we show how our best variant is able to almost reduce the unbalanced nature of the game by half.
Complete DouDizhu Agents: Bid Learning from Pretrained Cardplay

DouDizhu is a challenging game that includes aspects of both competition and cooperation. In this game, players start by bidding based on their hand cards. They then follow by a round of cardplay in which the goal is to empty one’s hand of cards first. So far, other teams have elected to mainly concentrate on the cardplay phase in which agents such as DouZero and PerfectDou have recently achieved top performances. In this paper, we focus on bidding as it is a key component that greatly influences the starting states of the cardplay phase. We train bidding agents with an algorithm similar to DouZero and use PerfectDou for the cardplay phase. We then compare our agent to win-rate based and cheating agents. Our trained agent beat the win-rate agents.
Explainable Search: An Exploratory Study in SameGame

The field of Explainable Artificial Intelligence has gained popularity in recent years, due to the need for users to understand AI-made decisions, in order to increase their trust in the AI system. However, not much work has been performed on explaining recommendations made by search algorithms, which do not focus on single decisions, but on complex plans of action. This paper investigates promising directions for research in Explainable Search (XS), by evaluating with a user study different types of explanations for a search-based algorithm. Preliminary results suggest that users prefer explanations generated using context-based features, which are not only based on the current state of the problem, but are extracted from different parts of the tree generated by the search algorithm.
Game Level Blending using a Learned Level Representation

Game level blending via machine learning, the process of combining features of game levels to create unique and novel game levels using Procedural Content Generation via Machine Learning (PCGML) techniques, has gained increasing popularity in recent years. However, many existing techniques rely on human-annotated level representations, which limits game level blending to a limited number of annotated games. Even with annotated games, researchers often need to author an additional shared representation to make blending possible. In this paper, we present a novel approach to game level blending that employs Clustering-based Tile Embeddings (CTE), a learned level representation technique that can serve as a level representation for unannotated games and a unified level representation across games without the need for human annotation. CTE represents game level tiles as a continuous vector representation, unifying their visual, contextual, and behavioral information. We apply this approach to two classic Nintendo games, Lode Runner and The Legend of Zelda. We run an evaluation comparing the CTE representation to a common, human-annotated representation in the blending task and find that CTE has comparable or better performance without the need for human annotation.
Controllable Path of Destruction

Path of Destruction (PoD) is a self-supervised method for learning iterative generators. The core idea is to produce a training set by destroying a set of artifacts, and for each destructive step create a training instance based on the corresponding repair action. A generator trained on this dataset can then generate new artifacts by "repairing" from arbitrary states. The PoD method is very data-efficient in terms of original training examples and well-suited to functional artifacts composed of categorical data, such as game levels and discrete 3D structures. In this paper, we extend the Path of Destruction method to allow designer control over aspects of the generated artifacts. Controllability is introduced by adding conditional inputs to the state-action pairs that make up the repair trajectories. We test the controllable PoD method in a 2D dungeon setting, as well as in the domain of small 3D Lego cars.
A Visual Analytics Approach to Understanding Proximal Policy Optimization

PPO(Proximal Policy Optimization) is one of the most widely used and effective reinforcement learning algorithms, with successes in a variety of domains. However, understanding and diagnosing PPO can be challenging. In this work, we propose an interactive visual analytic system called PPOViz that helps users better understand and diagnose the PPO algorithm. PPOViz not only provides better visualization of the agent’s behavior, but also uses the Grad-CAM technique to make the agent’s model more interpretable. In addition, the gradient analysis of loss allows users to obtain other more useful information from loss. To demonstrate the effectiveness of PPOViz, we use the SpaceInvaders environment in Atari games to show how PPOViz helps users better understand and diagnose ways to improve the PPO algorithm.