Identifying Off-Task Users in a Large-Scale, Game-Based Practice Assessment Matthew Emery Roblox Inc. memery@roblox.com David Laing Roblox Inc. dlaing@roblox.com Philip Simmons Roblox Inc. psimmons@roblox.com Jacob Seybert Roblox Inc. jseybert@roblox.com Katrina Yu Roblox Inc. kyu@roblox.com Erica Snow Roblox Inc. esnow@roblox.com ABSTRACT Kaiju Cats, a game-based practice assessment, was devel- oped and launched on the Roblox platform to prepare job candidates for a game-based hiring assessment. Since Septem- ber 2023, Kaiju Cats has been played over 50,000 times; however, many of these playthroughs were completed by non-candidates who were motivated to redeem avatar re- wards rather than to prepare for the hiring assessment. To better gain insight into how players may be approaching and interacting with the practice test, we deployed a finite mixture model to distinguish between on-task and off-task playthroughs. This initial analysis provided useful insights and laid out next steps for the continued development and iteration of our game-based practice assessments. Keywords game-based assessment, clustering, finite mixture model, mixed effect model 1. INTRODUCTION 1.1 Game-Based Hiring Assessments at Roblox Roblox is an online game platform and game creation sys- tem with a growing employee base. Each year, Roblox re- ceives thousands of applications from new college graduates for entry-level positions in software engineering and product management. In 2021, Roblox developed and operational- ized a game-based hiring assessment for these roles, designed to measure cognitive skills such as creative problem-solving and systems thinking. 1.2 Roblox Practice Assessment: Kaiju Cats Leveraging evidence that practice tests can elevate both or- ganizational and candidate success[3], we launched the Kaiju Cats practice assessment in 2023 to strategically prepare ap- plicants for our hiring process. Kaiju Cats integrates user interface patterns from our hiring assessment, complete with introductory and concluding screens, an intuitive tutorial, and a real-time timer. It familiarizes candidates with the specific UI and gameplay mechanics they will encounter in the hiring assessment and immerses them in a strategic sce- nario that produces similar cognitive and decision-making challenges. The game is designed to engage candidates in complex problem- solving tasks that require careful resource management and strategic planning, core competencies that are crucial across diverse roles at Roblox as identified through an exhaustive job analysis. Players must effectively allocate a limited bud- get and predict the outcomes of various strategies within the constraints of time. This setup illustrates the application of problem-solving skills, as candidates iterate their strategies in response to feedback, mimicking the dynamic and inter- active nature of the problem-solving scenarios with which applicants must engage. In Kaiju Cats, players are challenged to create a plan that directs a trio of cats across a city. The objective of the game is to maximize the cats’ total power—the sum of the numeric score associated with each cat—by causing them to destroy buildings and reach their respective destinations, cat beds located at the opposite side of the city. Players start with a fixed amount of dollars, which they can spend to define their plan. Each time players submit a plan for testing, they see the resulting power gains. Players have fifteen minutes to submit a plan that maximizes the cats’ power. Within this time limit, players can submit as many plans as they wish. Kaiju Cats was designed and developed in collaboration with game designers and engineers over a period of six months. The development phase included regular "think aloud" ses- sions with test users, which helped us ensure that the game appealed to a wide audience and effectively facilitated our learning objectives. Moreover, these sessions allowed us to confirm that players engaged