The Game of Hidden Rules: A New Challenge for Machine Learning
Authors: Eric Pulick
Shubham Kumar Bharti
Yiding Chen
Vladimir Menkov
Yonatan Dov Mintz
Paul Kantor
Vicki M. Bier
Venue: Submitted to ICLR 2023 (2023)
Keywords: benchmark
environment
rule learning

Abstract: Systematic examination of learning tasks remains an important but understudied area of machine learning (ML) research. To date, most ML research has focused on measuring performance on new tasks or surpassing state of the art performance on existing tasks. These efforts are vital but do not explain why some tasks are more difficult than others. Understanding how task characteristics affect difficulty is critical to formalizing ML's strengths and limitations; a rigorous assessment of which types of tasks are well-suited to a specific algorithm and, conversely, which algorithms are well-suited to a specific task would mark an important step forward for the field. To assist researchers in this effort, we introduce a novel learning environment designed to study how task characteristics affect measured difficulty for the learner. This tool frames learning tasks as a ``board-clearing game,'' which we call the Game of Hidden Rules (GOHR). In each instance of the game, the researcher encodes a specific rule, unknown to the learner, that determines which moves are allowed at each state of the game. The learner must infer the rule through play. We detail the game's expressive rule syntax and show how it gives researchers granular control over learning tasks. We present sample rules, a sample ML algorithm, and methods to assess algorithm performance. Separately, we provide additional benchmark rules, a public leaderboard for performance on these rules, and documentation for installing and using the GOHR environment.

Under review as a conference paper at ICLR 2023
THE GAME OF HIDDEN RULES: A NEW CHALLENGE FOR MACHINE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Systematic examination of learning tasks remains an important but understudied area of machine learning (ML) research. To date, most ML research has focused on measuring performance on new tasks or surpassing state of the art performance on existing tasks. These efforts are vital but do not explain why some tasks are more difficult than others. Understanding how task characteristics affect difficulty is critical to formalizing ML's strengths and limitations; a rigorous assessment which types of tasks are well-suited to specific algorithm and, conversely, which algorithms are well-suited to a specific task would mark an important step forward for the field. To assist researchers in this effort, we introduce novel learning environment designed to study how task characteristics affect measured difficulty for the learner. This tool frames learning tasks "board-clearing game," which we call the Game of Hidden Rules (GOHR). In each instance the game, the researcher encodes a specific rule, unknown to the learner, that determines which moves are allowed at each state of the game. The learner must infer the rule through play. We detail the game's expressive rule syntax and show how it gives researchers granular control over learning tasks. We present example rules, an example ML algorithm, and methods to assess algorithm performance. Separately, we provide additional benchmark rules, public leaderboard for performance on these rules, and documentation for installing and using the GOHR environment.
INTRODUCTION
Learning computational representations of rules has been one of the main objectives of the field of machine learning (ML) since its inception. In contrast to pattern recognition and classification (the other main domains of ML), rule learning is concerned with identifying a policy or computational representation of the hidden process by which data has been generated. These sorts of learning tasks have been common in applications of ML to real world settings such as biological research (Khatib) et al. 2011 imitation learning (Hussein et al. 2018), and game play (Mnih et al., 2013) Silver et al. 2018). Since this process involves sequential experimentation with the system, much of the recent work exploring rule learning has focused using reinforcement learning (RL. for learning rules as optimal policies of Markov decision processes.
An important question is whether some characteristics make particular rules easier or harder to learn by specific algorithm (or in general). To date, this has been difficult question to answer, since many rules of interest in the real world are multifaceted and not well characterized. For instance, while there are effective RL algorithms that can play backgammon, chess, and go, these games differ in significant ways and it is not clear how much each structural variation contributes to differences in overall difficulty for the learner. In order to investigate these questions, new ways of generating rules and data must be devised that allow for researchers to examine these characteristics in controlled environment.
In this paper, we propose a new data environment called the Game Hidden Rules (GOHR), which aims to help researchers in this endeavor. The main component of the environment is game played in a board with game pieces of different shapes and colors. The task of the learner is to clear the board in each round by moving the game pieces to "buckets" at the corners of the board according to a hidden rule, known to the researcher but not to the learner. Our environment allows researchers express hidden rule using rich syntax that map to many current tasks of interest.

---

Under review as a conference paper at ICLR 2023
in both the classification and RL settings. A key advantage of our environment is that researchers can control each aspect of the hidden rules and test them at a granular level, allowing for experiments that determine exactly which characteristies make some learning tasks harder than others and which algorithms are better at learning specific types of rules.
The rest of the paper proceeds follows. We first describe how environment relates other lata-generating environments from the literature in Section [ In Section 3 we describe the GOHR and its rule syntax, explaining how the environment can be used to investigate the effects of rule structure. We introduce our ML competition and refer readers to benchmark rules, instructions, and documentation available at our public site in Section 4 In Section we present example rules and analysis for an example algorithm. Finally, we conclude with some discussion on the implications of our results and on other questions that can be answered by the GOHR environment in Section
LITERATURE REVIEW
Games have historically served as rich benchmark environments for RL, with novel challenges in each game environment spurring progress for the field as a whole. RL has tackled increasingly complex classical board games, such as backgammon, chess, shogi, and go (Tesauro) 1994] Campbell et al.) Silver al. 2016; 2017; 2018), eventually surpassing the performance of human experts in each. Of late, video-game environments have also become drivers of progress in RL. Beginning with human-level performance in Atari 2600 games (Mnih et al., 2013; Badia et al.) 2020), machine players have become competitive with humans in a variety of environments, including and multiplayer games such as Quake III, StarCraft II, and Dota (Jaderberg et al. [019 OpenAl jet al. [2019] Vinyals et al.) 2017 2019). Instrumental this progress has been the growing space of benchmark environments and supporting tools, such the Arcade Learning Environment (Bellemare iet al. 2013). General Video Game AI (Perez-Liebana et al. 2016). OpenAI Gym (Brockman et al. 2016). Gym Retro (Nichol al. 2018). Obstacle Tower (Juliani al. 2019). Procgen (Cobbe et al. 2020), and NetHack environments (Küttler et al. 2020; Samvelyan et al. (202T). Taken together, these represent an impressive range of new tasks for RL agents, bridging many gaps and challenges in achieving aspects of artificial intelligence.
tall-tiim-sttttegg
Benchmarks such as classic board and video games offer human-relatable assessments of MI. but are not readily configurable. This makes it challenging to use them for the systematic study of task characteristics and their impact on learning. Recent literature has proposed variations on these environments, such as alternative forms of chess (Tomašev al. 2022). While these environments provide additional insights, they are variations of already complex systems and do not provide the granularity needed to study the impact of task characteristics learning. We propose the use of the GOHR environment to study these questions since it is constructed specifically for configurability.
GOHR distinguishes itself useful environment in four important ways. First, each hidden rule represents a deterministic mapping between the game's ' pieces and the four available buckets, allowing for clear distinctions between the characteristics of each learning task. Second, the game's rule syntax introduces a vast space of hidden rules for study, ranging from trivial to complex. Third, the rule syntax allows for fine variations in task definition, enabling experiments that study controlled differences in learning tasks. Fourth, encoding different hidden rules does not affect the learning environment. In contrast to existing benchmarking tools, this decouples the study of different learning tasks from associated changes to the environment itself, making it possible to compare the effects of task characteristics under identical environmental conditions.
GAME OF HIDDEN RULES
In this section we describe the GOHR's 's structure, syntax, and the expressivity of the rule language. In each episode of our game, the player is presented with game board containing game pieces, each drawn from a configurable set of shapes and colors. The player's objective is to remove game pieces from play by dragging and dropping them into buckets located at each corner of the game board. A hidden rule, unknown to the player, determines which pieces may be placed into which buckets at a given point in the game play. For instance, rule might assign game pieces to specific buckets based on their shape or color. If the player makes a move permitted by the rule, the corresponding game piece is removed from play; otherwise, it remains in its original location. The episode concludes
2

---

Under review as a conference paper at ICLR 2023
the player has fully satisfied the rule. Typically, this occurs when all game pieces have been cleared from play, but some rules may be fully satisfied even with some pieces remaining on the board. The scoring for GOHR rewards players for completing episodes in as few moves possible, incentivizing players to quickly discern the hidden rule.
The GOHR is played board is indexed with a label, 1 1-36, and has z and coordinates 1-6. Each bucket indexed with a label, 0-3, with x, y coordinates in the set {0, 0), (0, 7), (7, 0), (7, 7) A diagram of the game board, including all numeric labels, can be seen in Figure M
board with 36 identical cells, arranged in 6 grid. Each cell on the
In each GOHR experiment, the game engine generates game pieces from user-defined sets of shapes and colors. Depending on the experimental goals, these sets can be of arbitrary size. If no specification provided the experimental setup, the game engine defaults squares, stars) and 4 colors (red, blue, black, yellow). The experiment designer may add shapes or colors the experiment in the associated color and shape configuration files. sample board, built using a set of 4 shapes and 4 colors, is shown in Figure
of 4 shapes (circles, triangles,
Columns
(Bucket 0)
2
5
6
(Bucket 1)
Row 6
31
32
33
34
35
36
Row 5
25
26
27
28
29
30
Row 4
19
20
21
22
23
24
Row
13
14
15
16
17
18
Row 2
7
8
9
10
11
12
Row
2
3
5
(Bucket 3)
(Bucket 2)
Figure 2: Game board, configured with 4 col- ors and shapes, as it would be seen by a human player.
Figure 1: Game board diagram, with numeric position labels included.
Boards in the GOHR can be defined in advance of play (to permit precise experimenter control), or can be generated randomly by the game engine according to a set of input parameters. This flexibility allows the experimenter evaluate the impact of deterministic (e.g., to determine whether seeing a particular set boards in a specified order improves learning). When generating boards randomly, the experimenter must specify the minimum and maximum numbers of game pieces, colors, and shapes to appear on each new board; the game engine randomly selects values in the range [minimum, maximum] for each quantity and generates boards accordingly. Additional documentation can be found at our public site, as noted in Section
randomized "curricula"
3.1 RULE DEFINITION
A rule playable within GOHR is defined in standalone file. Each rule is constructed from one or more rule lines, each which built from one or more atoms. For instance, a two-line rule with five atoms might look like:
(atom 1) (atom 2) (atom 3)
(atom 4) (atom 5)
Only one rule line is active at a time; this active line determines the current rule state (how game pieces may be placed into buckets for the player's next move). In the example above, the rule state formed by the contents of either atoms 1, 2, and 3 or atoms and 5, depending on which line is active. Later sections discuss the mechanisms by which the active rule line change during play.
3

---

Under review as a conference paper at ICLR 2023
Each atom maps set of game pieces set of permitted buckets and is defined follows:
(count, shapes, colors, positions, buckets)
The "count" parameter defines the number of successful moves that the atom permits (also referred to as its "metering"). Any game pieces matching the "shapes," " "colors," and "positions" specified in the atom accepted in any element of the set of "buckets." Multiple values can be listed for each non-count field and are grouped in square brackets. For example, the rule that "stars and triangles go in bucket 0, squares in the bottom rows go in bucket 1, squares in the top 3 rows go in bucket 2, and circles go in buckets 2 or 3" would be constructed with the following four atoms on single line:
(*-[star, triangle],* * * (* (S,square, * " (* ,squrrr," * (R4,R5,R6).2) (* (*,circle,* # *,,,,,,))))
The * character is a wildcard. For the count field, means the atom is not metered; for shape, color, or position, it means any value for that feature is permissible. In this example. atoms apply any indicated shape regardless of their color. Since the atoms are not metered, the objects may be removed any time that rule line is active. For brevity, the above, we use shorthand to refer to each row of the game board (R#); please refer to the detailed manual noted in Section for additional notes on position related syntax.
3.2 RULE EXPRESSIVITY
The above syntax permits vast range of rules within the GOHR. Stationary rules, such as the previous example, are those where the permitted bucket set for each game piece never changes during play. Thus, stationary rules permit the experimenter study the practical difficulty associated with learning feature-base position, or combination thereof. Though stationary rules have no dependence on play history, these rules can still be made difficult, such as by designating a specific bucket for every possible combination of game piece's features. Importantly, since the set of colors and shapes used is left to the experimenter, such rules can become arbitrarily complex.
patterns-i.e.. rules that map game pieces to buckets based on their color, shape,
Non-stationary rules are those which the permitted bucket set for game pieces changes during play. Non-stationas rules can involve following defined sequence, such as clearing a triangle, square, a circle, and then a star in that order (repeating the sequence as needed). It is also possible to create non-stationary rules with implicit priority between objects, such as clearing all available red pieces before all available blue pieces, or clearing pieces on the board from top to bottom. Methods for implementing non-stationary rules are described below. Whether stationary or non-stationary, we emphasize that all rules within GOHR are deterministic.
As mentioned, each atom contains "count" field dictating the number of corresponding correct moves that the atom permits. When a player makes a move satisfying one (or more) of the atoms on the active rule line, the counts associated with the satisfied atoms are decremented by one. An atom with a count of 0 is considered exhausted and no longer accepts moves. At each move, the game engine evaluates the active rule line and the current board; if there are no valid moves available the player, the engine makes the next rule line active and resets all counts in the new line. If there are no lines below the current rule line, the first rule line is made active again. This functionality can be used to encode sequences into the hidden rules. For example, the following two rules require the player alternate between placing game pieces in the top (0.1) and bottom (2.3) buckets, with subtly different mechanics.
Strict Alternation (1, * * [0.II)
Ambiguous Alternation
3, 0.II) (1, $ *, $ [2.3])
(1, * *.* (2.3)
In the strict case, the player's first move is allowed into only the top buckets and every correct move exhausts the current rule line. As a result, the active rule line will alternate with each correct move until the board is cleared. In the ambiguous case, a similar alternation occurs, but the order depends on the player's first correct move. Since both atoms are on the same rule line, the player's 's first move may go in any of the four buckets. After one successful move, the player may only make move satisfying the remaining, non-exte non-exhausted atom. After two successful moves, all atoms in the rule line exhausted and both are reset. This process repeats until the board is cleared.
4

---

Under review as a conference paper at ICLR 2023
GOHR also permits the experimenter to attach count to each rule line. When the line is active, this count decrements each time any atom on the line is satisfied. For instance, a rule that alternates between uniquely assigning shapes and colors to buckets would look like:
1 (*, star, *, *, 0) (*, square. *, *, 1) (+, circle, * *, 2) (*, triangle, $, .. (*, *, black, *, 0) (s, *, yellow, s, (*, blue, *, 2) (*, red, * 3)
If the rule-line count is exhausted, the game engine makes the next line active, even if there non-exhausted atoms on that line. For this example, the active rule line will alternate after each successful move, regardless of which atom in the active line is satisfied. If no count provided for given rule line, the game engine assumes that the rule line is not metered and can be used until all atoms that line are exhausted or no valid move exists for the game pieces currently on the board.
The GOHR rule syntax allows the experimenter to write expressions in an atom's 's bucket field that define sequences based on previous successful moves. The game engine stores values for the bucket that most recently accepted any game piece (p) as well as the bucket that most recently accepted an item of each color (pe) and shape (ps). A simple rule expressible using these values is "objects must be placed in buckets in clockwise pattern, beginning with any bucket":
(1, * [0,1,2.3) (+, 8, p+1)
The expressions used in the buckets field are evaluated modulo 4 to ensure that the resultant expression gives values in the range 0-3. The game engine also supports the terms "nearby" and "remotest" as bucket definitions, which allow the experimenter to require that game pieces be put into either the closest furthest bucket, respectively, evaluated by Euclidean distance.
The arrangement of atoms allows the experimenter to encode priority between component tasks within rule. For instance, the rule that "all red pieces go into bucket 1, then all blue pieces go into bucket 2" would be expressed as follows:
red, * 1) 2,uu, * 2)
Since both the first rule line and associated atom are not metered, they can never become exhausted, if there evaluates if there are any valid moves available to the player given the current rule line; there are no valid moves available, the engine shifts control the next line. In this example, once the player has cleared all red game pieces from the board, the engine will make the second rule line active.
more red game pieces left the board. However, as noted above, the engine
4 DOCUMENTATION AND COMPETITION
GOHR is designed to be flexible benchmarking environment for testing various rule learning related hypotheses. Beyond using GOHR for the study of task characteristics and their impact on difficulty, we hope that researchers will share performance results on benchmark rules with us so that we can disseminate community- wide benchmark performance. To this effect, we plan host public leader-board through June 2023 that will present results for benchmark rules, updated basis. The public site houses documentation for downloading and setting up the captive game server (CGS), resources for configuring and running experiments, benchmark rules supplementing those presented in Section 5
weekly
as well as the public leader-board.
EXAMPLE ANALYSIS
To illustrate how GOHR can be used for comparing rule difficulty, we provide a few example rules and test them using one specific ML algorithm (MLA). We use these example results to show possible performance metrics, statistical tests, and data presentation methods. Though we briefly discuss structural differences in the proposed example rules, the rules were chosen primarily to illustrate range of possibilities within the game; we expect that researchers will design targeted sets of rules to assess the impact of particular task characteristics on learning difficulty. Likewise, researchers may
5

---

Under review as a conference paper at ICLR 2023
perform similar analyses using different learning algorithms compare performance across common rules. Here we use a reinforcement-learnigg algorithm, but batch methods applied to, for example, data generated by random moves, could also be studied with GOHR, and may reveal unexpected relations regarding the difficulty of learning particular rules.
5.1 EXAMPLE RULES
We discuss the following example rules, played using the default four shapes and colors. Boards randomly generated with nine game pieces each, including least one piece of every shape and color. We note that the first rule is stationary, while the latter three rules are non-stationary.
1. Color Match: Each of the four default colors goes in a specific bucket, in any order.
(*, ", black, *, 0) (*, yellow. * 1) (*, *, blue, ", 2) (*, *, red, 3)
2. Clockwise: The first game piece can go in any bucket, but each subsequent piece must go in the next clockwise bucket
(1, # [0,1,2.3))
(*, * (p+ 1))
3. B23 then B01: Game pieces must be alternatively placed in the bottom and top buckets (1, *, *, [2,3])
(1, * ",**,, (0.1]
4. B3 then B1: Game pieces must be alternatively placed buckets 3 and (1, * 9,,3)
(1, 1,
5.2 EXAMPLE MACHINE LEARNING ALGORITTM
We model the GOHR as an episodic Markov decision process (MDP) characterized by the tuple, M = (S, A, R, P. ju, H). A the action space, where each action puts the object in row r and column in bucket b: A {Tr c, r E [6], 66].6= E {00 0), (0, 7), (7, 0), (7, 7)}}. Before introducing the state space, we first formally describe the board. In a board B, the cell at row r, column is characterized by color and shape: B Blr,c].cou color E {red, black, blue, yellow, 0). B(r, c).shape E (star, square, triangle, circle, 0). where we define the shape and color of an empty cell to be S is the state space, where each state S, = (Bo, Ao, B, A,, including all the historical boards B,'s ' and actions A,'s up to time step t. We define B-1 to be an empty board before the board generation process. P is the deterministic transition probability matrix specified by the hidden rule. Importantly, when action A, is not accepted, resulting in no change to the board, we still insert the board B and action A, to S In that case, B.41 Br. R is the deterministic reward function. If an action is accepted or the board is already cleared, the reward set be 0: otherwise, the reward is 1. Specifically: R(S, A,) 0 if B441 # B, or B, B R(S,, A,) = 1,, otherwise is the initial state distribution. Since the initial state So = (Bo). which only includes the initial board, μ is determined by the board generating process in Section 3 H is the time horizon that characterizes the number of time steps in each episode. The MDP evolves according to Section 3 We define the value function as the sum of future rewards by
B, B,) is sequence
taking action according policy n, i.e. V*((S) E - R(S,, Aw) where the expectation is over the potential randomness of the policy. Similarly. we define the state-actie action value function by the sum of future rewards by taking action A, at state S, and following policy after that, i.e. (S,,,, A,) R(S,, A,) + (S++1). Because the step information has already been encoded in the state, we do not need to use step h as the subscript for the value function and state-action value function. The goal of the learner is to find policy и to maximize the expected cumulative reward Eso- (S))).
Even though the MDP M has finite state space S, the size of S scales exponentially in H, which makes it impractical to learn a proper policy directly. Instead, we handcraft a feature mapping o(---) to extract only the essential information for some particular hidden rules. The feature function maps each state-action pair (S,, A) to a 17220-dimeniionll Boolean vector of S,, A). Recall that each action A chooses cell (r, c) and puts the object in this cell in bucket b. of St, A) encodes the following

---

Under review as a conference paper at ICLR 2023
information Boolean variables: the shape and color information of the cell chosen by A, the target bucket, and the shape, color, and bucket information of the last accepted action. Products of some of the Boolean variables also included in (SS,, A). Interested readers are referred to Appendix A.III for additional details on the feature representation.
We variant of DQN (Mnih al. 2013), model-free algorithm, Our MLA models the state-action value function Q as a linear function in the feature vector øi Q(s, a; 0) 0 o(s,o) a) and tries estimate the optimal Q" using stochastic gradient updates on moving loss L:
learning algorithm.
L( E [vv - a(s,a; 0))
where y = r max,' Qss', a'; ( is the target Q estimate obtained for the sample (s, a, r, 8') We distinguish the running parameter iterate and the target parameter furget Unlike the DQN in Mnih et al. (2013), Glurge is updated less frequently to stabilize the algorithm.
Additionally, the algorithm maintains buffer memory of size 1000 for past experiences and samples a batch of 128 tuples to calculate gradients at each time step. We run the MLA with an e-greedy behavior policy. The starts at 0.9 and terminates at 0.001. decays exponentially with step-size 200. More precisely, é 0.001 - (0.9 0.001) exp( - number movex/200). For each rule we run 100 trials, with each trial including 200 episodes length H 100. The full learning algorithm is described in Appendix A.1.2 and relevant hardware information is described in Appendix A.1.3
5.3 METRICS AND EXAMPLE RESULTS
In each independent learning run for given rule, we train our MI. from scratch by playing a set number of randomly generated episodes of that rule; each learning run uses different random seeds for the MLA and the game engine-board generator. We measure a rule's difficulty using the cumulated number errors made across the episodes of each independent learning run. As an MLA plays and learns from its experiences, its error rate decreases and the cumulated error count for that learning run levels off. This indicates that the agent has found policy that can play the rule essentially error-free free. Rules that are more difficult to learn result in more cumulated errors before the agent arrives at an error-free policy. By recording many learning runs, we obtain distribution cumulated error values that can be used to perform statistical comparisons between rules. For example rules and MLA, 200 episodes were sufficient for the cumulated error curves to flatten; we refer to the cumulated error after 200 episodes as the Terminal Cumulated Error (TCE) for that learning run. For other rules or algorithms, longer learning runs might be necessary. Figure shows cumulated error curves of example MLA for 100 separate learning runs of the Color Match rule (each consisting of 200 episodes), Such figures give insight into the shape of the TCE distribution for that rule and MLA. In addition, these data can be used to generate median learning curves that show the median cumulated етrог in each episode and associated 95% confidence intervals across all runs, as seen in Figure 4 Median learning curves visually rank rule difficulty, with higher curves indicating greater difficulty. Box-and-whisker plots for the distributions of TCE for each example rule are shown in Figure 5 with higher distributions again indicating greater difficulty.
10"
10"
Cumulated Errrs
Episode
Density
Figure 3: Cumulated error curves for 100 MLA learning runs of the Color Match rule. The y-axis logarithmic to capture the distribution's 's skewness. The marginal plot to the right displays kernel density estimate for the TCE for this rule and MLA.
7

---

Under review as a conference paper at ICLR 2023
350
300
Cumulated Errors
250
200
Color Match
150
Clockwise
B3 then B1
100
B23 then B01
Episode
Figure 4: Median cumulated enrors at each episode across 100 MLA learning runs for each rule. The shaded regions denote 95% confidence intervals for the medians, calculated with 50,000 bootstraps.
3000
2000
1000
500
Cumulative Errors
400
300
Color Match
Clockwise
B3 then B1
B23 then B01
Rule
Figure 5: Box-and-whisker plots of TCE for each learning run for all example rules. The chart has a broken y-axis with scale change to capture the most extreme outliers. The colored area the interquartile range (IQR). Whiskers denote data within 1.5 IQRs of the upper and lower quartiles.
Although the heavily- skewed TCE distribution seen for Color Match was not seen for all example rules, it highlights the need for non-paramtrric tests when comparing the difficulty of two more rules. To perform pairwise comparison rule difficulty, for instance between two specific rules, A and B, recommend using the Mann- Mann-Whiney- wilceoxn U-Test (Nachar 2008) with TCE as the moint-metric. The test compares every TCE value associated with rule against every TCE value associated with rule B; the associated test statistic sums the number of times that values from exceed values from B, with ties counting for 0.5. The difficulty ranking seen in Figure was confirmed by pairwise one-sided U-Tests; the largest p value across pairwise was less than 0.002.
Although this analysis intended only illustrate the process for measuring algorithm performance, we note a few interesting structural differences in our example rules which could merit further study. We believe a framework for analyzing task characteristics in the GOHR should consider the number of underlying logical mappings in a rule, the arrangement of these mappings across features or sequential mechanies, and the cardinality of these mappings (e.g. 1:1, 1:n, etc.). Under this framework, Color Match requires that the player learn four 1:1 mappings between colors and buckets. Clockwise requires that the player learn four 1:1 mappings between the prior bucket in the clockwise sequence and the next bucket in the sequence. Like Clockwise, B3 then B1 considers 1:1 mappings based on the last correct move, but the player must learn only two such mappings rather than four. B23 then B01 requires the player to learn two sequential 1:1 mappings, similar to B3 then B1, but the player can learn any one of four pairs of mappings as part of an error-fres policy. Despite their structural differences, note that Color Match, Clockwise, and B3 then B1 each give the player
8

---

Under review as a conference paper at ICLR 2023
the same probability of making correct moves at random (1/4); we thus refer to these rules as being equally permissive. By this definition, B23 then B01 is the most permissive rule in this set of example rules, it has corresponding probability of 1/2.
Recalling the difficulty ordering shown in Figure confirmed by pairwise U-tests, the rules with four underlying mappings (Color Match and Clockwise) were more difficult for our example MLA than those with only two (B3 then B1 and B23 then B01). B3 then B1 was found to be more difficult than the more permissive B23 then B01. Color Match and Clockwise require learning the same number of mappings, but Color Match was measured to be more difficult. Although Color Match is stationary and Clockwise is non- -onttttttooa,, we believe this performance difference is primarily an artifact of our MLA's featurization method. These learning tasks are only meant to suggest interesting avenues for future research; we believe that the GOHR equips researchers to employ larger and more granular rule sets to pursue ideas similar to those raised here in order to rigorously identify how particular task characteristics affect difficulty for given learning algorithm.
In addition to comparing the difficulty of two different rules for given algorithm, the U-Test also supports comparison between different MLAs. The U-Test statistics can be used to form an "ease ratio," representing the ratio of learning runs where rule A was found easier than rule B by given MLA. We propose aggregating ease ratios for many MLAs across sets of benchmark rules to create tournament graph as described in (Srang et al. (2022). The ranking method corrects for the fact that algorithms R and S may have solved a different subset of rules, leading to the possibility that R encountered larger fraction of "eaay"" rules (where rule difficulty can be assessed by the TCE of all algorithms that attempt a particular rule). Additional information regarding the methods to be used in ranking competition submissions can be found the public site
6 DISCUSSION
The GOHR tool provides capability for studying the performance of MLAs in novel and principled way. Using the expressive rule syntax, researchers can make precise changes to rules of interest to study how they affect algorithm performance. As suggested in Section 5] this approach can be used to compare the performance of different algorithms on any specific set of rules and to see the effect of rule characteristics on MLA performance. This is useful complement to present competitions among ML algorithms on existing board games and video games that can look only at complex tasks without a fine level of granularity. The original proposers, Bier et al. (2019), envisioned a simpler version of what we call the GOHR as a tool for comparing human and machine learning; here we extend this notion a probe of differing ML frameworks.
Each choice of ML parametrization, learning model, and optimization is expected to produce different results. With the GOHR, we believe it will be possible to explore how these choices interact with specific rule characteristics to result in differences in practical performance. With this goal in mind, we are sharing the complete suite of tools with all interested researchers. We hope that researchers using the GOHR tool for ML research will share their findings to help this inquiry.
We believe there are at least two interesting broad research questions that can be answered with given ML algorithm learn many substantially different rules?
the GOHR environment, First, Algorithms capable of this kind of general learning within the GOHR environment may prove more capable in other domains. Second, when results for many algorithms available, will it be possible to cluster rules into intrinsic difficulty classes based on those experimental findings? Such a classification may reveal conceptual task characteristics, extensible beyond GOHR, that characterize learning complexity more generally. We hope this platform will be helpful to ML researchers in answering such questions.
7 REPRODUCIBILITY STATEMENT
Our paper presents example results intended to show the expressivity of the rule syntax as well as example means of creating and analyzing experiments; these results are not intended to serve as a specific performance benchmark. Regardless, we provide the code used to generate these results in the supplementary materials, with corresponding instructions in the Appendix so that they can be reproduced by interested researchers.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Adria Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the Atari Human Benchmark. In Proceedings of the 37th International Conference Machine Learning pp. 507-517. PMLR, November 2020. URL httts::/pprccedd
www...s......p......m...m htt.://........ooree////////aaaa hmml
ISSN: 2640-3498.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47:253-279, June 2013. ISSN 1076-9757 doi: 10.1613/jair.3912. URI.ttpss: /ttp://wwwww.. jair www.jarir..riaaa.......
2bhp//airr/arricce//veeew///
Vicki Bier, Paul B. Kantor, Gary Lupyan, and Xiaojin Zhu. Can We Distinguish Machine Learning from Human Learning? Technical report, 2019. URL http://arsdw.orr//bbb.. 03466
Leprint: 1910.03466.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym, June 2016. [tt.:t::://axxxi..rrrr [es].
01540
Number: arXiv:1606.011
Murray Campbell, A.Joseph Hoane, and Feng-hsiung Hsu. Deep Blue. Artificial Intelligence,
134(1-2):57-83, January 2002. ISSN 00043702. doi: 1011006500044377000000001113 URL
Bttpo://Annkinghh...eeeerr..... wtw.://swweeee.cceeeeeeeeeeeeeee /ttp://hww..a.aaaoooooooooor/aar
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging Procedural Generation to Benchmark Reinforeement Learning. In Proceedings of the 37th International Conference
Machine Learning, pp. 2048-2056. PMLR, November 2020. URL [RRL:ttts:/::poceeeeet
mlr alr.press//el
www.press/vlll https///11///obbeeeeem
ISSN: 2640-3498.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation Learning: A Survey of Learning Methods. ACM Computing Surveys, 50(2):1-35, March 2018. ISSN 0360- 0300, 1557-7341. doi: 10.1145/3054912. URL https: /dl acm. http///aa...g////////////// www.cn/.gg/o/o/....l/..1
3054912
Max Jaderberg, Wojciech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Cas- tañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in 3D multiplayer games with population- based rein- forcement learning. Science, 364(6443):859-888) 865, May 2019. ISSN 8073, 1095-9203. doi: 10.1 1el11//cienee.aaateeeeo aau6249. URL bistts://wixe sc ence. www./i..o//ooooooo.o.oooooo.... 112.
basedd
aau aau 6249
Arthur Juliani, Ahmed Khalifa, Vincent- Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning. Technical Report 1rXiv:1902.01137 arXiv, July 2019, URL [es] type: article.
http: fittp://arxise
01378 arXiv:
Firas Khatib, Seth Cooper, Michael D. Tyka, Kefan Xu, Ilya Makedon, Zoran Popović, David Baker, and Foldit Players. Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences, 108(47):1894) 11118899999955 November 1. ISSN 0027-8424, 1091- 6490. doi: 10. 10.1073/pnas. 115889888
URL https::/ppaas
pnas 1eas.110 11155981188
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Ed- ward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 7671-7684. Curran Asso- ciates, Inc., 2020. URL 569f 5
10

---

Under review as a conference paper at ICLR 2023
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013. URL
atttps://arxiv: org/ httg///s//ia 1ates/312.550 5602
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Na- ture, (11((554)):529-555555 533, February 2015. ISSN 0028-0836, 1476-4687. doi: 10.1038/nature 14236. URI http: www. tareer.e.. com/ art cttm//atiicccee/mattrrr11122222
Nadim Nachar. The Mann-Whiume U: A Test for Assessing Whether Two Independent Samples Come from the Same Distribution. Tutorials in Quantitative Methods for Psychology, 4(1):33-20
March 2008. ISSN 1913-4126. doi: 10. 2tt099/////mm00........... URL httt:://wwt tgmpp
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta Learn Fast: A
New Benchmark for Generalization in RL. Technical Report 100.::18000037200 arXiv, April 2018. arXiv: 10V.0037003 [es, stat] type: article.
URL [tLttt////axxi org/ abs /t.b//8844 03720
OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with Large Scale Deep Reinforeement Learning. Technical Report arXiv: 10..99....68000 arXiv, December 2019. URL ltttp:://ara.e. org/ hal///b//// /1912 06680 arXiv:1912.06653 [cs, stat] type: article.
Diego Perez-Liebana, Spyridon Samothrakis, Julian Togelius, Tom Schaul, Simon M. Lucas, Adrien Couetoux, Jerry Lee, Chong-U Lim, and Tommy Thompson. The 2014 General Video Game Playing Competition. IEEE Transactions on Computational Intelligence and Al in Games, 8(3): 029-243 September 2016. ISSN 1943-068X, 1943-0698. doi: 10. 100/TCIAAG.201552422000 URL http:://eeeee: wwweeeeppooeeeooooo.... ore. leee www.or////oo.... http:////mmmmmm///322/22
Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Kuttler, Edward Grefenstette. and Tim Rocktäschel. MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Re- search. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Informa- fion Processing Systems Track on Datasets and Benchmarks, volume I, 2021. URL Tattaa//.ottssssseeeeeenmmaaassp www.arrcees-r--peeeeeoo--eoeeeee ngs. newrips..ppp... cc http//pppr//221111 Paper roundl. pdf
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap. Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 489, January 2016. ISSN 0028- 0836. 1476-4687. doi: 10. 1...8//natree1661.... URL http: / www nawuuratuee.e.. com articles/ naturel6961
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre. George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7676):354 359, October 2017. ISSN 0028- 0836, 1476-4687,
doi: 10. 10.1038/aature242700. URL http: www. natureree com/ artiiles/
na naturee22270
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and Go through self- self-play. Science, 362(6419): 11044499::14411111 8075, 1095-9203. doi: 10.1 10. 11 /2/116//ao /ct///cienn encee.ccc........ aar6404
December 2018. ISSN 0036-
URL https:: /www. wwwence.ecc.cccco.ce org/
11

---

Under review as a conference paper at ICLR 2023
Alexander Strang, Karen C. Abbott, and Peter J. Thomas. The Network HHD: Quantifying Cyclic Competition in Perfo-Perf Models of Tournaments. SIAM Review, 64(2)360- 391, May 2022. ISSN 0036- 1445, 1095-7200. doi: 10.1137/20M1122101. URL /Rtpptpp///epubs... siam org/doi/10. 1tt.//0.111770000 321012
Gerald Tesauro. TD-Gammon, a Self- Teaching Backgammon Program, Achieves Master-Level Play. Neural Computation, 6(2):215-219, March 1994. ISSN 0899-7667, 1530-888X, doi: 10.1162/ neco.1994.6.2.222 URL http:////...eee....eeeeee.eeeiee
mit http:://...aaaacccccc.//////a/a/
5771
Nenad Tomašev, Ulrich Paquet, Demis Hassabis, and Vladimir Kramnik. Reimagining chess with AlphaZero. Communicattions of the ACM, 65(2):60 66, February 2022. ISSN 0001-0782, 1557- 7317. doi: 10.1445/346039. URL https: al acm. org/ /tt/1o..115//444333333
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney. Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David Silver, Timothy Lillicrap. Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp. and Rodney Tsing. StarCraft II: A New Challenge for Reinforcement Learning. Technical Report arXiv: 077822 arXiv, August 2017. URL httpp://axxi..orgp///pp. 1708. 04782 ExXiv1170800482 [cs] type: article.
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnekki Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre. Ziyu Wang, Tobias PfafT, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap. Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft II using multi reinforcement learning. Nature, 575(7782): doi: 1038/44 1886-099-77244 1724--. URL http: www. na tur com art C ls 4 586 01 1424--1-1-211111-
350-354, November 2019. ISSN 0028-0836, 1476-4687
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 DETAILS OF THE REINFORCEMENT LEARNING ALGORITIIM
A.1.1 FEATURE REPRESENTATION
The feature vector (S,, A) is constructed with the following four different types of components:
The 12 current-step unary features o (S, where be a color, shape, or bucket. If is a color:
Ou(S,, A) = 1 BB AAr, A.c].color =n).
is a shape:
(St, A) 1( A.c]. = u).
is a bucket:
Φ (S,, 1(A.b =].
2. The 48 current-step binary features, of binary features that depend on the current board configuration ou, : S A + {0, 1} where (u, v) can be (color, shape), (color, bucket) or (shape, bucket) tuple. These features uu A) are defined as follows:
u,
(u,r A)
color x shape
1(B][Ar,, = LiBr[Arr, A.c) v}
color x bucket
[.B.[AArr,..cc.oooo = u} A 1(A.b = 4}
shape x bucket
(.B.AAr,, u} ^ 1 v}
The 60 last-successful-stee and current-step binary features, set of binary features constructed from the board and action information at the last successful step and the current step. For a time step t >> 0, let
{(t) max(r E {-1,0, ... , 1) s.t. B B,}
be the latest successful time step. Since B_1 an empty board, B- * Br, 2 until the game is over, so ((t) is well defined.
The features are defined follows: ((u). x A - (0, 1}. where (u) is extracted from the board and action information at the last successful time step, and is extracted from the board and action information at the current time step. Note that u can the empty set when E(t) 1.
(u), v
(()). (St, A)
(color") x color
1(B
= u} ^ A.c.coolr = r}
(shape") x shape
L ((Bu))AAua)) Arco shape = u} ^
shape v}
(bucket") x bucket
1(A u) ^ 1{A.b w}
where _ (0), Yz [color, shape, bucket)
The 3600 last-successsullsttep and current-step 4-ary features, a set of 4-ary features constructed from the board and action information at the last successful step and the current step.
((. S + (0,1)
where (u, v)) are extracted from the board and action information at the last successful time step. and y, 2 are extracted from the current time step. These features, ((u.)) (St, A), are defined as follows:
13

---

Under review as a conference paper at ICLR 2023
(S,,
I Bee Aeg = =u}
(shape", ,olorr)
A1 (Bees) (A(r) Ato. =v}
xshape, color
AT (1BB((Arr A.c..shape = A A.c. =]
u}
(shape", color x shape, bucket
^1 A4 o}
AL 1 2}
1 ((0
(shape", colorr)
^ 41 = 2A
Xcolor, bucket
A
(shape", bucket")
A1 Aro) u) 1 [Aq(t).b o)
1 (
x shape, color
A.c] = 2}
(shape", bucket")
1f [A Aca = u) A 1 { = m}
x shape, bucket
Al MA 2]
(shape", bucket")
1 A4Brgg/Akkg...AA40ccchah-------
1 [ o}
xcolor, bucket
A AIDDAAA;AAAA.ooor==)]111...
(color", bucket")
1 (ecce 1 A,cg__ u) 1 ( Ac(t) A
xshape, color (color", bucket")
1 { [Be s.c....appe-///
A.c].color =}
L(Becy]AAce)) Act),,),oor = 4} ^ 1 (A((t).b = =u}
xshape, bucket
A1 A [A.r, y) 1[A.b =}
y)
(color", bucket
1 (B(o))A(())) Agy. u) 1 Ag(t) v}
xcolor, bucket
A1(B, A.c. color cocor
A.b
For any (S, A)tuule, the final feature vector d(S,A) is created by stacking up all of the above features, which gives a feature function @ : x A + (0,1)370)
A.1.2 LEARNING ALGORITIM
Algorithm Linear Variant of Deep O-lcarning(DON) with Experience Replay
Input:
target iteration T, episode count M, buffer size N. batch size B. Initialize experience replay buffer memory D with capacity N.
Initialize the iterate parameters 0junze <- 0, 01,1 <- 0. for target iteration T = 1,... do for episode
1,... .M do Draw $1 ~ μ..
for time step = 1.... .H do
Draw ae ~ e Unif( ()) + (1- - - 1 1 max, Q(s a'; a'; Oc,)))
Take action a, and observe reward and observation 2++1-
Set 8141 + (8t, at F1+1) and enque (Ng, at Tt, 8141) to D.
Uniformly sample a batch {(8), "j. r)) from D and
set vij E [B], y r, + 1 is terminal) y max Q(5 d':0 m Update fe,c41 using Q -objecttve 1 E/ (3) - ass 0))² 1=0...' end for Update the parameter for next episode fct1,1 + 0c.H+1. end for
Update the target parameter 0. + 00MH+ end for
14

---

Under review as a conference paper at ICLR 2023
A.1.3 TRAINING HARDWARE
For all experiments, we used local server running Ubuntu 20.04 LTS, with Intel Xeon Silver 4214R CPUs (2.40 GHz) and 196GB of RAM. No GPUs were used in our experiments.
A.1.4 RELEVANT CODE
The following files are included as part of the zipped supplementary materials. We describe the use for each file:
captive: codes for the game server;
rules rules_filest files: rule and board configurations;
r __codee: DQN implementation:
dqn the DQN algorithm:
driver_dqn py: experiment setup;
rule gawa_eeniine__..eeeeaaa_aa_sseee py and game_game___ env py: codes interact with the game server;
rule_sets, py: featurization;
run_code ss: set the path to code of captive game server;
params params/last_sunee cerst__smmer____e rererrule///eeee experiment configuration for each rule;
run jun_script/run last_ summer dgn rand. sh: script run the experi- ment.
A.1.5 TRAINING PROCEDURE
The code can be run by changing the directory to ro_codes and using the following command: bash run_script/rus wtt.scrrttruunlaattt tetsssssumeer_a___eaeeeeeee sh.
15


Reviews:
[{"review_id": "IWcIaWShPuj", "reviewer": "Reviewer_9f8z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper the authors present a new test domain for ML ideas, called the Game of Hidden Rules, meant to help researchers disentangle how different aspects of the task and setting impact the level of difficulty for learning algorithms. In each episode of the game, the learning agent is presented with a game board containing game pieces, each drawn from a configurable set of shapes and colors. The objective is to remove game pieces from play by dragging and dropping them into buckets located at each corner of the game board. A hidden rule determines which pieces may be placed into which buckets at a given point in the game play. For instance, a rule might assign game pieces to specific buckets based on their shape or color. If the agent attempts a move that is not permitted, the piece remains unmoved. Rules can also be nonstationary, for instance by changing the logic of which moves are allowed during play. ", "summary_of_the_review": "This is an interesting idea but I don't think there's enough of a contribution here at the moment to be useful to the general ICLR community. There is a ton of potential though, so I truly hope the authors bulk up this study with more results and insights before calling upon us to embrace it as a new central test domain for ML research.", "strength_and_weaknesses": "Strengths\n-----------\nThe authors propose an interesting game idea, certainly, and I can certainly see it could be useful as a domain.\n\nWeaknesses \n------------\nIt's not clear to me whether proposing this game is in itself enough of a contribution to merit publication in a venue such as ICLR. Especially since there is not a sufficiently strong analysis of how different learning algorithms fair in this domain, at least as baseline, which is something I would expect given the nature of this paper. \n\nFurthermore, it's not clear to me what uniquely qualifies this game to study the way task parameters impact learning, more than countless other domains used in contemporary ML, from Starcraft and FLOW to classic games such as chess and Go, which all have their configurable characteristics, which researchers indeed tweak. Furthermore there are plenty of games in which the transition dynamics may change from episode to episode - what makes this domain better suited for study than any of the countless games authors have come up with in decades of ML research? I am not claiming this is not the case, just that I don't think it's sufficiently clear from the paper what the differentiators are.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GScz9GCqOd", "reviewer": "Reviewer_FEUd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduce a novel learning environment to study how task characteristics affect measured difficulty for the learner. Besides, it detailed the expressive rule syntax in the environment and showed how to control over tasks. Some example results and analysis are presented in the paper and additional benchmark rules are provided in public site.", "summary_of_the_review": "Novel environment but I'm not sure about the usefulness.", "strength_and_weaknesses": "Strengths:\n\nThe proposed environment is novel and seems to be used to study the impact of task characteristics on learning since it is constructed specifically for configurability.\n\nThe GOHR has expressive rule syntax so that researchers can control over tasks and make precise changes to rules of interest to study how they affect algorithm performance.\n\nWeaknesses:\n\nThis paper claims that researchers can control and design tasks in GOHR. In my understanding, this controllable environment should have the ability to compare different tasks once the task is designed so that researchers can change the task design in advance. However, the examples in this paper utilize the cumulated error curves to evaluate the difficulty between tasks. If the evaluation is realized after the experiment, what is the use of this so-called controllable task since we need to carry out experiments after designing tasks to compare different tasks.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F8igDo-nsh", "reviewer": "Reviewer_Ccmc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors describe an expressive platform, GOHR, for generating game\nrules (of a certain number and complexity) and for evaluating how fast\nthe various agents figure out the rules.  Such a platform can lead to\ninsights into difficulty of rule sets, and relative capabilities of\ndifferent learning approaches, through systematic studies. For\ninstance, are there strategies that are uniformly good, or in some\nsense pareto optimal? The authors also present an example MDP-based RL\nagent, and a few different types of rules, and study the relative\nhardness of learning those rules with respect to the agent (how long\nthe agent takes, until it makes no mistakes).\n", "summary_of_the_review": "\nThe work in trying to give researchers control over level difficulty in a space of tasks, and providing an easy to use\nplatform is a good contribution.  I am concerned the space remains too narrow and may not generalize to the real world.  \nHowever, overall, I am positive on the paper.\n", "strength_and_weaknesses": "Strengths:\n\n-- Paper is clear and well written.\n\n-- A platform and a suite of tools is provided so researchers can do\n   different analyses (the authors show some of what one can do).\n\nWeaknesses (More details after the list):\n\n-- Because the setting is so contrived (by definition or from the\n get-go!) and combinatoric, how well any findings translate to\n real-world performance or difficulty is always questionable (this is\n admittedly an issue with any game and/or synthetic setting). If the\n authors could provide an example of a real-world problem (drug\n discovery, optimization, planning, etc) where engaging in this study\n could provide insights, that would strengthen the paper.\n\n-- [related to above] It would be good to have more uncertainty (see\n below). The authors focus on and motivate determinism, perhaps to\n allow for systematic study and comparisons (page 2, deterministic\n rules leading to 'clear distinctions.. [among].. learning tasks', but\n uncertain rules and environments could be insightful too and lead to\n clear distinctions..).\n\n\n----------------------------------\n\nMore details:\n\n\n-- How about non-deterministic rules and incomplete rule sets? so that\n the agent needs to learn or anticipate that the world is noisy and\n uncertain? This would be more applicable to real life: (often) no\n rule learned is perfect, and no rule set is complete!  As the authors\n explained, there could be some non-determinism in how rules are\n applied (eg in the order the rules become applicable), but that's\n very limited.\n\n-- For the above case, perhaps the agent needs to learn when to stop\n learning, ie it has learned all that it could and can do no better\n (which changes the interface to the platform).\n\n-- More non-stationarity could be another complicating factor: the rule\n set changing over time, but not in a deterministic/periodic fashion.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject