- You-Shall-Not-Pass is a two-party competitive game as Figure 6 shows. Two humanoid robots face each other at the beginning of the game. The red robot then starts running towards the finish line (indicated by the red line), while the blue robot is tasked with preventing the runner from achieving its goal. In our evaluation, the red robot wins the game if it can get across the finish line. Otherwise, the blue blocker wins. The red robot gets the reward  $+1$  once it wins the game and -1 when it loses the game
- Kick-And-Defend is a two-party competitive game as Figure 7 shows. Two humanoid robots compete in a soccer penalty shootout. The placements of kicker, defender, and ball are all chosen at random. If the ball falls between the goalposts, the kicker wins; otherwise, the defender wins. The reward is  $+1$  for the kicker to win the game and -1 when the kicker loses.
- CartPole is one of OpenAI's open-source environments as Figure 8 shows. The "cartpole" agent is a reverse pendulum in which a "cart" attempts to vertically balance a "pole" with a slight angle adjustment. The only forces that can be used are represented by  $+1$  and  $-1$ , corresponding to a left and right movement, respectively. The episode ends if the cart moves more than 2.4 units away from the center or if the angle deviates from vertical by 15 degrees. For each timestamp that the episode hasn't ended, the reward is increased by one. The maximum reward available is 500.
- Pendulum simulates the inverted pendulum swingup problem as Figure 9 shows. The pendulum starts in a random position, and the goal is to swing it up and keep it upright. The pendulum angle is represented by  $\theta$ , which starts at random from  $-\pi$  to  $\pi$ . The default reward function is based on the angle  $\theta$  and the agent's action  $a$ . To maximize rewards, the agent must keep the angle  $\theta$  at zero, rotate at the slowest possible speed, and exert the least amount of effort.
- StarCraft II is a popular real-time strategy game as Figure 10 shows. The player controls an army to defeat the opponent's base by collecting resources and building units. The player's observation is the minimap and the screen. The minimap shows the entire map, while the screen shows the player's view. The player can take actions such as selecting units, and issuing commands to the units. The player wins if the opponent's base is destroyed, and the game ends if the player's base is destroyed. The winning, and losing outcomes of the player correspond to  $+1$  and  $-1$  rewards, respectively.
- Connect 4 uses a  $6 \times 7$  grid as Figure 2 shows. Players take turns dropping tokens into the grid. The tokens land straight down, taking up the lowest available space in the column. The player wins if any four of the tokens are in a row horizontally, vertically, or diagonally. The game could also end in a draw if the grid is full while no player wins. The winning, draw, and losing outcomes of the player correspond to  $+1$ ,  $0$ , and  $-1$  rewards, respectively.
- Tic-Tac-Toe uses a  $3 \times 3$  grid as Figure 11 shows. Two players iteratively place a symbol on an available position. The player wins if any three of the symbols are in a row horizontally, vertically, or