SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 8: Hyper-parameters for Google Football.
ppo_epoch	15	num_minibatch	2	entropy coef	0.01
use_gae	True	gamma	0.99	gae_lambda	0.95
critic_lr	7e-4	lr	7e-4	weight Decay	0
adam血脂	1e-5	n-rollout Threads	50	ppo Episode_length	200
data_chunk_length	10	steps	25M	max_grad_norm	0.5
bcRegularExpressioncoef	5.0	bc_batch_size	256	network	RNN
G.3 Computing Resources
Each seed is run on a GPU server with one NVIDIA P100 GPU, and Intel(R) Xeon(R) Gold 6145 CPU @ 2.00GHz CPU. Each run can finish within 24 hours.
G.4 Target Models & Dataset Collection
For all the environments, we user different runs of self-play to get the group of targets. In order to ensure a fair comparison, we use the exactly the same number of random seeds to generate these targets, without any selection. In our experiments, we observe that different runs with different seeds can produce diverse targets. For instance, in Google Football, different targets may have different tendencies to pick which side to start a attack (left or right). Moreover, as can be seen from Table 4, the difficulties to exploit these targets are diverse. For MuJoCo and Google Football, we use dataset consisting of 5 trajectories. We collect these trajectories using the target model together with a rollout policy. We observe that the 3 kinds of rollout policies: (1) random policy, (2) environment's bot, (3) target itself, do not have significant impacts on the experiment results in the MuJoCo and Google Football environments.
G.5 Visualization
Figure 8: The visualization of policy behavior in MuJoCo. Top. The trajectory in dataset, where the blocker is the real opponent policy  $\pi_B$  and the runner is the policy which collects the dataset. Middle. The trajectory during the training of CSP. Both agents are controlled by CSP. Bottom. The evaluation of CSP's runner with the real opponent blocker.
In Figure 8, we further consolidate our claim by visualizing the policy behavior in MuJoCo. As can be seen in this example, in a trajectory contained in dataset, the runner (blue agent) typically makes a detour through the upper path to avoid being pushed down by the blocker (red agent). The opponent blocker also has the tendency to walk upwards to stop the runner. The runner trained by CSP learns to exploit the policy represented by the dataset, and prefers making a detour through the lower path, as can be seen from the middle and bottom lines of replays. Although the behavior of runner going downwards is not contained in the dataset, the blocker trained by CSP can still learn to stop such runner, which makes the runner even more robust and stronger than the real target model. Therefore, when confronted with the real opponent blocker, who only knows how to defend the upper path, the runner trained by CSP steadily wins the game.
G.6 Reward.
For predator-prey, we use the environment's original reward. For MuJoCo, we use dense rewards for locomotion learning as in previous works [2, 10]. In Google Football, we use both the sparse scoring reward as well as the dense checkpoint reward which awards the attacker according to the ball handler's distance to the goal. Although the rewards for the attacker are provided by the environment, Google Football does not provide rewards for the defender team in academy scenarios. In our experiments, we just use the negative of the attacker's reward to train defender's policy to make the game zero-sum.
G.7 Explanation of Training & Testing Performances
The observed lower training performance of CSP than the baseline, as shown in Figure 1, is expected behavior since the opponent model used by CSP during training differs from that of BC-First. In BC-First, the opponent model is solely pre-trained on a dataset and remains fixed, leading to significant vulnerabilities in out-of-distribution states which can be easily exploited. Conversely, in CSP, the opponent model is trained in an adversarial manner simultaneously with the exploitation policy. Due to its evolutionary nature, it can compensate for its vulnerabilities and becomes significantly more
challenging to exploit. Consequently, the training performance of BC-First is higher. During testing, both exploitation policies undergo evaluation using the same real target model.