PCOR

Training and offline evaluation code for PCOR, plus the baselines used in the paper.

PCOR (this method) is latent-conditioned PPO with actor FiLM: --actor-latent-film --actor-film-mod-scale 0.9 (mod_obs). The encoder produces z, and FiLM modulates h(obs) by gamma = 1 + 0.9 * tanh(gamma_net(z)).

PACE is the same latent pipeline without FiLM: actor MLP(cat(obs, z)).

Entry points: train_.py and offline_test_.py. GSCU lives under baselines/GSCU/. Always run from the repository root.

Method	Role	--exp-name	Implementation
PCOR	ours	pcor	--actor-latent-film --actor-film-mod-scale 0.9
PACE	baseline	pace	same latent PPO as PCOR, no FiLM flags
PACE-TF	baseline	pace_tf_dropout (Overcooked) / pace_tf (MPE)	--encoder-base tf
LILI	baseline	lili	--auxiliary-transition-pred-coef 1.0 --self-action-mode --last-episode-only --collect-next-obs
PPO	baseline	ppo	--joint-training --recurrent-policy --share-actor-critic
Generalist	baseline	generalist	PPO + --use-meta-episode
LIAM	baseline	liam	PPO + --collect-peer-traj --auxiliary-peer-obs-pred-coef 1.0 --auxiliary-peer-act-pred-coef 1.0
LIAM-X	baseline	liamx	LIAM + --use-meta-episode
GSCU	baseline	gscu_oc_s<seed> / gscu_mpe_s<seed>	baselines/GSCU/ (VAE + conditional RL)

Full commands: Overcooked §2.2–2.3, Predator-Prey §2.4.

1. Environment setup

Requires Conda, Python 3.8, and CUDA (training uses GPU by default). Install as follows:

conda create -n pcor python=3.8 setuptools=65.6.3
conda activate pcor

conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
conda install scikit-learn==1.0.2 -c conda-forge
conda install -y psutil pandas matplotlib cloudpickle "gymnasium>=0.28.1,<0.29" "typing_extensions>=4.0" -c conda-forge

pip install --default-timeout=120 wheel==0.38.4
pip install --default-timeout=120 -e ./gym-0.21.0
pip install --default-timeout=120 PettingZoo==1.9.0 pygame==2.0.1 wandb==0.14.0 tqdm==4.65.0 stable-baselines3==1.7.0 seaborn==0.13.0 pyglet==1.5.27

cd environment/overcooked/gym_cooking/rebar
pip install -e .
cd ../..
pip install -e .
cd ../..

You can also run install.sh. Use source install.sh in an interactive shell that already has Conda initialized; a non-interactive bash install.sh will not persist conda activate.

Sanity check:

python -c "import torch, gym; print(torch.__version__, torch.cuda.is_available(), gym.__version__)"

2. Training

After conda activate pcor, run python train_.py ... from the repo root.

Logs and checkpoints:

logs/<Env>/ppo_<exp-name>_seed<N>/
  ├── <exp-name>.log
  └── ppo/
        ├── latest.pt
        └── <env_steps>.pt

If that directory already exists, an interactive run asks whether to overwrite; background / nohup needs --overwrite-log.

--num-processes must be divisible by --train-pool-size (Overcooked: 18 → 18/36/54/72; MPE: 16 → 16/32/48/64). Reduce it if the machine has fewer CPU cores.

Commands below use seed 5. Change --seed for other seeds.

2.1 Overcooked: shared flags

conda activate pcor

OC_ENV=(
  --env-name Overcooked
  --env-config './environment/overcooked/config/forced_coord_broccoli_dense_reward_cross_recipe_upper_random_init_step01_el40_partial.yaml'
  --player-id 0 --recipe-type cross
)
OC_PPO=(
  --algo ppo --num-steps 1000 --num-processes 72 --use-gae --eps 1e-8 --use-proper-time-limits
  --lr 1e-3 --entropy-coef 3e-2 --value-loss-coef 0.5 --num-epochs 4 --num-mini-batch 18 --gamma 0.99 --gae-lambda 0.95
  --max-grad-norm 15.0
  --num-env-steps 30000000
  --save-interval 5000000 --log-interval 10000 --eval-interval 100000 --eval-episodes 5
  --train-pool-size 18 --eval-pool-size 9 --rule-based-opponents 18
  --seed 5
)
# Latent methods (PCOR / PACE / PACE-TF / LILI)
OC_LATENT=(
  --latent-training --e2e-obj --deterministic-latent --use-latent-critic
  --pre-hidden-dims 128 128 --agg-func mean --post-hidden-dims 128
  --latent-dim 128 --kl-coef 0
  --hidden-dims 128 128 --act-func relu
  --history-middle-sampling --merge-encoder-computation
  --history-full-size 5 --history-size 5 --history-use-episodes
  --include-current-episode --use-meta-episode --self-obs-mode
  --clear-history-on-full --separate-history
)
OC_PACE_AUX=(
  --auxiliary-policy-cls-coef 1.0
  --policy-cls-reward-coef 0.2 --policy-cls-reward-type accuracy --policy-cls-reward-mode max_full
  --policy-cls-warmup-steps 1000000 --policy-cls-reward-decay-steps 25000000
  --visit-reward-coef 0.0 --visit-reward-type episode
)
# RNN methods (PPO / Generalist / LIAM / LIAM-X)
OC_RNN=(
  --joint-training --hidden-dims 128 128 --act-func relu
  --recurrent-policy --rnn-hidden-dim 128 --rnn-chunk-length 20 --share-actor-critic
  --history-size 5
)

2.2 Overcooked: PCOR (ours)

python train_.py "${OC_ENV[@]}" "${OC_PPO[@]}" "${OC_LATENT[@]}" "${OC_PACE_AUX[@]}" \
  --encoder-base mlp \
  --exp-name pcor \
  --actor-latent-film --actor-film-mod-scale 0.9

2.3 Overcooked: baselines

PACE — same as PCOR, but actor MLP(cat(obs, z)) (no FiLM):

python train_.py "${OC_ENV[@]}" "${OC_PPO[@]}" "${OC_LATENT[@]}" "${OC_PACE_AUX[@]}" \
  --encoder-base mlp \
  --exp-name pace

PACE-TF — PACE with a transformer encoder:

python train_.py "${OC_ENV[@]}" "${OC_PPO[@]}" "${OC_LATENT[@]}" "${OC_PACE_AUX[@]}" \
  --encoder-base tf --tf-n-layers 3 --tf-n-heads 1 --tf-hidden-dim 128 --tf-ff-dim 512 --tf-dropout 0.1 \
  --exp-name pace_tf_dropout

LILI — latent encoder + next-obs / transition prediction (no policy-classifier reward):

python train_.py "${OC_ENV[@]}" "${OC_PPO[@]}" "${OC_LATENT[@]}" \
  --encoder-base mlp \
  --auxiliary-transition-pred-coef 1.0 --self-action-mode --last-episode-only --collect-next-obs \
  --visit-reward-coef 0.0 --visit-reward-type episode \
  --exp-name lili

PPO:

python train_.py "${OC_ENV[@]}" "${OC_PPO[@]}" "${OC_RNN[@]}" \
  --exp-name ppo

Generalist — PPO + meta-episode:

python train_.py "${OC_ENV[@]}" "${OC_PPO[@]}" "${OC_RNN[@]}" \
  --use-meta-episode \
  --exp-name generalist

LIAM — PPO + peer observation/action prediction:

python train_.py "${OC_ENV[@]}" "${OC_PPO[@]}" "${OC_RNN[@]}" \
  --all-has-last-action --collect-peer-traj \
  --auxiliary-peer-obs-pred-coef 1.0 --auxiliary-peer-act-pred-coef 1.0 \
  --exp-name liam

LIAM-X — LIAM + meta-episode:

python train_.py "${OC_ENV[@]}" "${OC_PPO[@]}" "${OC_RNN[@]}" \
  --all-has-last-action --collect-peer-traj \
  --auxiliary-peer-obs-pred-coef 1.0 --auxiliary-peer-act-pred-coef 1.0 \
  --use-meta-episode \
  --exp-name liamx

GSCU — three-stage pipeline (embedding data → VAE → conditional RL). From the repo root, with PYTHONPATH set to the repo:

export PYTHONPATH="${PWD}${PYTHONPATH:+:$PYTHONPATH}"
VER=gscu_oc_s5
GSCU_ARGS=(--env_config './environment/overcooked/config/forced_coord_broccoli_dense_reward_cross_recipe_upper_random_init_step01_el40_partial.yaml' --player-id 0 --GSCU_config Normal --seed 5)

python baselines/GSCU/overcooked/embedding_learning/data_generation.py -v "$VER" "${GSCU_ARGS[@]}"
python baselines/GSCU/overcooked/embedding_learning/train_vae.py -v "$VER" "${GSCU_ARGS[@]}"

# Use the encoder filename written under baselines/GSCU/overcooked/model_params/VAE/
python baselines/GSCU/overcooked/conditional_RL/train_conditional_rl.py \
  -v "$VER" "${GSCU_ARGS[@]}" \
  --train_steps 30000000 --num_steps 100 \
  --log-interval 10000 --eval-interval 100000 --save-interval 5000000 --eval-episodes 100 \
  -e encoder_vae_param_${VER}_<timestamp>.pt

Checkpoint: baselines/GSCU/overcooked/model_params/RL/seed5/params_${VER}.pt.

Background example:

nohup python train_.py ... --overwrite-log > logs/pcor_seed5.nohup 2>&1 &

2.4 Predator-Prey (MPE)

Shared flags (15M steps; pool size 16):

conda activate pcor

MPE_ENV=(
  --env-name MPE
  --scenario simple_tag_multi_partial --num-agents 4 --num-good-agents 2
  --obs-radius 0.2 --horizon 50 --shaped-reward --collide-reward-once --watch-tower
  --player-id 0
)
MPE_PPO=(
  --algo ppo --num-steps 1000 --num-processes 64 --use-gae --eps 1e-8 --use-proper-time-limits
  --lr 1e-3 --entropy-coef 3e-2 --value-loss-coef 0.5 --num-epochs 2 --num-mini-batch 150 --gamma 0.99 --gae-lambda 0.95
  --max-grad-norm 15.0
  --num-env-steps 15000000
  --save-interval 5000000 --log-interval 10000 --eval-interval 100000 --eval-episodes 5
  --train-pool-size 16 --eval-pool-size 24 --rule-based-opponents 16 --separate-patterns
  --seed 5
)
MPE_LATENT=(
  --latent-training --e2e-obj --deterministic-latent --use-latent-critic
  --pre-hidden-dims 128 128 --agg-func mean --post-hidden-dims 128
  --latent-dim 128 --kl-coef 0
  --hidden-dims 128 128 --act-func relu
  --history-middle-sampling --merge-encoder-computation
  --history-full-size 5 --history-size 5 --history-use-episodes
  --include-current-episode --use-meta-episode --self-obs-mode --self-action-mode
  --clear-history-on-full --separate-history
)
MPE_PACE_AUX=(
  --auxiliary-policy-cls-coef 1.0
  --policy-cls-reward-coef 0.1 --policy-cls-reward-type accuracy --policy-cls-reward-mode full
  --policy-cls-warmup-steps 500000 --policy-cls-reward-decay-steps 15000000
  --visit-reward-type interaction --visit-reward-coef 0.0
)
MPE_RNN=(
  --joint-training --hidden-dims 128 128 --act-func relu
  --recurrent-policy --rnn-hidden-dim 128 --rnn-chunk-length 20 --share-actor-critic
  --history-size 5
)

PCOR:

python train_.py "${MPE_ENV[@]}" "${MPE_PPO[@]}" "${MPE_LATENT[@]}" "${MPE_PACE_AUX[@]}" \
  --encoder-base mlp \
  --exp-name pcor \
  --actor-latent-film --actor-film-mod-scale 0.9

PACE:

python train_.py "${MPE_ENV[@]}" "${MPE_PPO[@]}" "${MPE_LATENT[@]}" "${MPE_PACE_AUX[@]}" \
  --encoder-base mlp \
  --exp-name pace

PACE-TF:

python train_.py "${MPE_ENV[@]}" "${MPE_PPO[@]}" "${MPE_LATENT[@]}" "${MPE_PACE_AUX[@]}" \
  --encoder-base tf --tf-n-layers 3 --tf-n-heads 1 --tf-hidden-dim 128 --tf-ff-dim 512 --tf-dropout 0.1 \
  --exp-name pace_tf

LILI:

python train_.py "${MPE_ENV[@]}" "${MPE_PPO[@]}" "${MPE_LATENT[@]}" \
  --encoder-base mlp \
  --auxiliary-transition-pred-coef 1.0 --self-action-mode --last-episode-only --collect-next-obs \
  --visit-reward-coef 0.0 --visit-reward-type episode \
  --exp-name lili

PPO / Generalist / LIAM / LIAM-X:

python train_.py "${MPE_ENV[@]}" "${MPE_PPO[@]}" "${MPE_RNN[@]}" --exp-name ppo
python train_.py "${MPE_ENV[@]}" "${MPE_PPO[@]}" "${MPE_RNN[@]}" --use-meta-episode --exp-name generalist
python train_.py "${MPE_ENV[@]}" "${MPE_PPO[@]}" "${MPE_RNN[@]}" \
  --all-has-last-action --collect-peer-traj \
  --auxiliary-peer-obs-pred-coef 1.0 --auxiliary-peer-act-pred-coef 1.0 \
  --exp-name liam
python train_.py "${MPE_ENV[@]}" "${MPE_PPO[@]}" "${MPE_RNN[@]}" \
  --all-has-last-action --collect-peer-traj \
  --auxiliary-peer-obs-pred-coef 1.0 --auxiliary-peer-act-pred-coef 1.0 \
  --use-meta-episode \
  --exp-name liamx

GSCU (MPE):

export PYTHONPATH="${PWD}${PYTHONPATH:+:$PYTHONPATH}"
VER=gscu_mpe_s5
python baselines/GSCU/predator_prey/embedding_learning/data_generation.py -v "$VER" -seed 5
python baselines/GSCU/predator_prey/embedding_learning/train_vae.py -v "$VER"
python baselines/GSCU/predator_prey/conditional_RL/train_conditional_rl.py \
  -v "$VER" -seed 5 -e encoder_vae_param_${VER}_<timestamp>.pt \
  --num-env-steps 15000000 --log-interval 10000 --eval-interval 100000 --save-interval 5000000 --eval-episodes 100

3. Offline evaluation

offline_test_.py evaluates train_.py checkpoints against a fixed opponent pool. Architecture is loaded from the .pt file, but env, opponent-pool, and history flags must match training.

Checkpoint directory (default, from --env-name --algo --exp-name --seed):

logs/<Env>/ppo_<exp-name>_seed<N>/ppo/

Or pass --offline-checkpoint-dir explicitly.

• --offline-eval-mode episode: latest.pt only (default)
• --offline-eval-mode env_steps: every numeric <steps>.pt
• --offline-opponent-sets train eval all (eval = test)
• --test-episodes per opponent (32 if omitted)

Outputs: logs/<Env>/ppo_<exp-name>_seed<N>/offline_test_sweep/offline_summary.csv

3.1 Overcooked: PCOR

Reuse the Overcooked arrays from §2.1, then:

python offline_test_.py "${OC_ENV[@]}" "${OC_LATENT[@]}" \
  --algo ppo --exp-name pcor --seed 5 \
  --train-pool-size 18 --eval-pool-size 9 --rule-based-opponents 18 \
  --encoder-base mlp --latent-training --e2e-obj \
  --offline-eval-mode episode \
  --offline-opponent-sets train eval all \
  --test-episodes 32

For a baseline, keep the matching training flags and change --exp-name (pace, lili, ppo, …). Sweep intermediate checkpoints with --offline-eval-mode env_steps.

Explicit directory:

  --offline-checkpoint-dir logs/Overcooked/ppo_pcor_seed5/ppo

3.2 MPE: PCOR

python offline_test_.py "${MPE_ENV[@]}" "${MPE_LATENT[@]}" \
  --algo ppo --exp-name pcor --seed 5 \
  --train-pool-size 16 --eval-pool-size 24 --rule-based-opponents 16 --separate-patterns \
  --encoder-base mlp --latent-training --e2e-obj \
  --offline-eval-mode episode \
  --offline-opponent-sets train eval all \
  --test-episodes 32

3.3 GSCU (Overcooked)

python baselines/GSCU/overcooked/online_test/offline_eval.py \
  --env-config './environment/overcooked/config/forced_coord_broccoli_dense_reward_cross_recipe_upper_random_init_step01_el40_partial.yaml' \
  --player-id 0 --recipe-type cross --seed 5 \
  --train-pool-size 18 --eval-pool-size 9 --rule-based-opponents 18 \
  -v gscu_oc_s5 \
  -e encoder_vae_param_gscu_oc_s5_<timestamp>.pt \
  -d decoder_param_gscu_oc_s5_<timestamp>.pt \
  -r params_gscu_oc_s5.pt \
  --offline-opponent-sets train eval all --test-episodes 32

Inspect a train_.py summary with:

column -s, -t logs/Overcooked/ppo_pcor_seed5/offline_test_sweep/offline_summary.csv

mean_reward / mean_success_rate (Overcooked) or mean_watchtower_visits_per_interaction (MPE) are averages over the corresponding opponent pool.