[tab] /fast/project/HFMI_ProFOUND/mihir/logs/train/dyn/diffusion/train_dyn_dit_32509.log (log)
#!/usr/bin/env bash

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=24:00:00
#SBATCH --cpus-per-task=5
#SBATCH --gres=gpu:1
#SBATCH --mem=100GB
#SBATCH --output=/fast/project/HFMI_ProFOUND/mihir/logs/train/dyn/diffusion/%x_%j.log
#SBATCH --error=/fast/project/HFMI_ProFOUND/mihir/logs/train/dyn/diffusion/%x_%j.log
#SBATCH --job-name=train_dyn_dit

cat $0

source .venv/bin/activate

array_records_dir_train=/fast/project/HFMI_ProFOUND/mihir/data/data_coinrun/coinrun_episodes_50m/train
array_records_dir_val=/fast/project/HFMI_ProFOUND/mihir/data/data_coinrun/coinrun_episodes_50m/val

job_name=$SLURM_JOB_NAME
slurm_job_id=$SLURM_JOB_ID

CHECKPOINT_DIR=/fast/project/HFMI_ProFOUND/mihir/checkpoints/dyn/$job_name/$slurm_job_id
mkdir -p $CHECKPOINT_DIR

srun python jasmine/train_dynamics.py \
    --save_ckpt \
    --image_height=64 \
    --image_width=64 \
    --ckpt_dir $CHECKPOINT_DIR \
    --batch_size=36 \
    --max_lr=1e-4 \
    --val_interval 1000 \
    --dyna_type=diffusion \
    --use_gt_actions \
    --diffusion_denoise_steps=128 \
    --name=train_dyn_dit_coinrun_lr_1e-4_gt_actions_$slurm_job_id \
    --tags dyn diffusion coinrun lr_1e-4 gt_actions \
    --entity instant-uv \
    --project jafar \
    --data_dir $array_records_dir_train \
    --val_data_dir $array_records_dir_val \

/var/spool/slurm/job32509/slurm_script: line 15: .venv/bin/activate: No such file or directory
wandb: Currently logged in as: mihir-mahajan2002 (instant-uv) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: setting up run 06avn49b
wandb: Tracking run with wandb version 0.22.1
wandb: Run data is saved locally in /fast/home/mihir.mahajan/Projects/jasmine_runner_1/wandb/run-20251022_190919-06avn49b
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run train_dyn_dit_coinrun_lr_1e-4_gt_actions_32509
wandb: ⭐️ View project at https://wandb.ai/instant-uv/jafar
wandb: 🚀 View run at https://wandb.ai/instant-uv/jafar/runs/06avn49b
WARNING:absl:Path LICENSE could not be identified as a finalized checkpoint path using <class 'orbax.checkpoint._src.path.atomicity.AtomicRenameTemporaryPath'>. Got error: Expected AtomicRenameTemporaryPath (LICENSE) to be a directory.
WARNING:absl:Path pyproject.toml could not be identified as a finalized checkpoint path using <class 'orbax.checkpoint._src.path.atomicity.AtomicRenameTemporaryPath'>. Got error: Expected AtomicRenameTemporaryPath (pyproject.toml) to be a directory.
WARNING:absl:Path README.md could not be identified as a finalized checkpoint path using <class 'orbax.checkpoint._src.path.atomicity.AtomicRenameTemporaryPath'>. Got error: Expected AtomicRenameTemporaryPath (README.md) to be a directory.
Running on 1 devices.
Counting all components: ['action_embed', 'dynamics', 'tokenizer']
Parameter counts:
{'action_embed': 192, 'dynamics': 41050160, 'tokenizer': 34489696, 'total': 75540048}
Total memory size: 39.7 GB, Output size: 0.9 GB, Temp size: 38.8 GB, Argument size: 0.9 GB, Host temp size: 0.0 GB.
FLOPs: 2.600e+12, Bytes: 2.539e+11 (236.4 GB), Intensity: 10.2 FLOPs/byte
Starting training from step 0...

Memstats: After params initialized:
	Using (GB) 0.93 / 71.27 (1.304897%) on cuda:0
Calculating validation metrics...
/fast/home/mihir.mahajan/Projects/jasmine/.venv/lib/python3.12/site-packages/wandb/sdk/data_types/image.py:91: RuntimeWarning: invalid value encountered in cast
  return data.clip(0, 255).astype(np.uint8)
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 12, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 7, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 10, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 6, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 15, which is shorter than the requested sequence length 16.
Filtering out episode with length 11, which is shorter than the requested sequence length 16.
Filtering out episode with length 4, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 3, which is shorter than the requested sequence length 16.
Filtering out episode with length 13, which is shorter than the requested sequence length 16.
Filtering out episode with length 14, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
Filtering out episode with length 1, which is shorter than the requested sequence length 16.
Filtering out episode with length 8, which is shorter than the requested sequence length 16.
Filtering out episode with length 2, which is shorter than the requested sequence length 16.
Filtering out episode with length 9, which is shorter than the requested sequence length 16.
Filtering out episode with length 5, which is shorter than the requested sequence length 16.
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[2025-10-22T19:32:36.796] error: *** JOB 32509 ON hai007 CANCELLED AT 2025-10-22T19:32:36 DUE to SIGNAL Terminated ***
[2025-10-22T19:32:36.797] error: *** STEP 32509.0 ON hai007 CANCELLED AT 2025-10-22T19:32:36 DUE to SIGNAL Terminated ***