[tab] /home/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/logs/logs_training_tokenizer/train_tokenizer_coinrun_3292307.log (log)
#!/usr/bin/env bash

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=24:00:00
#SBATCH --partition=accelerated
#SBATCH --account=hk-project-p0023960
#SBATCH --cpus-per-task=8
#SBATCH --gres=gpu:4
#SBATCH --output=logs/logs_training_tokenizer/%x_%j.log
#SBATCH --error=logs/logs_training_tokenizer/%x_%j.log
#SBATCH --mail-user=avocadoaling@gmail.com
#SBATCH --job-name=train_tokenizer_coinrun
#SBATCH --mail-type=ALL

# Log the sbatch script
cat $0

module unload mpi/openmpi/5.0
module unload devel/cuda/12.4
source .venv_jafar/bin/activate

job_name=$SLURM_JOB_NAME
slurm_job_id=$SLURM_JOB_ID

ws_dir='/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared'

CHECKPOINT_DIR=$ws_dir/data/checkpoints/$job_name_$slurm_job_id
LOG_DIR=$ws_dir/logs/$job_name_$slurm_job_id
mkdir -p $CHECKPOINT_DIR
mkdir -p $LOG_DIR

data_dir='/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data/coinrun/coinrun_tfrecords'

srun python train_tokenizer.py \
    --batch_size=192 \
    --experiment_name $job_name \
    --ckpt_dir $CHECKPOINT_DIR \
    --log_checkpoint_interval=1000 \
    --log_image_interval=10 \
    --image_height 64 \
    --image_width 64 \
    --log \
    --entity instant-uv \
    --data_dir $data_dir \
    --project jafar
GpuFreq=control_disabled
2025-06-25 19:08:17.627289: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1750871297.723938 1200206 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1750871297.736182 1200206 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1750871297.919551 1200206 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871297.919575 1200206 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871297.919577 1200206 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1750871297.919578 1200206 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
wandb: Currently logged in as: avocadoali (instant-uv) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.19.11
wandb: Run data is saved locally in /hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/wandb/run-20250625_190836-a7d8lqo4
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run train_tokenizer_coinrun
wandb: ⭐️ View project at https://wandb.ai/instant-uv/jafar
wandb: 🚀 View run at https://wandb.ai/instant-uv/jafar/runs/a7d8lqo4
W0000 00:00:1750871345.670879 1200206 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2025-06-25 19:10:34.248330: W external/xla/xla/hlo/transforms/simplifiers/hlo_rematerialization.cc:3022] Can't reduce memory use below 27.07GiB (29061856030 bytes) by rematerialization; only reduced to 55.73GiB (59841661312 bytes), down from 66.61GiB (71521158716 bytes) originally
2025-06-25 19:11:00.916184: W external/xla/xla/tsl/framework/bfc_allocator.cc:501] Allocator (GPU_0_bfc) ran out of memory trying to allocate 55.19GiB (rounded to 59262521088)requested by op 
2025-06-25 19:11:00.917792: W external/xla/xla/tsl/framework/bfc_allocator.cc:512] ****________________________________________________________________________________________________
E0625 19:11:00.918054 1200206 pjrt_stream_executor_client.cc:2917] Execution of replica 0 failed: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 59262520888 bytes. [tf-allocator-allocation-error='']
Running on 1 devices.
Starting training from step 0...
jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/train_tokenizer.py", line 224, in <module>
    train_state, loss, recon, metrics = train_step(train_state, inputs)
jaxlib._jax.XlaRuntimeError: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 59262520888 bytes.
jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/train_tokenizer.py", line 224, in <module>
    train_state, loss, recon, metrics = train_step(train_state, inputs)
jaxlib._jax.XlaRuntimeError: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 59262520888 bytes.
wandb: 
wandb: 🚀 View run train_tokenizer_coinrun at: https://wandb.ai/instant-uv/jafar/runs/a7d8lqo4
wandb: Find logs at: ../../../../../hkfs/home/project/hk-project-pai00039/tum_ind3695/projects/jafar_dyn/wandb/run-20250625_190836-a7d8lqo4/logs
srun: error: hkn0729: task 0: Exited with exit code 1

============================= JOB FEEDBACK =============================

Job ID: 3292307
Cluster: hk
User/Group: tum_ind3695/hk-project-pai00039
Account: hk-project-p0023960
State: FAILED (exit code 1)
Partition: accelerated
Nodes: 1
Cores per node: 8
Nodelist: hkn0729
CPU Utilized: 00:06:52
CPU Efficiency: 24.64% of 00:27:52 core-walltime
Job Wall-clock time: 00:03:29
Starttime: Wed Jun 25 19:07:36 2025
Endtime: Wed Jun 25 19:11:05 2025
Memory Utilized: 7.94 GB
Memory Efficiency: 0.00% of 0.00 MB
Energy Consumed: 114393 Joule / 31.7758333333333 Watthours
Average node power draw: 547.334928229665 Watt