[tab] /hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/logs/logs_mihir/train_tokenizer_model_size_scaling_37M_3313565.log (log)
#!/usr/bin/env bash

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --time=15:00:00
#SBATCH --partition=accelerated
#SBATCH --cpus-per-task=5
#SBATCH --gres=gpu:4
#SBATCH --output=/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/logs/logs_mihir/%x_%j.log
#SBATCH --error=/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/logs/logs_mihir/%x_%j.log
#SBATCH --job-name=train_tokenizer_model_size_scaling_37M

# Log the sbatch script
cat $0

module unload mpi/openmpi/5.0
module unload devel/cuda/12.4
source .venv/bin/activate

tf_records_dir=/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data/open_ai_minecraft_tfrecord
ws_dir='/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/'

job_name=$SLURM_JOB_NAME
slurm_job_id=$SLURM_JOB_ID

CHECKPOINT_DIR=$ws_dir/checkpoints/$job_name_$slurm_job_id
mkdir -p $CHECKPOINT_DIR

env | grep SLURM

srun python train_tokenizer.py \
    --ckpt_dir $CHECKPOINT_DIR \
    --batch_size=96 \
    --min_lr=1e-4 \
    --max_lr=1e-4 \
    --log_image_interval=500 \
    --log_checkpoint_interval=500 \
    --log \
    --name=tokenizer-model-size-scaling-38M-$slurm_job_id \
    --tags tokenizer model-size-scaling 38M \
    --entity instant-uv \
    --project jafar \
    --data_dir $tf_records_dir
/var/spool/slurmd/job3313565/slurm_script: line 18: .venv/bin/activate: No such file or directory
SLURM_JOB_USER=tum_cte0515
SLURM_TASKS_PER_NODE=4(x2)
SLURM_JOB_UID=999226
SLURM_TASK_PID=407251
SLURM_JOB_GPUS=0,1,2,3
SLURM_LOCALID=0
SLURM_SUBMIT_DIR=/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar_jobs
SLURMD_NODENAME=hkn0427
SLURM_JOB_START_TIME=1751512068
SLURM_CLUSTER_NAME=hk
SLURM_JOB_END_TIME=1751566068
SLURM_CPUS_ON_NODE=24
SLURM_JOB_CPUS_PER_NODE=24(x2)
SLURM_GPUS_ON_NODE=4
SLURM_GTIDS=0
SLURM_JOB_PARTITION=accelerated
SLURM_TRES_PER_TASK=cpu=5
SLURM_OOM_KILL_STEP=0
SLURM_JOB_NUM_NODES=2
SLURM_JOBID=3313565
SLURM_JOB_QOS=normal
SLURM_PROCID=0
SLURM_CPUS_PER_TASK=5
SLURM_NTASKS=8
SLURM_TOPOLOGY_ADDR=hkibb.hkibbi1.hkibbi1e13.hkn0427
SLURM_TOPOLOGY_ADDR_PATTERN=switch.switch.switch.node
SLURM_SCRIPT_CONTEXT=prolog_task
SLURM_NODELIST=hkn[0427,0512]
SLURM_JOB_ACCOUNT=hk-project-p0023960
SLURM_PRIO_PROCESS=0
SLURM_NPROCS=8
SLURM_NNODES=2
SLURM_SUBMIT_HOST=hkn1991.localdomain
SLURM_JOB_ID=3313565
SLURM_NODEID=0
SLURM_CONF=/etc/slurm/slurm.conf
SLURM_JOB_NAME=train_tokenizer_model_size_scaling_37M
SLURM_NTASKS_PER_NODE=4
SLURM_JOB_GID=502226
SLURM_JOB_NODELIST=hkn[0427,0512]
GpuFreq=control_disabled
GpuFreq=control_disabled
2025-07-03 05:08:18.872363: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-07-03 05:08:18.872465: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-07-03 05:08:18.880385: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-07-03 05:08:18.881133: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1751512098.885489 1996265 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1751512098.885687 1996263 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1751512098.890202 1996265 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
E0000 00:00:1751512098.890496 1996263 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1751512098.893232 1996264 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1751512098.894694 1996262 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1751512098.897804 1996264 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
E0000 00:00:1751512098.898943 1996262 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1751512098.904052 1996263 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.904069 1996263 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.904071 1996263 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.904072 1996263 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.904007 1996265 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.904022 1996265 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.904024 1996265 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.904025 1996265 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.911233 1996264 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.911249 1996264 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.911250 1996264 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.911252 1996264 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.911616 1996262 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.911632 1996262 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.911633 1996262 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512098.911635 1996262 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512102.159976 1996265 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
W0000 00:00:1751512102.194663 1996264 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
W0000 00:00:1751512102.194824 1996262 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
W0000 00:00:1751512102.197642 1996263 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2025-07-03 05:08:32.084282: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-07-03 05:08:32.084284: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-07-03 05:08:32.084281: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-07-03 05:08:32.084277: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1751512112.113479  407324 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1751512112.113674  407322 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1751512112.113674  407325 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1751512112.113888  407323 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1751512112.119218  407322 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
E0000 00:00:1751512112.119216  407323 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
E0000 00:00:1751512112.119218  407324 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
E0000 00:00:1751512112.119212  407325 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1751512112.203959  407322 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.203994  407322 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.203996  407322 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.203998  407322 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.203955  407323 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.203999  407323 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.204001  407323 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.204003  407323 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.203953  407324 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.203994  407324 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.203996  407324 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.203998  407324 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.203956  407325 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.203995  407325 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.203998  407325 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512112.204000  407325 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1751512132.265995  407322 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
W0000 00:00:1751512132.265923  407323 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
W0000 00:00:1751512132.265926  407324 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
W0000 00:00:1751512132.265921  407325 gpu_device.cc:2341] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
wandb: Currently logged in as: mihir-mahajan2002 (instant-uv) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.19.11
wandb: Run data is saved locally in /hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar_jobs/wandb/run-20250703_050936-1kuvkzad
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run tokenizer-model-size-scaling-38M-3313565
wandb: ⭐️ View project at https://wandb.ai/instant-uv/jafar
wandb: 🚀 View run at https://wandb.ai/instant-uv/jafar/runs/1kuvkzad
Running on 8 devices.
Counting all components: ['encoder', 'vq', 'decoder']
Parameter counts:
{'encoder': 18978432, 'vq': 32768, 'decoder': 18978416, 'total': 37989616}
Starting training from step 0...
Step 0, loss: 0.34403467178344727
Step 1, loss: 0.20522570610046387
Step 2, loss: 0.13297320902347565
Step 3, loss: 0.12472856044769287
Step 4, loss: 0.11909522116184235
Step 5, loss: 0.1155850738286972
Step 6, loss: 0.1258772909641266
Step 7, loss: 0.13581234216690063
Step 8, loss: 0.12142564356327057
Step 9, loss: 0.12192649394273758
Step 10, loss: 0.10417830944061279
Step 11, loss: 0.11738790571689606
Step 12, loss: 0.11391294747591019
Step 13, loss: 0.1308845579624176
Step 14, loss: 0.10983618348836899
Step 15, loss: 0.11098787933588028
Step 16, loss: 0.10668867081403732
Step 17, loss: 0.1114255040884018
Step 18, loss: 0.09659188240766525
Step 19, loss: 0.10099179297685623
Step 20, loss: 0.10332012176513672
Step 21, loss: 0.10993895679712296
Step 22, loss: 0.11687967926263809
Step 23, loss: 0.09648826718330383
Step 24, loss: 0.08426374942064285
Step 25, loss: 0.11532411724328995
Step 26, loss: 0.10817211866378784
Step 27, loss: 0.10362100601196289
Step 28, loss: 0.09701235592365265
Step 29, loss: 0.09972862899303436
Step 30, loss: 0.09519169479608536
Step 31, loss: 0.09001486748456955
Step 32, loss: 0.08831518888473511
Step 33, loss: 0.08256299793720245
Step 34, loss: 0.0901721939444542
Step 35, loss: 0.0847548171877861
Step 36, loss: 0.08388174325227737
Step 37, loss: 0.08075693994760513
Step 38, loss: 0.08081647008657455
Step 39, loss: 0.10267360508441925
Step 40, loss: 0.09803157299757004
Step 41, loss: 0.08619318157434464
Step 42, loss: 0.09996463358402252
Step 43, loss: 0.08521658182144165
Step 44, loss: 0.10208437591791153
Step 45, loss: 0.08445227146148682
Step 46, loss: 0.11115314066410065
Step 47, loss: 0.08237376809120178
Step 48, loss: 0.09492920339107513
Step 49, loss: 0.09434117376804352
Step 50, loss: 0.09538457542657852
Step 51, loss: 0.10146772116422653
Step 52, loss: 0.10333891957998276
Step 53, loss: 0.09780734777450562
Step 54, loss: 0.0894136130809784
Step 55, loss: 0.08602304011583328
Step 56, loss: 0.08182331174612045
Step 57, loss: 0.08126511424779892
Step 58, loss: 0.09783581644296646
Step 59, loss: 0.09782543033361435
Step 60, loss: 0.09924279153347015
Step 61, loss: 0.08977486938238144
Step 62, loss: 0.10259989649057388
Step 63, loss: 0.10501067340373993
Step 64, loss: 0.10390648245811462
Step 65, loss: 0.08770249038934708
Step 66, loss: 0.09540095180273056
Step 67, loss: 0.09132639318704605
Step 68, loss: 0.10009634494781494
Step 69, loss: 0.09285572171211243
Step 70, loss: 0.08500776439905167
Step 71, loss: 0.09973502159118652
Step 72, loss: 0.08392619341611862
Step 73, loss: 0.09875445067882538
Step 74, loss: 0.10540761053562164
Step 75, loss: 0.09229494631290436
Step 76, loss: 0.10541470348834991
Step 77, loss: 0.08958648890256882
Step 78, loss: 0.08453366160392761
Step 79, loss: 0.090049609541893
Step 80, loss: 0.09214366227388382
Running on 8 devices.
Counting all components: ['encoder', 'vq', 'decoder']
Parameter counts:
{'encoder': 18978432, 'vq': 32768, 'decoder': 18978416, 'total': 37989616}
Starting training from step 0...
Step 0, loss: 0.34403467178344727
Step 1, loss: 0.20522570610046387
Step 2, loss: 0.13297320902347565
Step 3, loss: 0.12472856044769287
Step 4, loss: 0.11909522116184235
Step 5, loss: 0.1155850738286972
Step 6, loss: 0.1258772909641266
Step 7, loss: 0.13581234216690063
Step 8, loss: 0.12142564356327057
Step 9, loss: 0.12192649394273758
Step 10, loss: 0.10417830944061279
Step 11, loss: 0.11738790571689606
Step 12, loss: 0.11391294747591019
Step 13, loss: 0.1308845579624176
Step 14, loss: 0.10983618348836899
Step 15, loss: 0.11098787933588028
Step 16, loss: 0.10668867081403732
Step 17, loss: 0.1114255040884018
Step 18, loss: 0.09659188240766525
Step 19, loss: 0.10099179297685623
Step 20, loss: 0.10332012176513672
Step 21, loss: 0.10993895679712296
Step 22, loss: 0.11687967926263809
Step 81, loss: 0.0938016027212143
Step 82, loss: 0.09607720375061035
Step 83, loss: 0.09525217115879059
Step 84, loss: 0.09727976471185684
Step 85, loss: 0.0878567323088646
Step 86, loss: 0.09850167483091354
Step 87, loss: 0.10143165290355682
Step 88, loss: 0.08512572199106216
Step 89, loss: 0.09779776632785797
Step 90, loss: 0.08772671967744827
Step 91, loss: 0.09805448353290558
Step 92, loss: 0.10006199777126312
Step 93, loss: 0.0881110280752182
Step 94, loss: 0.08710669726133347
Step 95, loss: 0.10951592773199081
Step 96, loss: 0.0913202092051506
Step 97, loss: 0.09101113677024841
Step 98, loss: 0.1027674525976181
Step 99, loss: 0.10735452175140381
Step 100, loss: 0.0928899422287941
Step 101, loss: 0.10774534195661545
Step 102, loss: 0.11379466950893402
Step 103, loss: 0.12731407582759857
Step 104, loss: 0.08464271575212479
Step 105, loss: 0.09762269258499146
Step 106, loss: 0.08544913679361343
Step 107, loss: 0.08664936572313309
Step 108, loss: 0.09289372712373734
Step 109, loss: 0.09938964247703552
Step 23, loss: 0.09648826718330383
Step 24, loss: 0.08426374942064285
Step 25, loss: 0.11532411724328995
Step 26, loss: 0.10817211866378784
Step 27, loss: 0.10362100601196289
Step 28, loss: 0.09701235592365265
Step 29, loss: 0.09972862899303436
Step 30, loss: 0.09519169479608536
Step 31, loss: 0.09001486748456955
Step 32, loss: 0.08831518888473511
Step 33, loss: 0.08256299793720245
Step 34, loss: 0.0901721939444542
Step 35, loss: 0.0847548171877861
Step 36, loss: 0.08388174325227737
Step 37, loss: 0.08075693994760513
Step 38, loss: 0.08081647008657455
Step 39, loss: 0.10267360508441925
Step 40, loss: 0.09803157299757004
Step 41, loss: 0.08619318157434464
Step 42, loss: 0.09996463358402252
Step 43, loss: 0.08521658182144165
Step 44, loss: 0.10208437591791153
Step 45, loss: 0.08445227146148682
Step 46, loss: 0.11115314066410065
Step 47, loss: 0.08237376809120178
Step 48, loss: 0.09492920339107513
Step 49, loss: 0.09434117376804352
Step 50, loss: 0.09538457542657852
Step 51, loss: 0.10146772116422653
Step 110, loss: 0.10275784134864807
Step 52, loss: 0.10333891957998276
Step 53, loss: 0.09780734777450562
Step 54, loss: 0.0894136130809784
Step 55, loss: 0.08602304011583328
Step 56, loss: 0.08182331174612045
Step 57, loss: 0.08126511424779892
Step 58, loss: 0.09783581644296646
Step 59, loss: 0.09782543033361435
Step 60, loss: 0.09924279153347015
Step 61, loss: 0.08977486938238144
Step 62, loss: 0.10259989649057388
Step 63, loss: 0.10501067340373993
Step 64, loss: 0.10390648245811462
Step 65, loss: 0.08770249038934708
Step 66, loss: 0.09540095180273056
Step 67, loss: 0.09132639318704605
Step 68, loss: 0.10009634494781494
Step 69, loss: 0.09285572171211243
Step 70, loss: 0.08500776439905167
Step 71, loss: 0.09973502159118652
Step 72, loss: 0.08392619341611862
Step 73, loss: 0.09875445067882538
Step 74, loss: 0.10540761053562164
Step 75, loss: 0.09229494631290436
Step 76, loss: 0.10541470348834991
Step 77, loss: 0.08958648890256882
Step 78, loss: 0.08453366160392761
Step 79, loss: 0.090049609541893
Step 80, loss: 0.09214366227388382
Running on 8 devices.
Counting all components: ['encoder', 'vq', 'decoder']
Parameter counts:
{'encoder': 18978432, 'vq': 32768, 'decoder': 18978416, 'total': 37989616}
Starting training from step 0...
Step 0, loss: 0.34403467178344727
Step 1, loss: 0.20522570610046387
Step 2, loss: 0.13297320902347565
Step 3, loss: 0.12472856044769287
Step 4, loss: 0.11909522116184235
Step 5, loss: 0.1155850738286972
Step 6, loss: 0.1258772909641266
Step 7, loss: 0.13581234216690063
Step 8, loss: 0.12142564356327057
Step 9, loss: 0.12192649394273758
Step 10, loss: 0.10417830944061279
Step 11, loss: 0.11738790571689606
Step 12, loss: 0.11391294747591019
Step 13, loss: 0.1308845579624176
Step 14, loss: 0.10983618348836899
Step 15, loss: 0.11098787933588028
Step 16, loss: 0.10668867081403732
Step 17, loss: 0.1114255040884018
Step 18, loss: 0.09659188240766525
Step 19, loss: 0.10099179297685623
Step 20, loss: 0.10332012176513672
Step 21, loss: 0.10993895679712296
Step 22, loss: 0.11687967926263809
Step 81, loss: 0.0938016027212143
Step 82, loss: 0.09607720375061035
Step 83, loss: 0.09525217115879059
Step 84, loss: 0.09727976471185684
Step 85, loss: 0.0878567323088646
Step 86, loss: 0.09850167483091354
Step 87, loss: 0.10143165290355682
Step 88, loss: 0.08512572199106216
Step 89, loss: 0.09779776632785797
Step 90, loss: 0.08772671967744827
Step 91, loss: 0.09805448353290558
Step 92, loss: 0.10006199777126312
Step 93, loss: 0.0881110280752182
Step 94, loss: 0.08710669726133347
Step 95, loss: 0.10951592773199081
Step 96, loss: 0.0913202092051506
Step 97, loss: 0.09101113677024841
Step 98, loss: 0.1027674525976181
Step 99, loss: 0.10735452175140381
Step 100, loss: 0.0928899422287941
Step 101, loss: 0.10774534195661545
Step 102, loss: 0.11379466950893402
Step 103, loss: 0.12731407582759857
Step 104, loss: 0.08464271575212479
Step 105, loss: 0.09762269258499146
Step 106, loss: 0.08544913679361343
Step 107, loss: 0.08664936572313309
Step 108, loss: 0.09289372712373734
Step 109, loss: 0.09938964247703552
Step 23, loss: 0.09648826718330383
Step 24, loss: 0.08426374942064285
Step 25, loss: 0.11532411724328995
Step 26, loss: 0.10817211866378784
Step 27, loss: 0.10362100601196289
Step 28, loss: 0.09701235592365265
Step 29, loss: 0.09972862899303436
Step 30, loss: 0.09519169479608536
Step 31, loss: 0.09001486748456955
Step 32, loss: 0.08831518888473511
Step 33, loss: 0.08256299793720245
Step 34, loss: 0.0901721939444542
Step 35, loss: 0.0847548171877861
Step 36, loss: 0.08388174325227737
Step 37, loss: 0.08075693994760513
Step 38, loss: 0.08081647008657455
Step 39, loss: 0.10267360508441925
Step 40, loss: 0.09803157299757004
Step 41, loss: 0.08619318157434464
Step 42, loss: 0.09996463358402252
Step 43, loss: 0.08521658182144165
Step 44, loss: 0.10208437591791153
Step 45, loss: 0.08445227146148682
Step 46, loss: 0.11115314066410065
Step 47, loss: 0.08237376809120178
Step 48, loss: 0.09492920339107513
Step 49, loss: 0.09434117376804352
Step 50, loss: 0.09538457542657852
Step 51, loss: 0.10146772116422653
Step 110, loss: 0.10275784134864807
Step 52, loss: 0.10333891957998276
Step 53, loss: 0.09780734777450562
Step 54, loss: 0.0894136130809784
Step 55, loss: 0.08602304011583328
Step 56, loss: 0.08182331174612045
Step 57, loss: 0.08126511424779892
Step 58, loss: 0.09783581644296646
Step 59, loss: 0.09782543033361435
Step 60, loss: 0.09924279153347015
Step 61, loss: 0.08977486938238144
Step 62, loss: 0.10259989649057388
Step 63, loss: 0.10501067340373993
Step 64, loss: 0.10390648245811462
Step 65, loss: 0.08770249038934708
Step 66, loss: 0.09540095180273056
Step 67, loss: 0.09132639318704605
Step 68, loss: 0.10009634494781494
Step 69, loss: 0.09285572171211243
Step 70, loss: 0.08500776439905167
Step 71, loss: 0.09973502159118652
Step 72, loss: 0.08392619341611862
Step 73, loss: 0.09875445067882538
Step 74, loss: 0.10540761053562164
Step 75, loss: 0.09229494631290436
Step 76, loss: 0.10541470348834991
Step 77, loss: 0.08958648890256882
Step 78, loss: 0.08453366160392761
Step 79, loss: 0.090049609541893
Step 80, loss: 0.09214366227388382
Step 111, loss: 0.08993981033563614
Step 112, loss: 0.0892101600766182
Step 113, loss: 0.08680275827646255
Step 114, loss: 0.10558158904314041
Step 115, loss: 0.0954781174659729
Step 116, loss: 0.07423624396324158
Step 117, loss: 0.0990038588643074
Step 118, loss: 0.09334039688110352
Step 119, loss: 0.10126888006925583
Step 120, loss: 0.09113641828298569
Step 121, loss: 0.08533597737550735
Step 122, loss: 0.08464204519987106
Step 123, loss: 0.09717801213264465
Step 124, loss: 0.1001516729593277
Step 125, loss: 0.09672964364290237
Step 126, loss: 0.10850290209054947
Step 127, loss: 0.09636508673429489
Step 128, loss: 0.08310791850090027
Step 129, loss: 0.09116649627685547
Step 130, loss: 0.08114764839410782
Step 131, loss: 0.09093617647886276
Step 132, loss: 0.09965819865465164
Step 133, loss: 0.08848806470632553
Step 134, loss: 0.10375767946243286
Step 135, loss: 0.07391893118619919
Step 136, loss: 0.08318819850683212
Step 137, loss: 0.08422021567821503
Step 138, loss: 0.10729701817035675
Step 81, loss: 0.0938016027212143
Step 82, loss: 0.09607720375061035
Step 83, loss: 0.09525217115879059
Step 84, loss: 0.09727976471185684
Step 85, loss: 0.0878567323088646
Step 86, loss: 0.09850167483091354
Step 87, loss: 0.10143165290355682
Step 88, loss: 0.08512572199106216
Step 89, loss: 0.09779776632785797
Step 90, loss: 0.08772671967744827
Step 91, loss: 0.09805448353290558
Step 92, loss: 0.10006199777126312
Step 93, loss: 0.0881110280752182
Step 94, loss: 0.08710669726133347
Step 95, loss: 0.10951592773199081
Step 96, loss: 0.0913202092051506
Step 97, loss: 0.09101113677024841
Step 98, loss: 0.1027674525976181
Step 99, loss: 0.10735452175140381
Step 100, loss: 0.0928899422287941
Step 101, loss: 0.10774534195661545
Step 102, loss: 0.11379466950893402
Step 103, loss: 0.12731407582759857
Step 104, loss: 0.08464271575212479
Step 105, loss: 0.09762269258499146
Step 106, loss: 0.08544913679361343
Step 107, loss: 0.08664936572313309
Step 108, loss: 0.09289372712373734
Step 109, loss: 0.09938964247703552
Step 139, loss: 0.08231347799301147
Step 140, loss: 0.09107917547225952
Step 141, loss: 0.09596633911132812
Step 142, loss: 0.08139874041080475
Step 143, loss: 0.09055333584547043
Step 144, loss: 0.08543172478675842
Step 145, loss: 0.08763666450977325
Step 146, loss: 0.0839858427643776
Step 147, loss: 0.08173791319131851
Step 148, loss: 0.11152316629886627
Step 149, loss: 0.09568681567907333
Step 150, loss: 0.10091811418533325
Step 151, loss: 0.0921492949128151
Step 152, loss: 0.09212498366832733
Step 153, loss: 0.09429033100605011
Step 154, loss: 0.09491580724716187
Step 155, loss: 0.09256651997566223
Step 156, loss: 0.10165049135684967
Step 157, loss: 0.09216190129518509
Step 158, loss: 0.09417818486690521
Step 159, loss: 0.08085789531469345
Step 160, loss: 0.09474841505289078
Step 161, loss: 0.09131263941526413
Step 162, loss: 0.1044401004910469
Step 163, loss: 0.09948422014713287
Step 164, loss: 0.09638313204050064
Step 165, loss: 0.10100198537111282
Step 166, loss: 0.08546408265829086
Step 110, loss: 0.10275784134864807
Step 167, loss: 0.09995202720165253
Step 168, loss: 0.0901375561952591
Step 169, loss: 0.1005106195807457
Step 170, loss: 0.09844204038381577
Step 171, loss: 0.09980332106351852
Step 172, loss: 0.09990215301513672
Step 173, loss: 0.09080781042575836
Step 174, loss: 0.09667211025953293
Step 175, loss: 0.09836149215698242
Step 176, loss: 0.09774930030107498
Step 177, loss: 0.08002316951751709
Step 178, loss: 0.10534548759460449
Step 179, loss: 0.0872192308306694
Step 180, loss: 0.09978308528661728
Step 181, loss: 0.10233504325151443
Step 182, loss: 0.09696666896343231
Step 183, loss: 0.10517977923154831
Step 184, loss: 0.09348481893539429
Step 185, loss: 0.08087792992591858
Step 186, loss: 0.10089821368455887
Step 187, loss: 0.10613270103931427
Step 188, loss: 0.11796258389949799
Step 189, loss: 0.08400174975395203
Step 190, loss: 0.09354954957962036
Step 191, loss: 0.10984043776988983
Step 192, loss: 0.088462233543396
Step 193, loss: 0.09041773527860641
Step 194, loss: 0.08473967015743256
Step 111, loss: 0.08993981033563614
Step 112, loss: 0.0892101600766182
Step 113, loss: 0.08680275827646255
Step 114, loss: 0.10558158904314041
Step 115, loss: 0.0954781174659729
Step 116, loss: 0.07423624396324158
Step 117, loss: 0.0990038588643074
Step 118, loss: 0.09334039688110352
Step 119, loss: 0.10126888006925583
Step 120, loss: 0.09113641828298569
Step 121, loss: 0.08533597737550735
Step 122, loss: 0.08464204519987106
Step 123, loss: 0.09717801213264465
Step 124, loss: 0.1001516729593277
Step 125, loss: 0.09672964364290237
Step 126, loss: 0.10850290209054947
Step 127, loss: 0.09636508673429489
Step 128, loss: 0.08310791850090027
Step 129, loss: 0.09116649627685547
Step 130, loss: 0.08114764839410782
Step 131, loss: 0.09093617647886276
Step 132, loss: 0.09965819865465164
Step 133, loss: 0.08848806470632553
Step 134, loss: 0.10375767946243286
Step 135, loss: 0.07391893118619919
Step 136, loss: 0.08318819850683212
Step 137, loss: 0.08422021567821503
Step 138, loss: 0.10729701817035675
Step 195, loss: 0.09430243819952011
Step 196, loss: 0.08992063999176025
Step 197, loss: 0.097152940928936
Step 198, loss: 0.0777839720249176
Step 199, loss: 0.08828026056289673
Step 200, loss: 0.09579538553953171
Step 201, loss: 0.07760141789913177
Step 202, loss: 0.08501187711954117
Step 203, loss: 0.08877477049827576
Step 204, loss: 0.09190297871828079
Step 205, loss: 0.09425204247236252
Step 206, loss: 0.07695277035236359
Step 207, loss: 0.09202922135591507
Step 208, loss: 0.1002676859498024
Step 209, loss: 0.08858001977205276
Step 210, loss: 0.08115501701831818
Step 211, loss: 0.08873545378446579
Step 212, loss: 0.1039958968758583
Step 213, loss: 0.09006499499082565
Step 214, loss: 0.08361829817295074
Step 215, loss: 0.10188568383455276
Step 216, loss: 0.10438724607229233
Step 217, loss: 0.09281189739704132
Step 218, loss: 0.09521891921758652
Step 219, loss: 0.1059456467628479
Step 220, loss: 0.10383511334657669
Step 221, loss: 0.09161245077848434
Step 222, loss: 0.0991496816277504
Step 139, loss: 0.08231347799301147
Step 140, loss: 0.09107917547225952
Step 141, loss: 0.09596633911132812
Step 142, loss: 0.08139874041080475
Step 143, loss: 0.09055333584547043
Step 144, loss: 0.08543172478675842
Step 145, loss: 0.08763666450977325
Step 146, loss: 0.0839858427643776
Step 147, loss: 0.08173791319131851
Step 148, loss: 0.11152316629886627
Step 149, loss: 0.09568681567907333
Step 150, loss: 0.10091811418533325
Step 151, loss: 0.0921492949128151
Step 152, loss: 0.09212498366832733
Step 153, loss: 0.09429033100605011
Step 154, loss: 0.09491580724716187
Step 155, loss: 0.09256651997566223
Step 156, loss: 0.10165049135684967
Step 157, loss: 0.09216190129518509
Step 158, loss: 0.09417818486690521
Step 159, loss: 0.08085789531469345
Step 160, loss: 0.09474841505289078
Step 161, loss: 0.09131263941526413
Step 162, loss: 0.1044401004910469
Step 163, loss: 0.09948422014713287
Step 164, loss: 0.09638313204050064
Step 165, loss: 0.10100198537111282
Step 166, loss: 0.08546408265829086
Step 223, loss: 0.08625559508800507
Step 224, loss: 0.07936974614858627
Step 167, loss: 0.09995202720165253
Step 168, loss: 0.0901375561952591
Step 169, loss: 0.1005106195807457
Step 170, loss: 0.09844204038381577
Step 171, loss: 0.09980332106351852
Step 172, loss: 0.09990215301513672
Step 173, loss: 0.09080781042575836
Step 174, loss: 0.09667211025953293
Step 175, loss: 0.09836149215698242
Step 176, loss: 0.09774930030107498
Step 177, loss: 0.08002316951751709
Step 178, loss: 0.10534548759460449
Step 179, loss: 0.0872192308306694
Step 180, loss: 0.09978308528661728
Step 181, loss: 0.10233504325151443
Step 182, loss: 0.09696666896343231
Step 183, loss: 0.10517977923154831
Step 184, loss: 0.09348481893539429
Step 185, loss: 0.08087792992591858
Step 186, loss: 0.10089821368455887
Step 187, loss: 0.10613270103931427
Step 188, loss: 0.11796258389949799
Step 189, loss: 0.08400174975395203
Step 190, loss: 0.09354954957962036
Step 191, loss: 0.10984043776988983
Step 192, loss: 0.088462233543396
Step 193, loss: 0.09041773527860641
Step 194, loss: 0.08473967015743256
Running on 8 devices.
Counting all components: ['encoder', 'vq', 'decoder']
Parameter counts:
{'encoder': 18978432, 'vq': 32768, 'decoder': 18978416, 'total': 37989616}
Starting training from step 0...
Step 0, loss: 0.34403467178344727
Step 1, loss: 0.20522570610046387
Step 2, loss: 0.13297320902347565
Step 3, loss: 0.12472856044769287
Step 4, loss: 0.11909522116184235
Step 5, loss: 0.1155850738286972
Step 6, loss: 0.1258772909641266
Step 7, loss: 0.13581234216690063
Step 8, loss: 0.12142564356327057
Step 9, loss: 0.12192649394273758
Step 10, loss: 0.10417830944061279
Step 11, loss: 0.11738790571689606
Step 12, loss: 0.11391294747591019
Step 13, loss: 0.1308845579624176
Step 14, loss: 0.10983618348836899
Step 15, loss: 0.11098787933588028
Step 16, loss: 0.10668867081403732
Step 17, loss: 0.1114255040884018
Step 18, loss: 0.09659188240766525
Step 19, loss: 0.10099179297685623
Step 20, loss: 0.10332012176513672
Step 21, loss: 0.10993895679712296
Step 22, loss: 0.11687967926263809
Step 195, loss: 0.09430243819952011
Step 196, loss: 0.08992063999176025
Step 197, loss: 0.097152940928936
Step 198, loss: 0.0777839720249176
Step 199, loss: 0.08828026056289673
Step 200, loss: 0.09579538553953171
Step 201, loss: 0.07760141789913177
Step 202, loss: 0.08501187711954117
Step 203, loss: 0.08877477049827576
Step 204, loss: 0.09190297871828079
Step 205, loss: 0.09425204247236252
Step 206, loss: 0.07695277035236359
Step 207, loss: 0.09202922135591507
Step 208, loss: 0.1002676859498024
Step 209, loss: 0.08858001977205276
Step 210, loss: 0.08115501701831818
Step 211, loss: 0.08873545378446579
Step 212, loss: 0.1039958968758583
Step 213, loss: 0.09006499499082565
Step 214, loss: 0.08361829817295074
Step 215, loss: 0.10188568383455276
Step 216, loss: 0.10438724607229233
Step 217, loss: 0.09281189739704132
Step 218, loss: 0.09521891921758652
Step 219, loss: 0.1059456467628479
Step 220, loss: 0.10383511334657669
Step 221, loss: 0.09161245077848434
Step 222, loss: 0.0991496816277504
Step 23, loss: 0.09648826718330383
Step 24, loss: 0.08426374942064285
Step 25, loss: 0.11532411724328995
Step 26, loss: 0.10817211866378784
Step 27, loss: 0.10362100601196289
Step 28, loss: 0.09701235592365265
Step 29, loss: 0.09972862899303436
Step 30, loss: 0.09519169479608536
Step 31, loss: 0.09001486748456955
Step 32, loss: 0.08831518888473511
Step 33, loss: 0.08256299793720245
Step 34, loss: 0.0901721939444542
Step 35, loss: 0.0847548171877861
Step 36, loss: 0.08388174325227737
Step 37, loss: 0.08075693994760513
Step 38, loss: 0.08081647008657455
Step 39, loss: 0.10267360508441925
Step 40, loss: 0.09803157299757004
Step 41, loss: 0.08619318157434464
Step 42, loss: 0.09996463358402252
Step 43, loss: 0.08521658182144165
Step 44, loss: 0.10208437591791153
Step 45, loss: 0.08445227146148682
Step 46, loss: 0.11115314066410065
Step 47, loss: 0.08237376809120178
Step 48, loss: 0.09492920339107513
Step 49, loss: 0.09434117376804352
Step 50, loss: 0.09538457542657852
Step 51, loss: 0.10146772116422653
Step 223, loss: 0.08625559508800507
Step 224, loss: 0.07936974614858627
Step 52, loss: 0.10333891957998276
Step 53, loss: 0.09780734777450562
Step 54, loss: 0.0894136130809784
Step 55, loss: 0.08602304011583328
Step 56, loss: 0.08182331174612045
Step 57, loss: 0.08126511424779892
Step 58, loss: 0.09783581644296646
Step 59, loss: 0.09782543033361435
Step 60, loss: 0.09924279153347015
Step 61, loss: 0.08977486938238144
Step 62, loss: 0.10259989649057388
Step 63, loss: 0.10501067340373993
Step 64, loss: 0.10390648245811462
Step 65, loss: 0.08770249038934708
Step 66, loss: 0.09540095180273056
Step 67, loss: 0.09132639318704605
Step 68, loss: 0.10009634494781494
Step 69, loss: 0.09285572171211243
Step 70, loss: 0.08500776439905167
Step 71, loss: 0.09973502159118652
Step 72, loss: 0.08392619341611862
Step 73, loss: 0.09875445067882538
Step 74, loss: 0.10540761053562164
Step 75, loss: 0.09229494631290436
Step 76, loss: 0.10541470348834991
Step 77, loss: 0.08958648890256882
Step 78, loss: 0.08453366160392761
Step 79, loss: 0.090049609541893
Step 80, loss: 0.09214366227388382
Step 111, loss: 0.08993981033563614
Step 112, loss: 0.0892101600766182
Step 113, loss: 0.08680275827646255
Step 114, loss: 0.10558158904314041
Step 115, loss: 0.0954781174659729
Step 116, loss: 0.07423624396324158
Step 117, loss: 0.0990038588643074
Step 118, loss: 0.09334039688110352
Step 119, loss: 0.10126888006925583
Step 120, loss: 0.09113641828298569
Step 121, loss: 0.08533597737550735
Step 122, loss: 0.08464204519987106
Step 123, loss: 0.09717801213264465
Step 124, loss: 0.1001516729593277
Step 125, loss: 0.09672964364290237
Step 126, loss: 0.10850290209054947
Step 127, loss: 0.09636508673429489
Step 128, loss: 0.08310791850090027
Step 129, loss: 0.09116649627685547
Step 130, loss: 0.08114764839410782
Step 131, loss: 0.09093617647886276
Step 132, loss: 0.09965819865465164
Step 133, loss: 0.08848806470632553
Step 134, loss: 0.10375767946243286
Step 135, loss: 0.07391893118619919
Step 136, loss: 0.08318819850683212
Step 137, loss: 0.08422021567821503
Step 138, loss: 0.10729701817035675
Step 81, loss: 0.0938016027212143
Step 82, loss: 0.09607720375061035
Step 83, loss: 0.09525217115879059
Step 84, loss: 0.09727976471185684
Step 85, loss: 0.0878567323088646
Step 86, loss: 0.09850167483091354
Step 87, loss: 0.10143165290355682
Step 88, loss: 0.08512572199106216
Step 89, loss: 0.09779776632785797
Step 90, loss: 0.08772671967744827
Step 91, loss: 0.09805448353290558
Step 92, loss: 0.10006199777126312
Step 93, loss: 0.0881110280752182
Step 94, loss: 0.08710669726133347
Step 95, loss: 0.10951592773199081
Step 96, loss: 0.0913202092051506
Step 97, loss: 0.09101113677024841
Step 98, loss: 0.1027674525976181
Step 99, loss: 0.10735452175140381
Step 100, loss: 0.0928899422287941
Step 101, loss: 0.10774534195661545
Step 102, loss: 0.11379466950893402
Step 103, loss: 0.12731407582759857
Step 104, loss: 0.08464271575212479
Step 105, loss: 0.09762269258499146
Step 106, loss: 0.08544913679361343
Step 107, loss: 0.08664936572313309
Step 108, loss: 0.09289372712373734
Step 109, loss: 0.09938964247703552
Step 139, loss: 0.08231347799301147
Step 140, loss: 0.09107917547225952
Step 141, loss: 0.09596633911132812
Step 142, loss: 0.08139874041080475
Step 143, loss: 0.09055333584547043
Step 144, loss: 0.08543172478675842
Step 145, loss: 0.08763666450977325
Step 146, loss: 0.0839858427643776
Step 147, loss: 0.08173791319131851
Step 148, loss: 0.11152316629886627
Step 149, loss: 0.09568681567907333
Step 150, loss: 0.10091811418533325
Step 151, loss: 0.0921492949128151
Step 152, loss: 0.09212498366832733
Step 153, loss: 0.09429033100605011
Step 154, loss: 0.09491580724716187
Step 155, loss: 0.09256651997566223
Step 156, loss: 0.10165049135684967
Step 157, loss: 0.09216190129518509
Step 158, loss: 0.09417818486690521
Step 159, loss: 0.08085789531469345
Step 160, loss: 0.09474841505289078
Step 161, loss: 0.09131263941526413
Step 162, loss: 0.1044401004910469
Step 163, loss: 0.09948422014713287
Step 164, loss: 0.09638313204050064
Step 165, loss: 0.10100198537111282
Step 166, loss: 0.08546408265829086
Step 110, loss: 0.10275784134864807
Step 167, loss: 0.09995202720165253
Step 168, loss: 0.0901375561952591
Step 169, loss: 0.1005106195807457
Step 170, loss: 0.09844204038381577
Step 171, loss: 0.09980332106351852
Step 172, loss: 0.09990215301513672
Step 173, loss: 0.09080781042575836
Step 174, loss: 0.09667211025953293
Step 175, loss: 0.09836149215698242
Step 176, loss: 0.09774930030107498
Step 177, loss: 0.08002316951751709
Step 178, loss: 0.10534548759460449
Step 179, loss: 0.0872192308306694
Step 180, loss: 0.09978308528661728
Step 181, loss: 0.10233504325151443
Step 182, loss: 0.09696666896343231
Step 183, loss: 0.10517977923154831
Step 184, loss: 0.09348481893539429
Step 185, loss: 0.08087792992591858
Step 186, loss: 0.10089821368455887
Step 187, loss: 0.10613270103931427
Step 188, loss: 0.11796258389949799
Step 189, loss: 0.08400174975395203
Step 190, loss: 0.09354954957962036
Step 191, loss: 0.10984043776988983
Step 192, loss: 0.088462233543396
Step 193, loss: 0.09041773527860641
Step 194, loss: 0.08473967015743256
Running on 8 devices.
Counting all components: ['encoder', 'vq', 'decoder']
Parameter counts:
{'encoder': 18978432, 'vq': 32768, 'decoder': 18978416, 'total': 37989616}
Starting training from step 0...
Step 0, loss: 0.34403467178344727
Step 1, loss: 0.20522570610046387
Step 2, loss: 0.13297320902347565
Step 3, loss: 0.12472856044769287
Step 4, loss: 0.11909522116184235
Step 5, loss: 0.1155850738286972
Step 6, loss: 0.1258772909641266
Step 7, loss: 0.13581234216690063
Step 8, loss: 0.12142564356327057
Step 9, loss: 0.12192649394273758
Step 10, loss: 0.10417830944061279
Step 11, loss: 0.11738790571689606
Step 12, loss: 0.11391294747591019
Step 13, loss: 0.1308845579624176
Step 14, loss: 0.10983618348836899
Step 15, loss: 0.11098787933588028
Step 16, loss: 0.10668867081403732
Step 17, loss: 0.1114255040884018
Step 18, loss: 0.09659188240766525
Step 19, loss: 0.10099179297685623
Step 20, loss: 0.10332012176513672
Step 21, loss: 0.10993895679712296
Step 22, loss: 0.11687967926263809
Step 195, loss: 0.09430243819952011
Step 196, loss: 0.08992063999176025
Step 197, loss: 0.097152940928936
Step 198, loss: 0.0777839720249176
Step 199, loss: 0.08828026056289673
Step 200, loss: 0.09579538553953171
Step 201, loss: 0.07760141789913177
Step 202, loss: 0.08501187711954117
Step 203, loss: 0.08877477049827576
Step 204, loss: 0.09190297871828079
Step 205, loss: 0.09425204247236252
Step 206, loss: 0.07695277035236359
Step 207, loss: 0.09202922135591507
Step 208, loss: 0.1002676859498024
Step 209, loss: 0.08858001977205276
Step 210, loss: 0.08115501701831818
Step 211, loss: 0.08873545378446579
Step 212, loss: 0.1039958968758583
Step 213, loss: 0.09006499499082565
Step 214, loss: 0.08361829817295074
Step 215, loss: 0.10188568383455276
Step 216, loss: 0.10438724607229233
Step 217, loss: 0.09281189739704132
Step 218, loss: 0.09521891921758652
Step 219, loss: 0.1059456467628479
Step 220, loss: 0.10383511334657669
Step 221, loss: 0.09161245077848434
Step 222, loss: 0.0991496816277504
Step 23, loss: 0.09648826718330383
Step 24, loss: 0.08426374942064285
Step 25, loss: 0.11532411724328995
Step 26, loss: 0.10817211866378784
Step 27, loss: 0.10362100601196289
Step 28, loss: 0.09701235592365265
Step 29, loss: 0.09972862899303436
Step 30, loss: 0.09519169479608536
Step 31, loss: 0.09001486748456955
Step 32, loss: 0.08831518888473511
Step 33, loss: 0.08256299793720245
Step 34, loss: 0.0901721939444542
Step 35, loss: 0.0847548171877861
Step 36, loss: 0.08388174325227737
Step 37, loss: 0.08075693994760513
Step 38, loss: 0.08081647008657455
Step 39, loss: 0.10267360508441925
Step 40, loss: 0.09803157299757004
Step 41, loss: 0.08619318157434464
Step 42, loss: 0.09996463358402252
Step 43, loss: 0.08521658182144165
Step 44, loss: 0.10208437591791153
Step 45, loss: 0.08445227146148682
Step 46, loss: 0.11115314066410065
Step 47, loss: 0.08237376809120178
Step 48, loss: 0.09492920339107513
Step 49, loss: 0.09434117376804352
Step 50, loss: 0.09538457542657852
Step 51, loss: 0.10146772116422653
Step 223, loss: 0.08625559508800507
Step 224, loss: 0.07936974614858627
Step 52, loss: 0.10333891957998276
Step 53, loss: 0.09780734777450562
Step 54, loss: 0.0894136130809784
Step 55, loss: 0.08602304011583328
Step 56, loss: 0.08182331174612045
Step 57, loss: 0.08126511424779892
Step 58, loss: 0.09783581644296646
Step 59, loss: 0.09782543033361435
Step 60, loss: 0.09924279153347015
Step 61, loss: 0.08977486938238144
Step 62, loss: 0.10259989649057388
Step 63, loss: 0.10501067340373993
Step 64, loss: 0.10390648245811462
Step 65, loss: 0.08770249038934708
Step 66, loss: 0.09540095180273056
Step 67, loss: 0.09132639318704605
Step 68, loss: 0.10009634494781494
Step 69, loss: 0.09285572171211243
Step 70, loss: 0.08500776439905167
Step 71, loss: 0.09973502159118652
Step 72, loss: 0.08392619341611862
Step 73, loss: 0.09875445067882538
Step 74, loss: 0.10540761053562164
Step 75, loss: 0.09229494631290436
Step 76, loss: 0.10541470348834991
Step 77, loss: 0.08958648890256882
Step 78, loss: 0.08453366160392761
Step 79, loss: 0.090049609541893
Step 80, loss: 0.09214366227388382
Running on 8 devices.
Counting all components: ['encoder', 'vq', 'decoder']
Parameter counts:
{'encoder': 18978432, 'vq': 32768, 'decoder': 18978416, 'total': 37989616}
Starting training from step 0...
Step 0, loss: 0.34403467178344727
Step 1, loss: 0.20522570610046387
Step 2, loss: 0.13297320902347565
Step 3, loss: 0.12472856044769287
Step 4, loss: 0.11909522116184235
Step 5, loss: 0.1155850738286972
Step 6, loss: 0.1258772909641266
Step 7, loss: 0.13581234216690063
Step 8, loss: 0.12142564356327057
Step 9, loss: 0.12192649394273758
Step 10, loss: 0.10417830944061279
Step 11, loss: 0.11738790571689606
Step 12, loss: 0.11391294747591019
Step 13, loss: 0.1308845579624176
Step 14, loss: 0.10983618348836899
Step 15, loss: 0.11098787933588028
Step 16, loss: 0.10668867081403732
Step 17, loss: 0.1114255040884018
Step 18, loss: 0.09659188240766525
Step 19, loss: 0.10099179297685623
Step 20, loss: 0.10332012176513672
Step 21, loss: 0.10993895679712296
Step 22, loss: 0.11687967926263809
Step 81, loss: 0.0938016027212143
Step 82, loss: 0.09607720375061035
Step 83, loss: 0.09525217115879059
Step 84, loss: 0.09727976471185684
Step 85, loss: 0.0878567323088646
Step 86, loss: 0.09850167483091354
Step 87, loss: 0.10143165290355682
Step 88, loss: 0.08512572199106216
Step 89, loss: 0.09779776632785797
Step 90, loss: 0.08772671967744827
Step 91, loss: 0.09805448353290558
Step 92, loss: 0.10006199777126312
Step 93, loss: 0.0881110280752182
Step 94, loss: 0.08710669726133347
Step 95, loss: 0.10951592773199081
Step 96, loss: 0.0913202092051506
Step 97, loss: 0.09101113677024841
Step 98, loss: 0.1027674525976181
Step 99, loss: 0.10735452175140381
Step 100, loss: 0.0928899422287941
Step 101, loss: 0.10774534195661545
Step 102, loss: 0.11379466950893402
Step 103, loss: 0.12731407582759857
Step 104, loss: 0.08464271575212479
Step 105, loss: 0.09762269258499146
Step 106, loss: 0.08544913679361343
Step 107, loss: 0.08664936572313309
Step 108, loss: 0.09289372712373734
Step 109, loss: 0.09938964247703552
Step 23, loss: 0.09648826718330383
Step 24, loss: 0.08426374942064285
Step 25, loss: 0.11532411724328995
Step 26, loss: 0.10817211866378784
Step 27, loss: 0.10362100601196289
Step 28, loss: 0.09701235592365265
Step 29, loss: 0.09972862899303436
Step 30, loss: 0.09519169479608536
Step 31, loss: 0.09001486748456955
Step 32, loss: 0.08831518888473511
Step 33, loss: 0.08256299793720245
Step 34, loss: 0.0901721939444542
Step 35, loss: 0.0847548171877861
Step 36, loss: 0.08388174325227737
Step 37, loss: 0.08075693994760513
Step 38, loss: 0.08081647008657455
Step 39, loss: 0.10267360508441925
Step 40, loss: 0.09803157299757004
Step 41, loss: 0.08619318157434464
Step 42, loss: 0.09996463358402252
Step 43, loss: 0.08521658182144165
Step 44, loss: 0.10208437591791153
Step 45, loss: 0.08445227146148682
Step 46, loss: 0.11115314066410065
Step 47, loss: 0.08237376809120178
Step 48, loss: 0.09492920339107513
Step 49, loss: 0.09434117376804352
Step 50, loss: 0.09538457542657852
Step 51, loss: 0.10146772116422653
Step 110, loss: 0.10275784134864807
Step 52, loss: 0.10333891957998276
Step 53, loss: 0.09780734777450562
Step 54, loss: 0.0894136130809784
Step 55, loss: 0.08602304011583328
Step 56, loss: 0.08182331174612045
Step 57, loss: 0.08126511424779892
Step 58, loss: 0.09783581644296646
Step 59, loss: 0.09782543033361435
Step 60, loss: 0.09924279153347015
Step 61, loss: 0.08977486938238144
Step 62, loss: 0.10259989649057388
Step 63, loss: 0.10501067340373993
Step 64, loss: 0.10390648245811462
Step 65, loss: 0.08770249038934708
Step 66, loss: 0.09540095180273056
Step 67, loss: 0.09132639318704605
Step 68, loss: 0.10009634494781494
Step 69, loss: 0.09285572171211243
Step 70, loss: 0.08500776439905167
Step 71, loss: 0.09973502159118652
Step 72, loss: 0.08392619341611862
Step 73, loss: 0.09875445067882538
Step 74, loss: 0.10540761053562164
Step 75, loss: 0.09229494631290436
Step 76, loss: 0.10541470348834991
Step 77, loss: 0.08958648890256882
Step 78, loss: 0.08453366160392761
Step 79, loss: 0.090049609541893
Step 80, loss: 0.09214366227388382
Step 111, loss: 0.08993981033563614
Step 112, loss: 0.0892101600766182
Step 113, loss: 0.08680275827646255
Step 114, loss: 0.10558158904314041
Step 115, loss: 0.0954781174659729
Step 116, loss: 0.07423624396324158
Step 117, loss: 0.0990038588643074
Step 118, loss: 0.09334039688110352
Step 119, loss: 0.10126888006925583
Step 120, loss: 0.09113641828298569
Step 121, loss: 0.08533597737550735
Step 122, loss: 0.08464204519987106
Step 123, loss: 0.09717801213264465
Step 124, loss: 0.1001516729593277
Step 125, loss: 0.09672964364290237
Step 126, loss: 0.10850290209054947
Step 127, loss: 0.09636508673429489
Step 128, loss: 0.08310791850090027
Step 129, loss: 0.09116649627685547
Step 130, loss: 0.08114764839410782
Step 131, loss: 0.09093617647886276
Step 132, loss: 0.09965819865465164
Step 133, loss: 0.08848806470632553
Step 134, loss: 0.10375767946243286
Step 135, loss: 0.07391893118619919
Step 136, loss: 0.08318819850683212
Step 137, loss: 0.08422021567821503
Step 138, loss: 0.10729701817035675
Step 81, loss: 0.0938016027212143
Step 82, loss: 0.09607720375061035
Step 83, loss: 0.09525217115879059
Step 84, loss: 0.09727976471185684
Step 85, loss: 0.0878567323088646
Step 86, loss: 0.09850167483091354
Step 87, loss: 0.10143165290355682
Step 88, loss: 0.08512572199106216
Step 89, loss: 0.09779776632785797
Step 90, loss: 0.08772671967744827
Step 91, loss: 0.09805448353290558
Step 92, loss: 0.10006199777126312
Step 93, loss: 0.0881110280752182
Step 94, loss: 0.08710669726133347
Step 95, loss: 0.10951592773199081
Step 96, loss: 0.0913202092051506
Step 97, loss: 0.09101113677024841
Step 98, loss: 0.1027674525976181
Step 99, loss: 0.10735452175140381
Step 100, loss: 0.0928899422287941
Step 101, loss: 0.10774534195661545
Step 102, loss: 0.11379466950893402
Step 103, loss: 0.12731407582759857
Step 104, loss: 0.08464271575212479
Step 105, loss: 0.09762269258499146
Step 106, loss: 0.08544913679361343
Step 107, loss: 0.08664936572313309
Step 108, loss: 0.09289372712373734
Step 109, loss: 0.09938964247703552
Step 139, loss: 0.08231347799301147
Step 140, loss: 0.09107917547225952
Step 141, loss: 0.09596633911132812
Step 142, loss: 0.08139874041080475
Step 143, loss: 0.09055333584547043
Step 144, loss: 0.08543172478675842
Step 145, loss: 0.08763666450977325
Step 146, loss: 0.0839858427643776
Step 147, loss: 0.08173791319131851
Step 148, loss: 0.11152316629886627
Step 149, loss: 0.09568681567907333
Step 150, loss: 0.10091811418533325
Step 151, loss: 0.0921492949128151
Step 152, loss: 0.09212498366832733
Step 153, loss: 0.09429033100605011
Step 154, loss: 0.09491580724716187
Step 155, loss: 0.09256651997566223
Step 156, loss: 0.10165049135684967
Step 157, loss: 0.09216190129518509
Step 158, loss: 0.09417818486690521
Step 159, loss: 0.08085789531469345
Step 160, loss: 0.09474841505289078
Step 161, loss: 0.09131263941526413
Step 162, loss: 0.1044401004910469
Step 163, loss: 0.09948422014713287
Step 164, loss: 0.09638313204050064
Step 165, loss: 0.10100198537111282
Step 166, loss: 0.08546408265829086
Step 110, loss: 0.10275784134864807
Step 167, loss: 0.09995202720165253
Step 168, loss: 0.0901375561952591
Step 169, loss: 0.1005106195807457
Step 170, loss: 0.09844204038381577
Step 171, loss: 0.09980332106351852
Step 172, loss: 0.09990215301513672
Step 173, loss: 0.09080781042575836
Step 174, loss: 0.09667211025953293
Step 175, loss: 0.09836149215698242
Step 176, loss: 0.09774930030107498
Step 177, loss: 0.08002316951751709
Step 178, loss: 0.10534548759460449
Step 179, loss: 0.0872192308306694
Step 180, loss: 0.09978308528661728
Step 181, loss: 0.10233504325151443
Step 182, loss: 0.09696666896343231
Step 183, loss: 0.10517977923154831
Step 184, loss: 0.09348481893539429
Step 185, loss: 0.08087792992591858
Step 186, loss: 0.10089821368455887
Step 187, loss: 0.10613270103931427
Step 188, loss: 0.11796258389949799
Step 189, loss: 0.08400174975395203
Step 190, loss: 0.09354954957962036
Step 191, loss: 0.10984043776988983
Step 192, loss: 0.088462233543396
Step 193, loss: 0.09041773527860641
Step 194, loss: 0.08473967015743256
Step 111, loss: 0.08993981033563614
Step 112, loss: 0.0892101600766182
Step 113, loss: 0.08680275827646255
Step 114, loss: 0.10558158904314041
Step 115, loss: 0.0954781174659729
Step 116, loss: 0.07423624396324158
Step 117, loss: 0.0990038588643074
Step 118, loss: 0.09334039688110352
Step 119, loss: 0.10126888006925583
Step 120, loss: 0.09113641828298569
Step 121, loss: 0.08533597737550735
Step 122, loss: 0.08464204519987106
Step 123, loss: 0.09717801213264465
Step 124, loss: 0.1001516729593277
Step 125, loss: 0.09672964364290237
Step 126, loss: 0.10850290209054947
Step 127, loss: 0.09636508673429489
Step 128, loss: 0.08310791850090027
Step 129, loss: 0.09116649627685547
Step 130, loss: 0.08114764839410782
Step 131, loss: 0.09093617647886276
Step 132, loss: 0.09965819865465164
Step 133, loss: 0.08848806470632553
Step 134, loss: 0.10375767946243286
Step 135, loss: 0.07391893118619919
Step 136, loss: 0.08318819850683212
Step 137, loss: 0.08422021567821503
Step 138, loss: 0.10729701817035675
Step 195, loss: 0.09430243819952011
Step 196, loss: 0.08992063999176025
Step 197, loss: 0.097152940928936
Step 198, loss: 0.0777839720249176
Step 199, loss: 0.08828026056289673
Step 200, loss: 0.09579538553953171
Step 201, loss: 0.07760141789913177
Step 202, loss: 0.08501187711954117
Step 203, loss: 0.08877477049827576
Step 204, loss: 0.09190297871828079
Step 205, loss: 0.09425204247236252
Step 206, loss: 0.07695277035236359
Step 207, loss: 0.09202922135591507
Step 208, loss: 0.1002676859498024
Step 209, loss: 0.08858001977205276
Step 210, loss: 0.08115501701831818
Step 211, loss: 0.08873545378446579
Step 212, loss: 0.1039958968758583
Step 213, loss: 0.09006499499082565
Step 214, loss: 0.08361829817295074
Step 215, loss: 0.10188568383455276
Step 216, loss: 0.10438724607229233
Step 217, loss: 0.09281189739704132
Step 218, loss: 0.09521891921758652
Step 219, loss: 0.1059456467628479
Step 220, loss: 0.10383511334657669
Step 221, loss: 0.09161245077848434
Step 222, loss: 0.0991496816277504
Step 139, loss: 0.08231347799301147
Step 140, loss: 0.09107917547225952
Step 141, loss: 0.09596633911132812
Step 142, loss: 0.08139874041080475
Step 143, loss: 0.09055333584547043
Step 144, loss: 0.08543172478675842
Step 145, loss: 0.08763666450977325
Step 146, loss: 0.0839858427643776
Step 147, loss: 0.08173791319131851
Step 148, loss: 0.11152316629886627
Step 149, loss: 0.09568681567907333
Step 150, loss: 0.10091811418533325
Step 151, loss: 0.0921492949128151
Step 152, loss: 0.09212498366832733
Step 153, loss: 0.09429033100605011
Step 154, loss: 0.09491580724716187
Step 155, loss: 0.09256651997566223
Step 156, loss: 0.10165049135684967
Step 157, loss: 0.09216190129518509
Step 158, loss: 0.09417818486690521
Step 159, loss: 0.08085789531469345
Step 160, loss: 0.09474841505289078
Step 161, loss: 0.09131263941526413
Step 162, loss: 0.1044401004910469
Step 163, loss: 0.09948422014713287
Step 164, loss: 0.09638313204050064
Step 165, loss: 0.10100198537111282
Step 166, loss: 0.08546408265829086
Step 223, loss: 0.08625559508800507
Step 224, loss: 0.07936974614858627
Step 167, loss: 0.09995202720165253
Step 168, loss: 0.0901375561952591
Step 169, loss: 0.1005106195807457
Step 170, loss: 0.09844204038381577
Step 171, loss: 0.09980332106351852
Step 172, loss: 0.09990215301513672
Step 173, loss: 0.09080781042575836
Step 174, loss: 0.09667211025953293
Step 175, loss: 0.09836149215698242
Step 176, loss: 0.09774930030107498
Step 177, loss: 0.08002316951751709
Step 178, loss: 0.10534548759460449
Step 179, loss: 0.0872192308306694
Step 180, loss: 0.09978308528661728
Step 181, loss: 0.10233504325151443
Step 182, loss: 0.09696666896343231
Step 183, loss: 0.10517977923154831
Step 184, loss: 0.09348481893539429
Step 185, loss: 0.08087792992591858
Step 186, loss: 0.10089821368455887
Step 187, loss: 0.10613270103931427
Step 188, loss: 0.11796258389949799
Step 189, loss: 0.08400174975395203
Step 190, loss: 0.09354954957962036
Step 191, loss: 0.10984043776988983
Step 192, loss: 0.088462233543396
Step 193, loss: 0.09041773527860641
Step 194, loss: 0.08473967015743256
Step 111, loss: 0.08993981033563614
Step 112, loss: 0.0892101600766182
Step 113, loss: 0.08680275827646255
Step 114, loss: 0.10558158904314041
Step 115, loss: 0.0954781174659729
Step 116, loss: 0.07423624396324158
Step 117, loss: 0.0990038588643074
Step 118, loss: 0.09334039688110352
Step 119, loss: 0.10126888006925583
Step 120, loss: 0.09113641828298569
Step 121, loss: 0.08533597737550735
Step 122, loss: 0.08464204519987106
Step 123, loss: 0.09717801213264465
Step 124, loss: 0.1001516729593277
Step 125, loss: 0.09672964364290237
Step 126, loss: 0.10850290209054947
Step 127, loss: 0.09636508673429489
Step 128, loss: 0.08310791850090027
Step 129, loss: 0.09116649627685547
Step 130, loss: 0.08114764839410782
Step 131, loss: 0.09093617647886276
Step 132, loss: 0.09965819865465164
Step 133, loss: 0.08848806470632553
Step 134, loss: 0.10375767946243286
Step 135, loss: 0.07391893118619919
Step 136, loss: 0.08318819850683212
Step 137, loss: 0.08422021567821503
Step 138, loss: 0.10729701817035675
Step 195, loss: 0.09430243819952011
Step 196, loss: 0.08992063999176025
Step 197, loss: 0.097152940928936
Step 198, loss: 0.0777839720249176
Step 199, loss: 0.08828026056289673
Step 200, loss: 0.09579538553953171
Step 201, loss: 0.07760141789913177
Step 202, loss: 0.08501187711954117
Step 203, loss: 0.08877477049827576
Step 204, loss: 0.09190297871828079
Step 205, loss: 0.09425204247236252
Step 206, loss: 0.07695277035236359
Step 207, loss: 0.09202922135591507
Step 208, loss: 0.1002676859498024
Step 209, loss: 0.08858001977205276
Step 210, loss: 0.08115501701831818
Step 211, loss: 0.08873545378446579
Step 212, loss: 0.1039958968758583
Step 213, loss: 0.09006499499082565
Step 214, loss: 0.08361829817295074
Step 215, loss: 0.10188568383455276
Step 216, loss: 0.10438724607229233
Step 217, loss: 0.09281189739704132
Step 218, loss: 0.09521891921758652
Step 219, loss: 0.1059456467628479
Step 220, loss: 0.10383511334657669
Step 221, loss: 0.09161245077848434
Step 222, loss: 0.0991496816277504
Step 139, loss: 0.08231347799301147
Step 140, loss: 0.09107917547225952
Step 141, loss: 0.09596633911132812
Step 142, loss: 0.08139874041080475
Step 143, loss: 0.09055333584547043
Step 144, loss: 0.08543172478675842
Step 145, loss: 0.08763666450977325
Step 146, loss: 0.0839858427643776
Step 147, loss: 0.08173791319131851
Step 148, loss: 0.11152316629886627
Step 149, loss: 0.09568681567907333
Step 150, loss: 0.10091811418533325
Step 151, loss: 0.0921492949128151
Step 152, loss: 0.09212498366832733
Step 153, loss: 0.09429033100605011
Step 154, loss: 0.09491580724716187
Step 155, loss: 0.09256651997566223
Step 156, loss: 0.10165049135684967
Step 157, loss: 0.09216190129518509
Step 158, loss: 0.09417818486690521
Step 159, loss: 0.08085789531469345
Step 160, loss: 0.09474841505289078
Step 161, loss: 0.09131263941526413
Step 162, loss: 0.1044401004910469
Step 163, loss: 0.09948422014713287
Step 164, loss: 0.09638313204050064
Step 165, loss: 0.10100198537111282
Step 166, loss: 0.08546408265829086
Step 223, loss: 0.08625559508800507
Step 224, loss: 0.07936974614858627
Step 167, loss: 0.09995202720165253
Step 168, loss: 0.0901375561952591
Step 169, loss: 0.1005106195807457
Step 170, loss: 0.09844204038381577
Step 171, loss: 0.09980332106351852
Step 172, loss: 0.09990215301513672
Step 173, loss: 0.09080781042575836
Step 174, loss: 0.09667211025953293
Step 175, loss: 0.09836149215698242
Step 176, loss: 0.09774930030107498
Step 177, loss: 0.08002316951751709
Step 178, loss: 0.10534548759460449
Step 179, loss: 0.0872192308306694
Step 180, loss: 0.09978308528661728
Step 181, loss: 0.10233504325151443
Step 182, loss: 0.09696666896343231
Step 183, loss: 0.10517977923154831
Step 184, loss: 0.09348481893539429
Step 185, loss: 0.08087792992591858
Step 186, loss: 0.10089821368455887
Step 187, loss: 0.10613270103931427
Step 188, loss: 0.11796258389949799
Step 189, loss: 0.08400174975395203
Step 190, loss: 0.09354954957962036
Step 191, loss: 0.10984043776988983
Step 192, loss: 0.088462233543396
Step 193, loss: 0.09041773527860641
Step 194, loss: 0.08473967015743256
Running on 8 devices.
Counting all components: ['encoder', 'vq', 'decoder']
Parameter counts:
{'encoder': 18978432, 'vq': 32768, 'decoder': 18978416, 'total': 37989616}
Starting training from step 0...
Step 0, loss: 0.34403467178344727
Step 1, loss: 0.20522570610046387
Step 2, loss: 0.13297320902347565
Step 3, loss: 0.12472856044769287
Step 4, loss: 0.11909522116184235
Step 5, loss: 0.1155850738286972
Step 6, loss: 0.1258772909641266
Step 7, loss: 0.13581234216690063
Step 8, loss: 0.12142564356327057
Step 9, loss: 0.12192649394273758
Step 10, loss: 0.10417830944061279
Step 11, loss: 0.11738790571689606
Step 12, loss: 0.11391294747591019
Step 13, loss: 0.1308845579624176
Step 14, loss: 0.10983618348836899
Step 15, loss: 0.11098787933588028
Step 16, loss: 0.10668867081403732
Step 17, loss: 0.1114255040884018
Step 18, loss: 0.09659188240766525
Step 19, loss: 0.10099179297685623
Step 20, loss: 0.10332012176513672
Step 21, loss: 0.10993895679712296
Step 22, loss: 0.11687967926263809
Step 195, loss: 0.09430243819952011
Step 196, loss: 0.08992063999176025
Step 197, loss: 0.097152940928936
Step 198, loss: 0.0777839720249176
Step 199, loss: 0.08828026056289673
Step 200, loss: 0.09579538553953171
Step 201, loss: 0.07760141789913177
Step 202, loss: 0.08501187711954117
Step 203, loss: 0.08877477049827576
Step 204, loss: 0.09190297871828079
Step 205, loss: 0.09425204247236252
Step 206, loss: 0.07695277035236359
Step 207, loss: 0.09202922135591507
Step 208, loss: 0.1002676859498024
Step 209, loss: 0.08858001977205276
Step 210, loss: 0.08115501701831818
Step 211, loss: 0.08873545378446579
Step 212, loss: 0.1039958968758583
Step 213, loss: 0.09006499499082565
Step 214, loss: 0.08361829817295074
Step 215, loss: 0.10188568383455276
Step 216, loss: 0.10438724607229233
Step 217, loss: 0.09281189739704132
Step 218, loss: 0.09521891921758652
Step 219, loss: 0.1059456467628479
Step 220, loss: 0.10383511334657669
Step 221, loss: 0.09161245077848434
Step 222, loss: 0.0991496816277504
Step 23, loss: 0.09648826718330383
Step 24, loss: 0.08426374942064285
Step 25, loss: 0.11532411724328995
Step 26, loss: 0.10817211866378784
Step 27, loss: 0.10362100601196289
Step 28, loss: 0.09701235592365265
Step 29, loss: 0.09972862899303436
Step 30, loss: 0.09519169479608536
Step 31, loss: 0.09001486748456955
Step 32, loss: 0.08831518888473511
Step 33, loss: 0.08256299793720245
Step 34, loss: 0.0901721939444542
Step 35, loss: 0.0847548171877861
Step 36, loss: 0.08388174325227737
Step 37, loss: 0.08075693994760513
Step 38, loss: 0.08081647008657455
Step 39, loss: 0.10267360508441925
Step 40, loss: 0.09803157299757004
Step 41, loss: 0.08619318157434464
Step 42, loss: 0.09996463358402252
Step 43, loss: 0.08521658182144165
Step 44, loss: 0.10208437591791153
Step 45, loss: 0.08445227146148682
Step 46, loss: 0.11115314066410065
Step 47, loss: 0.08237376809120178
Step 48, loss: 0.09492920339107513
Step 49, loss: 0.09434117376804352
Step 50, loss: 0.09538457542657852
Step 51, loss: 0.10146772116422653
Step 223, loss: 0.08625559508800507
Step 224, loss: 0.07936974614858627
Step 52, loss: 0.10333891957998276
Step 53, loss: 0.09780734777450562
Step 54, loss: 0.0894136130809784
Step 55, loss: 0.08602304011583328
Step 56, loss: 0.08182331174612045
Step 57, loss: 0.08126511424779892
Step 58, loss: 0.09783581644296646
Step 59, loss: 0.09782543033361435
Step 60, loss: 0.09924279153347015
Step 61, loss: 0.08977486938238144
Step 62, loss: 0.10259989649057388
Step 63, loss: 0.10501067340373993
Step 64, loss: 0.10390648245811462
Step 65, loss: 0.08770249038934708
Step 66, loss: 0.09540095180273056
Step 67, loss: 0.09132639318704605
Step 68, loss: 0.10009634494781494
Step 69, loss: 0.09285572171211243
Step 70, loss: 0.08500776439905167
Step 71, loss: 0.09973502159118652
Step 72, loss: 0.08392619341611862
Step 73, loss: 0.09875445067882538
Step 74, loss: 0.10540761053562164
Step 75, loss: 0.09229494631290436
Step 76, loss: 0.10541470348834991
Step 77, loss: 0.08958648890256882
Step 78, loss: 0.08453366160392761
Step 79, loss: 0.090049609541893
Step 80, loss: 0.09214366227388382
Running on 8 devices.
Counting all components: ['encoder', 'vq', 'decoder']
Parameter counts:
{'encoder': 18978432, 'vq': 32768, 'decoder': 18978416, 'total': 37989616}
Starting training from step 0...
Step 0, loss: 0.34403467178344727
Step 1, loss: 0.20522570610046387
Step 2, loss: 0.13297320902347565
Step 3, loss: 0.12472856044769287
Step 4, loss: 0.11909522116184235
Step 5, loss: 0.1155850738286972
Step 6, loss: 0.1258772909641266
Step 7, loss: 0.13581234216690063
Step 8, loss: 0.12142564356327057
Step 9, loss: 0.12192649394273758
Step 10, loss: 0.10417830944061279
Step 11, loss: 0.11738790571689606
Step 12, loss: 0.11391294747591019
Step 13, loss: 0.1308845579624176
Step 14, loss: 0.10983618348836899
Step 15, loss: 0.11098787933588028
Step 16, loss: 0.10668867081403732
Step 17, loss: 0.1114255040884018
Step 18, loss: 0.09659188240766525
Step 19, loss: 0.10099179297685623
Step 20, loss: 0.10332012176513672
Step 21, loss: 0.10993895679712296
Step 22, loss: 0.11687967926263809
Step 81, loss: 0.0938016027212143
Step 82, loss: 0.09607720375061035
Step 83, loss: 0.09525217115879059
Step 84, loss: 0.09727976471185684
Step 85, loss: 0.0878567323088646
Step 86, loss: 0.09850167483091354
Step 87, loss: 0.10143165290355682
Step 88, loss: 0.08512572199106216
Step 89, loss: 0.09779776632785797
Step 90, loss: 0.08772671967744827
Step 91, loss: 0.09805448353290558
Step 92, loss: 0.10006199777126312
Step 93, loss: 0.0881110280752182
Step 94, loss: 0.08710669726133347
Step 95, loss: 0.10951592773199081
Step 96, loss: 0.0913202092051506
Step 97, loss: 0.09101113677024841
Step 98, loss: 0.1027674525976181
Step 99, loss: 0.10735452175140381
Step 100, loss: 0.0928899422287941
Step 101, loss: 0.10774534195661545
Step 102, loss: 0.11379466950893402
Step 103, loss: 0.12731407582759857
Step 104, loss: 0.08464271575212479
Step 105, loss: 0.09762269258499146
Step 106, loss: 0.08544913679361343
Step 107, loss: 0.08664936572313309
Step 108, loss: 0.09289372712373734
Step 109, loss: 0.09938964247703552
Step 23, loss: 0.09648826718330383
Step 24, loss: 0.08426374942064285
Step 25, loss: 0.11532411724328995
Step 26, loss: 0.10817211866378784
Step 27, loss: 0.10362100601196289
Step 28, loss: 0.09701235592365265
Step 29, loss: 0.09972862899303436
Step 30, loss: 0.09519169479608536
Step 31, loss: 0.09001486748456955
Step 32, loss: 0.08831518888473511
Step 33, loss: 0.08256299793720245
Step 34, loss: 0.0901721939444542
Step 35, loss: 0.0847548171877861
Step 36, loss: 0.08388174325227737
Step 37, loss: 0.08075693994760513
Step 38, loss: 0.08081647008657455
Step 39, loss: 0.10267360508441925
Step 40, loss: 0.09803157299757004
Step 41, loss: 0.08619318157434464
Step 42, loss: 0.09996463358402252
Step 43, loss: 0.08521658182144165
Step 44, loss: 0.10208437591791153
Step 45, loss: 0.08445227146148682
Step 46, loss: 0.11115314066410065
Step 47, loss: 0.08237376809120178
Step 48, loss: 0.09492920339107513
Step 49, loss: 0.09434117376804352
Step 50, loss: 0.09538457542657852
Step 51, loss: 0.10146772116422653
Step 110, loss: 0.10275784134864807
Step 52, loss: 0.10333891957998276
Step 53, loss: 0.09780734777450562
Step 54, loss: 0.0894136130809784
Step 55, loss: 0.08602304011583328
Step 56, loss: 0.08182331174612045
Step 57, loss: 0.08126511424779892
Step 58, loss: 0.09783581644296646
Step 59, loss: 0.09782543033361435
Step 60, loss: 0.09924279153347015
Step 61, loss: 0.08977486938238144
Step 62, loss: 0.10259989649057388
Step 63, loss: 0.10501067340373993
Step 64, loss: 0.10390648245811462
Step 65, loss: 0.08770249038934708
Step 66, loss: 0.09540095180273056
Step 67, loss: 0.09132639318704605
Step 68, loss: 0.10009634494781494
Step 69, loss: 0.09285572171211243
Step 70, loss: 0.08500776439905167
Step 71, loss: 0.09973502159118652
Step 72, loss: 0.08392619341611862
Step 73, loss: 0.09875445067882538
Step 74, loss: 0.10540761053562164
Step 75, loss: 0.09229494631290436
Step 76, loss: 0.10541470348834991
Step 77, loss: 0.08958648890256882
Step 78, loss: 0.08453366160392761
Step 79, loss: 0.090049609541893
Step 80, loss: 0.09214366227388382
Step 111, loss: 0.08993981033563614
Step 112, loss: 0.0892101600766182
Step 113, loss: 0.08680275827646255
Step 114, loss: 0.10558158904314041
Step 115, loss: 0.0954781174659729
Step 116, loss: 0.07423624396324158
Step 117, loss: 0.0990038588643074
Step 118, loss: 0.09334039688110352
Step 119, loss: 0.10126888006925583
Step 120, loss: 0.09113641828298569
Step 121, loss: 0.08533597737550735
Step 122, loss: 0.08464204519987106
Step 123, loss: 0.09717801213264465
Step 124, loss: 0.1001516729593277
Step 125, loss: 0.09672964364290237
Step 126, loss: 0.10850290209054947
Step 127, loss: 0.09636508673429489
Step 128, loss: 0.08310791850090027
Step 129, loss: 0.09116649627685547
Step 130, loss: 0.08114764839410782
Step 131, loss: 0.09093617647886276
Step 132, loss: 0.09965819865465164
Step 133, loss: 0.08848806470632553
Step 134, loss: 0.10375767946243286
Step 135, loss: 0.07391893118619919
Step 136, loss: 0.08318819850683212
Step 137, loss: 0.08422021567821503
Step 138, loss: 0.10729701817035675
Step 81, loss: 0.0938016027212143
Step 82, loss: 0.09607720375061035
Step 83, loss: 0.09525217115879059
Step 84, loss: 0.09727976471185684
Step 85, loss: 0.0878567323088646
Step 86, loss: 0.09850167483091354
Step 87, loss: 0.10143165290355682
Step 88, loss: 0.08512572199106216
Step 89, loss: 0.09779776632785797
Step 90, loss: 0.08772671967744827
Step 91, loss: 0.09805448353290558
Step 92, loss: 0.10006199777126312
Step 93, loss: 0.0881110280752182
Step 94, loss: 0.08710669726133347
Step 95, loss: 0.10951592773199081
Step 96, loss: 0.0913202092051506
Step 97, loss: 0.09101113677024841
Step 98, loss: 0.1027674525976181
Step 99, loss: 0.10735452175140381
Step 100, loss: 0.0928899422287941
Step 101, loss: 0.10774534195661545
Step 102, loss: 0.11379466950893402
Step 103, loss: 0.12731407582759857
Step 104, loss: 0.08464271575212479
Step 105, loss: 0.09762269258499146
Step 106, loss: 0.08544913679361343
Step 107, loss: 0.08664936572313309
Step 108, loss: 0.09289372712373734
Step 109, loss: 0.09938964247703552
Step 139, loss: 0.08231347799301147
Step 140, loss: 0.09107917547225952
Step 141, loss: 0.09596633911132812
Step 142, loss: 0.08139874041080475
Step 143, loss: 0.09055333584547043
Step 144, loss: 0.08543172478675842
Step 145, loss: 0.08763666450977325
Step 146, loss: 0.0839858427643776
Step 147, loss: 0.08173791319131851
Step 148, loss: 0.11152316629886627
Step 149, loss: 0.09568681567907333
Step 150, loss: 0.10091811418533325
Step 151, loss: 0.0921492949128151
Step 152, loss: 0.09212498366832733
Step 153, loss: 0.09429033100605011
Step 154, loss: 0.09491580724716187
Step 155, loss: 0.09256651997566223
Step 156, loss: 0.10165049135684967
Step 157, loss: 0.09216190129518509
Step 158, loss: 0.09417818486690521
Step 159, loss: 0.08085789531469345
Step 160, loss: 0.09474841505289078
Step 161, loss: 0.09131263941526413
Step 162, loss: 0.1044401004910469
Step 163, loss: 0.09948422014713287
Step 164, loss: 0.09638313204050064
Step 165, loss: 0.10100198537111282
Step 166, loss: 0.08546408265829086
Step 110, loss: 0.10275784134864807
Step 167, loss: 0.09995202720165253
Step 168, loss: 0.0901375561952591
Step 169, loss: 0.1005106195807457
Step 170, loss: 0.09844204038381577
Step 171, loss: 0.09980332106351852
Step 172, loss: 0.09990215301513672
Step 173, loss: 0.09080781042575836
Step 174, loss: 0.09667211025953293
Step 175, loss: 0.09836149215698242
Step 176, loss: 0.09774930030107498
Step 177, loss: 0.08002316951751709
Step 178, loss: 0.10534548759460449
Step 179, loss: 0.0872192308306694
Step 180, loss: 0.09978308528661728
Step 181, loss: 0.10233504325151443
Step 182, loss: 0.09696666896343231
Step 183, loss: 0.10517977923154831
Step 184, loss: 0.09348481893539429
Step 185, loss: 0.08087792992591858
Step 186, loss: 0.10089821368455887
Step 187, loss: 0.10613270103931427
Step 188, loss: 0.11796258389949799
Step 189, loss: 0.08400174975395203
Step 190, loss: 0.09354954957962036
Step 191, loss: 0.10984043776988983
Step 192, loss: 0.088462233543396
Step 193, loss: 0.09041773527860641
Step 194, loss: 0.08473967015743256
Step 111, loss: 0.08993981033563614
Step 112, loss: 0.0892101600766182
Step 113, loss: 0.08680275827646255
Step 114, loss: 0.10558158904314041
Step 115, loss: 0.0954781174659729
Step 116, loss: 0.07423624396324158
Step 117, loss: 0.0990038588643074
Step 118, loss: 0.09334039688110352
Step 119, loss: 0.10126888006925583
Step 120, loss: 0.09113641828298569
Step 121, loss: 0.08533597737550735
Step 122, loss: 0.08464204519987106
Step 123, loss: 0.09717801213264465
Step 124, loss: 0.1001516729593277
Step 125, loss: 0.09672964364290237
Step 126, loss: 0.10850290209054947
Step 127, loss: 0.09636508673429489
Step 128, loss: 0.08310791850090027
Step 129, loss: 0.09116649627685547
Step 130, loss: 0.08114764839410782
Step 131, loss: 0.09093617647886276
Step 132, loss: 0.09965819865465164
Step 133, loss: 0.08848806470632553
Step 134, loss: 0.10375767946243286
Step 135, loss: 0.07391893118619919
Step 136, loss: 0.08318819850683212
Step 137, loss: 0.08422021567821503
Step 138, loss: 0.10729701817035675
Step 195, loss: 0.09430243819952011
Step 196, loss: 0.08992063999176025
Step 197, loss: 0.097152940928936
Step 198, loss: 0.0777839720249176
Step 199, loss: 0.08828026056289673
Step 200, loss: 0.09579538553953171
Step 201, loss: 0.07760141789913177
Step 202, loss: 0.08501187711954117
Step 203, loss: 0.08877477049827576
Step 204, loss: 0.09190297871828079
Step 205, loss: 0.09425204247236252
Step 206, loss: 0.07695277035236359
Step 207, loss: 0.09202922135591507
Step 208, loss: 0.1002676859498024
Step 209, loss: 0.08858001977205276
Step 210, loss: 0.08115501701831818
Step 211, loss: 0.08873545378446579
Step 212, loss: 0.1039958968758583
Step 213, loss: 0.09006499499082565
Step 214, loss: 0.08361829817295074
Step 215, loss: 0.10188568383455276
Step 216, loss: 0.10438724607229233
Step 217, loss: 0.09281189739704132
Step 218, loss: 0.09521891921758652
Step 219, loss: 0.1059456467628479
Step 220, loss: 0.10383511334657669
Step 221, loss: 0.09161245077848434
Step 222, loss: 0.0991496816277504
Step 139, loss: 0.08231347799301147
Step 140, loss: 0.09107917547225952
Step 141, loss: 0.09596633911132812
Step 142, loss: 0.08139874041080475
Step 143, loss: 0.09055333584547043
Step 144, loss: 0.08543172478675842
Step 145, loss: 0.08763666450977325
Step 146, loss: 0.0839858427643776
Step 147, loss: 0.08173791319131851
Step 148, loss: 0.11152316629886627
Step 149, loss: 0.09568681567907333
Step 150, loss: 0.10091811418533325
Step 151, loss: 0.0921492949128151
Step 152, loss: 0.09212498366832733
Step 153, loss: 0.09429033100605011
Step 154, loss: 0.09491580724716187
Step 155, loss: 0.09256651997566223
Step 156, loss: 0.10165049135684967
Step 157, loss: 0.09216190129518509
Step 158, loss: 0.09417818486690521
Step 159, loss: 0.08085789531469345
Step 160, loss: 0.09474841505289078
Step 161, loss: 0.09131263941526413
Step 162, loss: 0.1044401004910469
Step 163, loss: 0.09948422014713287
Step 164, loss: 0.09638313204050064
Step 165, loss: 0.10100198537111282
Step 166, loss: 0.08546408265829086
Step 223, loss: 0.08625559508800507
Step 224, loss: 0.07936974614858627
Step 167, loss: 0.09995202720165253
Step 168, loss: 0.0901375561952591
Step 169, loss: 0.1005106195807457
Step 170, loss: 0.09844204038381577
Step 171, loss: 0.09980332106351852
Step 172, loss: 0.09990215301513672
Step 173, loss: 0.09080781042575836
Step 174, loss: 0.09667211025953293
Step 175, loss: 0.09836149215698242
Step 176, loss: 0.09774930030107498
Step 177, loss: 0.08002316951751709
Step 178, loss: 0.10534548759460449
Step 179, loss: 0.0872192308306694
Step 180, loss: 0.09978308528661728
Step 181, loss: 0.10233504325151443
Step 182, loss: 0.09696666896343231
Step 183, loss: 0.10517977923154831
Step 184, loss: 0.09348481893539429
Step 185, loss: 0.08087792992591858
Step 186, loss: 0.10089821368455887
Step 187, loss: 0.10613270103931427
Step 188, loss: 0.11796258389949799
Step 189, loss: 0.08400174975395203
Step 190, loss: 0.09354954957962036
Step 191, loss: 0.10984043776988983
Step 192, loss: 0.088462233543396
Step 193, loss: 0.09041773527860641
Step 194, loss: 0.08473967015743256
Step 195, loss: 0.09430243819952011
Step 196, loss: 0.08992063999176025
Step 197, loss: 0.097152940928936
Step 198, loss: 0.0777839720249176
Step 199, loss: 0.08828026056289673
Step 200, loss: 0.09579538553953171
Step 201, loss: 0.07760141789913177
Step 202, loss: 0.08501187711954117
Step 203, loss: 0.08877477049827576
Step 204, loss: 0.09190297871828079
Step 205, loss: 0.09425204247236252
Step 206, loss: 0.07695277035236359
Step 207, loss: 0.09202922135591507
Step 208, loss: 0.1002676859498024
Step 209, loss: 0.08858001977205276
Step 210, loss: 0.08115501701831818
Step 211, loss: 0.08873545378446579
Step 212, loss: 0.1039958968758583
Step 213, loss: 0.09006499499082565
Step 214, loss: 0.08361829817295074
Step 215, loss: 0.10188568383455276
Step 216, loss: 0.10438724607229233
Step 217, loss: 0.09281189739704132
Step 218, loss: 0.09521891921758652
Step 219, loss: 0.1059456467628479
Step 220, loss: 0.10383511334657669
Step 221, loss: 0.09161245077848434
Step 222, loss: 0.0991496816277504
Step 223, loss: 0.08625559508800507
Step 224, loss: 0.07936974614858627
Step 225, loss: 0.0915277972817421
Step 226, loss: 0.09734950959682465
Step 227, loss: 0.09756748378276825
Step 228, loss: 0.0834878459572792
Step 229, loss: 0.10312364250421524
Step 230, loss: 0.09948989003896713
Step 231, loss: 0.08834394067525864
Step 232, loss: 0.0942598208785057
Step 233, loss: 0.08965849876403809
Step 234, loss: 0.08935556560754776
Step 235, loss: 0.089423768222332
Step 236, loss: 0.10344216972589493
Step 237, loss: 0.07752181589603424
Step 238, loss: 0.09235274791717529
Step 239, loss: 0.09368637204170227
Step 240, loss: 0.09884591400623322
Step 241, loss: 0.08072476089000702
Step 242, loss: 0.11127684265375137
Step 243, loss: 0.10105646401643753
Step 244, loss: 0.10253992676734924
Step 245, loss: 0.11403459310531616
Step 246, loss: 0.10600597411394119
Step 247, loss: 0.09744757413864136
Step 248, loss: 0.10752671957015991
Step 249, loss: 0.09257692843675613
Step 250, loss: 0.10353343188762665
Step 251, loss: 0.09556671977043152
Step 252, loss: 0.10835393518209457
Step 253, loss: 0.099678635597229
Step 254, loss: 0.08157074451446533
Step 255, loss: 0.092997707426548
Step 256, loss: 0.10462192445993423
Step 257, loss: 0.08885577321052551
Step 258, loss: 0.09344454854726791
Step 259, loss: 0.08895348757505417
Step 260, loss: 0.08721040934324265
Step 261, loss: 0.1102367639541626
Step 262, loss: 0.09160476177930832
Step 263, loss: 0.08556541055440903
Step 264, loss: 0.0972244068980217
Step 265, loss: 0.09133235365152359
Step 266, loss: 0.09668958932161331
Step 267, loss: 0.09747679531574249
Step 268, loss: 0.09259546548128128
Step 269, loss: 0.10741567611694336
Step 270, loss: 0.10364018380641937
Step 271, loss: 0.09192583709955215
Step 272, loss: 0.0866183191537857
Step 273, loss: 0.0995253473520279
Step 274, loss: 0.08897469192743301
Step 275, loss: 0.10126527398824692
Step 276, loss: 0.11130071431398392
Step 277, loss: 0.07839206606149673
Step 278, loss: 0.08253416419029236
Step 279, loss: 0.10969360172748566
Step 280, loss: 0.07669182121753693
Step 225, loss: 0.0915277972817421
Step 226, loss: 0.09734950959682465
Step 227, loss: 0.09756748378276825
Step 228, loss: 0.0834878459572792
Step 229, loss: 0.10312364250421524
Step 230, loss: 0.09948989003896713
Step 231, loss: 0.08834394067525864
Step 232, loss: 0.0942598208785057
Step 233, loss: 0.08965849876403809
Step 234, loss: 0.08935556560754776
Step 235, loss: 0.089423768222332
Step 236, loss: 0.10344216972589493
Step 237, loss: 0.07752181589603424
Step 238, loss: 0.09235274791717529
Step 239, loss: 0.09368637204170227
Step 240, loss: 0.09884591400623322
Step 241, loss: 0.08072476089000702
Step 242, loss: 0.11127684265375137
Step 243, loss: 0.10105646401643753
Step 244, loss: 0.10253992676734924
Step 245, loss: 0.11403459310531616
Step 246, loss: 0.10600597411394119
Step 247, loss: 0.09744757413864136
Step 248, loss: 0.10752671957015991
Step 249, loss: 0.09257692843675613
Step 250, loss: 0.10353343188762665
Step 251, loss: 0.09556671977043152
Step 252, loss: 0.10835393518209457
Step 281, loss: 0.08920909464359283
Step 282, loss: 0.0840398371219635
Step 283, loss: 0.10449270159006119
Step 284, loss: 0.07988334447145462
Step 285, loss: 0.0852045938372612
Step 286, loss: 0.09351927787065506
Step 287, loss: 0.10052774846553802
Step 288, loss: 0.09816160053014755
Step 289, loss: 0.08929572254419327
Step 290, loss: 0.08400838822126389
Step 291, loss: 0.09651218354701996
Step 292, loss: 0.11312537640333176
Step 293, loss: 0.09710629284381866
Step 294, loss: 0.10586521774530411
Step 295, loss: 0.08972751349210739
Step 296, loss: 0.09800779819488525
Step 297, loss: 0.09882067888975143
Step 298, loss: 0.09476090967655182
Step 299, loss: 0.08048875629901886
Step 300, loss: 0.0883999913930893
Step 301, loss: 0.10911105573177338
Step 302, loss: 0.11065614968538284
Step 303, loss: 0.08949830383062363
Step 304, loss: 0.08194184303283691
Step 305, loss: 0.07901506870985031
Step 306, loss: 0.09255432337522507
Step 307, loss: 0.10475584864616394
Step 308, loss: 0.09881763905286789
Step 253, loss: 0.099678635597229
Step 254, loss: 0.08157074451446533
Step 255, loss: 0.092997707426548
Step 256, loss: 0.10462192445993423
Step 257, loss: 0.08885577321052551
Step 258, loss: 0.09344454854726791
Step 259, loss: 0.08895348757505417
Step 260, loss: 0.08721040934324265
Step 261, loss: 0.1102367639541626
Step 262, loss: 0.09160476177930832
Step 263, loss: 0.08556541055440903
Step 264, loss: 0.0972244068980217
Step 265, loss: 0.09133235365152359
Step 266, loss: 0.09668958932161331
Step 267, loss: 0.09747679531574249
Step 268, loss: 0.09259546548128128
Step 269, loss: 0.10741567611694336
Step 270, loss: 0.10364018380641937
Step 271, loss: 0.09192583709955215
Step 272, loss: 0.0866183191537857
Step 273, loss: 0.0995253473520279
Step 274, loss: 0.08897469192743301
Step 275, loss: 0.10126527398824692
Step 276, loss: 0.11130071431398392
Step 277, loss: 0.07839206606149673
Step 278, loss: 0.08253416419029236
Step 279, loss: 0.10969360172748566
Step 280, loss: 0.07669182121753693
Step 309, loss: 0.09676898270845413
Step 310, loss: 0.09536278247833252
Step 311, loss: 0.08626880496740341
Step 312, loss: 0.09074317663908005
Step 313, loss: 0.0872684195637703
Step 314, loss: 0.09509485214948654
Step 315, loss: 0.08823093771934509
Step 316, loss: 0.09192297607660294
Step 317, loss: 0.09358572214841843
Step 318, loss: 0.09756763279438019
Step 319, loss: 0.0958862453699112
Step 320, loss: 0.09939354658126831
Step 321, loss: 0.09518513083457947
Step 322, loss: 0.09240536391735077
Step 323, loss: 0.10315807908773422
Step 324, loss: 0.09763992577791214
Step 325, loss: 0.08891928195953369
Step 326, loss: 0.09153340756893158
Step 327, loss: 0.1023201271891594
Step 328, loss: 0.08948364108800888
Step 329, loss: 0.08565863221883774
Step 330, loss: 0.09451939165592194
Step 331, loss: 0.09675204008817673
Step 332, loss: 0.10256344825029373
Step 333, loss: 0.10136643797159195
Step 334, loss: 0.0872085765004158
Step 335, loss: 0.10480964928865433
Step 336, loss: 0.09052358567714691
Step 281, loss: 0.08920909464359283
Step 282, loss: 0.0840398371219635
Step 283, loss: 0.10449270159006119
Step 284, loss: 0.07988334447145462
Step 285, loss: 0.0852045938372612
Step 286, loss: 0.09351927787065506
Step 287, loss: 0.10052774846553802
Step 288, loss: 0.09816160053014755
Step 289, loss: 0.08929572254419327
Step 290, loss: 0.08400838822126389
Step 291, loss: 0.09651218354701996
Step 292, loss: 0.11312537640333176
Step 293, loss: 0.09710629284381866
Step 294, loss: 0.10586521774530411
Step 295, loss: 0.08972751349210739
Step 296, loss: 0.09800779819488525
Step 297, loss: 0.09882067888975143
Step 298, loss: 0.09476090967655182
Step 299, loss: 0.08048875629901886
Step 300, loss: 0.0883999913930893
Step 301, loss: 0.10911105573177338
Step 302, loss: 0.11065614968538284
Step 303, loss: 0.08949830383062363
Step 304, loss: 0.08194184303283691
Step 305, loss: 0.07901506870985031
Step 306, loss: 0.09255432337522507
Step 307, loss: 0.10475584864616394
Step 308, loss: 0.09881763905286789
Step 337, loss: 0.10486156493425369
Step 338, loss: 0.09143216907978058
Step 309, loss: 0.09676898270845413
Step 310, loss: 0.09536278247833252
Step 311, loss: 0.08626880496740341
Step 312, loss: 0.09074317663908005
Step 313, loss: 0.0872684195637703
Step 314, loss: 0.09509485214948654
Step 315, loss: 0.08823093771934509
Step 316, loss: 0.09192297607660294
Step 317, loss: 0.09358572214841843
Step 318, loss: 0.09756763279438019
Step 319, loss: 0.0958862453699112
Step 320, loss: 0.09939354658126831
Step 321, loss: 0.09518513083457947
Step 322, loss: 0.09240536391735077
Step 323, loss: 0.10315807908773422
Step 324, loss: 0.09763992577791214
Step 325, loss: 0.08891928195953369
Step 326, loss: 0.09153340756893158
Step 327, loss: 0.1023201271891594
Step 328, loss: 0.08948364108800888
Step 329, loss: 0.08565863221883774
Step 330, loss: 0.09451939165592194
Step 331, loss: 0.09675204008817673
Step 332, loss: 0.10256344825029373
Step 333, loss: 0.10136643797159195
Step 334, loss: 0.0872085765004158
Step 335, loss: 0.10480964928865433
Step 336, loss: 0.09052358567714691
Step 339, loss: 0.07445961236953735
Step 340, loss: 0.0805797353386879
Step 341, loss: 0.09287401288747787
Step 342, loss: 0.09077334403991699
Step 343, loss: 0.09919575601816177
Step 344, loss: 0.08214805275201797
Step 345, loss: 0.10290668159723282
Step 346, loss: 0.09631843119859695
Step 347, loss: 0.08739935606718063
Step 348, loss: 0.10452226549386978
Step 349, loss: 0.08685916662216187
Step 350, loss: 0.089399553835392
Step 351, loss: 0.09517337381839752
Step 352, loss: 0.0884610190987587
Step 353, loss: 0.08564262837171555
Step 354, loss: 0.0826452225446701
Step 355, loss: 0.1019214615225792
Step 356, loss: 0.09059467166662216
Step 357, loss: 0.08375606685876846
Step 358, loss: 0.0884118601679802
Step 359, loss: 0.08985290676355362
Step 360, loss: 0.0931832417845726
Step 361, loss: 0.10271522402763367
Step 362, loss: 0.08568371087312698
Step 363, loss: 0.08449043333530426
Step 364, loss: 0.1045188307762146
Step 365, loss: 0.10840359330177307
Step 366, loss: 0.09198035299777985
Step 337, loss: 0.10486156493425369
Step 338, loss: 0.09143216907978058
Step 367, loss: 0.10256067663431168
Step 368, loss: 0.08583091199398041
Step 369, loss: 0.09691421687602997
Step 370, loss: 0.09554970264434814
Step 371, loss: 0.0964154377579689
Step 372, loss: 0.09535901993513107
Step 373, loss: 0.09217069298028946
Step 374, loss: 0.10800403356552124
Step 375, loss: 0.0908806174993515
Step 376, loss: 0.09458478540182114
Step 377, loss: 0.08760320395231247
Step 378, loss: 0.0933598205447197
Step 379, loss: 0.09907391667366028
Step 380, loss: 0.10368257761001587
Step 381, loss: 0.09667558968067169
Step 382, loss: 0.09884196519851685
Step 383, loss: 0.08062954246997833
Step 384, loss: 0.10893204063177109
Step 385, loss: 0.08792293071746826
Step 386, loss: 0.09512859582901001
Step 387, loss: 0.09335962682962418
Step 388, loss: 0.0782364085316658
Step 389, loss: 0.09548364579677582
Step 390, loss: 0.09789202362298965
Step 391, loss: 0.10904669016599655
Step 392, loss: 0.09689964354038239
Step 393, loss: 0.10324911028146744
Step 394, loss: 0.09940492361783981
Step 339, loss: 0.07445961236953735
Step 340, loss: 0.0805797353386879
Step 341, loss: 0.09287401288747787
Step 342, loss: 0.09077334403991699
Step 343, loss: 0.09919575601816177
Step 344, loss: 0.08214805275201797
Step 345, loss: 0.10290668159723282
Step 346, loss: 0.09631843119859695
Step 347, loss: 0.08739935606718063
Step 348, loss: 0.10452226549386978
Step 349, loss: 0.08685916662216187
Step 350, loss: 0.089399553835392
Step 351, loss: 0.09517337381839752
Step 352, loss: 0.0884610190987587
Step 353, loss: 0.08564262837171555
Step 354, loss: 0.0826452225446701
Step 355, loss: 0.1019214615225792
Step 356, loss: 0.09059467166662216
Step 357, loss: 0.08375606685876846
Step 358, loss: 0.0884118601679802
Step 359, loss: 0.08985290676355362
Step 360, loss: 0.0931832417845726
Step 361, loss: 0.10271522402763367
Step 362, loss: 0.08568371087312698
Step 363, loss: 0.08449043333530426
Step 364, loss: 0.1045188307762146
Step 365, loss: 0.10840359330177307
Step 366, loss: 0.09198035299777985
Step 395, loss: 0.08620208501815796
Step 396, loss: 0.0948353037238121
Step 397, loss: 0.09858851134777069
Step 398, loss: 0.08667661994695663
Step 399, loss: 0.07714290916919708
Step 400, loss: 0.09605338424444199
Step 401, loss: 0.08259181678295135
Step 402, loss: 0.10587847977876663
Step 403, loss: 0.10170406103134155
Step 404, loss: 0.09050613641738892
Step 405, loss: 0.0847139060497284
Step 406, loss: 0.08444532006978989
Step 407, loss: 0.102294921875
Step 408, loss: 0.09200237691402435
Step 409, loss: 0.1071396917104721
Step 410, loss: 0.08932185173034668
Step 411, loss: 0.09256329387426376
Step 412, loss: 0.08985482901334763
Step 413, loss: 0.09989946335554123
Step 414, loss: 0.09001356363296509
Step 415, loss: 0.08021972328424454
Step 416, loss: 0.09103049337863922
Step 417, loss: 0.0843631774187088
Step 418, loss: 0.10525093972682953
Step 419, loss: 0.09829505532979965
Step 420, loss: 0.1055281013250351
Step 421, loss: 0.0941363275051117
Step 422, loss: 0.10333140194416046
Step 367, loss: 0.10256067663431168
Step 368, loss: 0.08583091199398041
Step 369, loss: 0.09691421687602997
Step 370, loss: 0.09554970264434814
Step 371, loss: 0.0964154377579689
Step 372, loss: 0.09535901993513107
Step 373, loss: 0.09217069298028946
Step 374, loss: 0.10800403356552124
Step 375, loss: 0.0908806174993515
Step 376, loss: 0.09458478540182114
Step 377, loss: 0.08760320395231247
Step 378, loss: 0.0933598205447197
Step 379, loss: 0.09907391667366028
Step 380, loss: 0.10368257761001587
Step 381, loss: 0.09667558968067169
Step 382, loss: 0.09884196519851685
Step 383, loss: 0.08062954246997833
Step 384, loss: 0.10893204063177109
Step 385, loss: 0.08792293071746826
Step 386, loss: 0.09512859582901001
Step 387, loss: 0.09335962682962418
Step 388, loss: 0.0782364085316658
Step 389, loss: 0.09548364579677582
Step 390, loss: 0.09789202362298965
Step 391, loss: 0.10904669016599655
Step 392, loss: 0.09689964354038239
Step 393, loss: 0.10324911028146744
Step 394, loss: 0.09940492361783981
Step 423, loss: 0.08642852306365967
Step 424, loss: 0.09760038554668427
Step 425, loss: 0.08078045397996902
Step 426, loss: 0.08566514402627945
Step 427, loss: 0.09726865589618683
Step 428, loss: 0.08297593146562576
Step 429, loss: 0.1069452315568924
Step 430, loss: 0.08628760278224945
Step 431, loss: 0.08262874186038971
Step 432, loss: 0.08956742286682129
Step 433, loss: 0.08533556014299393
Step 434, loss: 0.0934138149023056
Step 435, loss: 0.10118114203214645
Step 436, loss: 0.08869095146656036
Step 437, loss: 0.08668148517608643
Step 438, loss: 0.09466224908828735
Step 439, loss: 0.0809057280421257
Step 440, loss: 0.08602495491504669
Step 441, loss: 0.10809214413166046
Step 442, loss: 0.08890371024608612
Step 443, loss: 0.10107913613319397
Step 444, loss: 0.09205890446901321
Step 445, loss: 0.08381463587284088
Step 446, loss: 0.10211075842380524
Step 447, loss: 0.09884621948003769
Step 448, loss: 0.0960962101817131
Step 449, loss: 0.08752726763486862
Step 450, loss: 0.09262149780988693
Step 395, loss: 0.08620208501815796
Step 396, loss: 0.0948353037238121
Step 397, loss: 0.09858851134777069
Step 398, loss: 0.08667661994695663
Step 399, loss: 0.07714290916919708
Step 400, loss: 0.09605338424444199
Step 401, loss: 0.08259181678295135
Step 402, loss: 0.10587847977876663
Step 403, loss: 0.10170406103134155
Step 404, loss: 0.09050613641738892
Step 405, loss: 0.0847139060497284
Step 406, loss: 0.08444532006978989
Step 407, loss: 0.102294921875
Step 408, loss: 0.09200237691402435
Step 409, loss: 0.1071396917104721
Step 410, loss: 0.08932185173034668
Step 411, loss: 0.09256329387426376
Step 412, loss: 0.08985482901334763
Step 413, loss: 0.09989946335554123
Step 414, loss: 0.09001356363296509
Step 415, loss: 0.08021972328424454
Step 416, loss: 0.09103049337863922
Step 417, loss: 0.0843631774187088
Step 418, loss: 0.10525093972682953
Step 419, loss: 0.09829505532979965
Step 420, loss: 0.1055281013250351
Step 421, loss: 0.0941363275051117
Step 422, loss: 0.10333140194416046
Step 451, loss: 0.09735321998596191
Step 452, loss: 0.08930101245641708
Step 423, loss: 0.08642852306365967
Step 424, loss: 0.09760038554668427
Step 425, loss: 0.08078045397996902
Step 426, loss: 0.08566514402627945
Step 427, loss: 0.09726865589618683
Step 428, loss: 0.08297593146562576
Step 429, loss: 0.1069452315568924
Step 430, loss: 0.08628760278224945
Step 431, loss: 0.08262874186038971
Step 432, loss: 0.08956742286682129
Step 433, loss: 0.08533556014299393
Step 434, loss: 0.0934138149023056
Step 435, loss: 0.10118114203214645
Step 436, loss: 0.08869095146656036
Step 437, loss: 0.08668148517608643
Step 438, loss: 0.09466224908828735
Step 439, loss: 0.0809057280421257
Step 440, loss: 0.08602495491504669
Step 441, loss: 0.10809214413166046
Step 442, loss: 0.08890371024608612
Step 443, loss: 0.10107913613319397
Step 444, loss: 0.09205890446901321
Step 445, loss: 0.08381463587284088
Step 446, loss: 0.10211075842380524
Step 447, loss: 0.09884621948003769
Step 448, loss: 0.0960962101817131
Step 449, loss: 0.08752726763486862
Step 450, loss: 0.09262149780988693
Step 453, loss: 0.08560588210821152
Step 451, loss: 0.09735321998596191
Step 452, loss: 0.08930101245641708
Step 225, loss: 0.0915277972817421
Step 226, loss: 0.09734950959682465
Step 227, loss: 0.09756748378276825
Step 228, loss: 0.0834878459572792
Step 229, loss: 0.10312364250421524
Step 230, loss: 0.09948989003896713
Step 231, loss: 0.08834394067525864
Step 232, loss: 0.0942598208785057
Step 233, loss: 0.08965849876403809
Step 234, loss: 0.08935556560754776
Step 235, loss: 0.089423768222332
Step 236, loss: 0.10344216972589493
Step 237, loss: 0.07752181589603424
Step 238, loss: 0.09235274791717529
Step 239, loss: 0.09368637204170227
Step 240, loss: 0.09884591400623322
Step 241, loss: 0.08072476089000702
Step 242, loss: 0.11127684265375137
Step 243, loss: 0.10105646401643753
Step 244, loss: 0.10253992676734924
Step 245, loss: 0.11403459310531616
Step 246, loss: 0.10600597411394119
Step 247, loss: 0.09744757413864136
Step 248, loss: 0.10752671957015991
Step 249, loss: 0.09257692843675613
Step 250, loss: 0.10353343188762665
Step 251, loss: 0.09556671977043152
Step 252, loss: 0.10835393518209457
Step 453, loss: 0.08560588210821152
Step 253, loss: 0.099678635597229
Step 254, loss: 0.08157074451446533
Step 255, loss: 0.092997707426548
Step 256, loss: 0.10462192445993423
Step 257, loss: 0.08885577321052551
Step 258, loss: 0.09344454854726791
Step 259, loss: 0.08895348757505417
Step 260, loss: 0.08721040934324265
Step 261, loss: 0.1102367639541626
Step 262, loss: 0.09160476177930832
Step 263, loss: 0.08556541055440903
Step 264, loss: 0.0972244068980217
Step 265, loss: 0.09133235365152359
Step 266, loss: 0.09668958932161331
Step 267, loss: 0.09747679531574249
Step 268, loss: 0.09259546548128128
Step 269, loss: 0.10741567611694336
Step 270, loss: 0.10364018380641937
Step 271, loss: 0.09192583709955215
Step 272, loss: 0.0866183191537857
Step 273, loss: 0.0995253473520279
Step 274, loss: 0.08897469192743301
Step 275, loss: 0.10126527398824692
Step 276, loss: 0.11130071431398392
Step 277, loss: 0.07839206606149673
Step 278, loss: 0.08253416419029236
Step 279, loss: 0.10969360172748566
Step 280, loss: 0.07669182121753693
Step 225, loss: 0.0915277972817421
Step 226, loss: 0.09734950959682465
Step 227, loss: 0.09756748378276825
Step 228, loss: 0.0834878459572792
Step 229, loss: 0.10312364250421524
Step 230, loss: 0.09948989003896713
Step 231, loss: 0.08834394067525864
Step 232, loss: 0.0942598208785057
Step 233, loss: 0.08965849876403809
Step 234, loss: 0.08935556560754776
Step 235, loss: 0.089423768222332
Step 236, loss: 0.10344216972589493
Step 237, loss: 0.07752181589603424
Step 238, loss: 0.09235274791717529
Step 239, loss: 0.09368637204170227
Step 240, loss: 0.09884591400623322
Step 241, loss: 0.08072476089000702
Step 242, loss: 0.11127684265375137
Step 243, loss: 0.10105646401643753
Step 244, loss: 0.10253992676734924
Step 245, loss: 0.11403459310531616
Step 246, loss: 0.10600597411394119
Step 247, loss: 0.09744757413864136
Step 248, loss: 0.10752671957015991
Step 249, loss: 0.09257692843675613
Step 250, loss: 0.10353343188762665
Step 251, loss: 0.09556671977043152
Step 252, loss: 0.10835393518209457
Step 281, loss: 0.08920909464359283
Step 282, loss: 0.0840398371219635
Step 283, loss: 0.10449270159006119
Step 284, loss: 0.07988334447145462
Step 285, loss: 0.0852045938372612
Step 286, loss: 0.09351927787065506
Step 287, loss: 0.10052774846553802
Step 288, loss: 0.09816160053014755
Step 289, loss: 0.08929572254419327
Step 290, loss: 0.08400838822126389
Step 291, loss: 0.09651218354701996
Step 292, loss: 0.11312537640333176
Step 293, loss: 0.09710629284381866
Step 294, loss: 0.10586521774530411
Step 295, loss: 0.08972751349210739
Step 296, loss: 0.09800779819488525
Step 297, loss: 0.09882067888975143
Step 298, loss: 0.09476090967655182
Step 299, loss: 0.08048875629901886
Step 300, loss: 0.0883999913930893
Step 301, loss: 0.10911105573177338
Step 302, loss: 0.11065614968538284
Step 303, loss: 0.08949830383062363
Step 304, loss: 0.08194184303283691
Step 305, loss: 0.07901506870985031
Step 306, loss: 0.09255432337522507
Step 307, loss: 0.10475584864616394
Step 308, loss: 0.09881763905286789
Step 253, loss: 0.099678635597229
Step 254, loss: 0.08157074451446533
Step 255, loss: 0.092997707426548
Step 256, loss: 0.10462192445993423
Step 257, loss: 0.08885577321052551
Step 258, loss: 0.09344454854726791
Step 259, loss: 0.08895348757505417
Step 260, loss: 0.08721040934324265
Step 261, loss: 0.1102367639541626
Step 262, loss: 0.09160476177930832
Step 263, loss: 0.08556541055440903
Step 264, loss: 0.0972244068980217
Step 265, loss: 0.09133235365152359
Step 266, loss: 0.09668958932161331
Step 267, loss: 0.09747679531574249
Step 268, loss: 0.09259546548128128
Step 269, loss: 0.10741567611694336
Step 270, loss: 0.10364018380641937
Step 271, loss: 0.09192583709955215
Step 272, loss: 0.0866183191537857
Step 273, loss: 0.0995253473520279
Step 274, loss: 0.08897469192743301
Step 275, loss: 0.10126527398824692
Step 276, loss: 0.11130071431398392
Step 277, loss: 0.07839206606149673
Step 278, loss: 0.08253416419029236
Step 279, loss: 0.10969360172748566
Step 280, loss: 0.07669182121753693
Step 309, loss: 0.09676898270845413
Step 310, loss: 0.09536278247833252
Step 311, loss: 0.08626880496740341
Step 312, loss: 0.09074317663908005
Step 313, loss: 0.0872684195637703
Step 314, loss: 0.09509485214948654
Step 315, loss: 0.08823093771934509
Step 316, loss: 0.09192297607660294
Step 317, loss: 0.09358572214841843
Step 318, loss: 0.09756763279438019
Step 319, loss: 0.0958862453699112
Step 320, loss: 0.09939354658126831
Step 321, loss: 0.09518513083457947
Step 322, loss: 0.09240536391735077
Step 323, loss: 0.10315807908773422
Step 324, loss: 0.09763992577791214
Step 325, loss: 0.08891928195953369
Step 326, loss: 0.09153340756893158
Step 327, loss: 0.1023201271891594
Step 328, loss: 0.08948364108800888
Step 329, loss: 0.08565863221883774
Step 330, loss: 0.09451939165592194
Step 331, loss: 0.09675204008817673
Step 332, loss: 0.10256344825029373
Step 333, loss: 0.10136643797159195
Step 334, loss: 0.0872085765004158
Step 335, loss: 0.10480964928865433
Step 336, loss: 0.09052358567714691
Step 281, loss: 0.08920909464359283
Step 282, loss: 0.0840398371219635
Step 283, loss: 0.10449270159006119
Step 284, loss: 0.07988334447145462
Step 285, loss: 0.0852045938372612
Step 286, loss: 0.09351927787065506
Step 287, loss: 0.10052774846553802
Step 288, loss: 0.09816160053014755
Step 289, loss: 0.08929572254419327
Step 290, loss: 0.08400838822126389
Step 291, loss: 0.09651218354701996
Step 292, loss: 0.11312537640333176
Step 293, loss: 0.09710629284381866
Step 294, loss: 0.10586521774530411
Step 295, loss: 0.08972751349210739
Step 296, loss: 0.09800779819488525
Step 297, loss: 0.09882067888975143
Step 298, loss: 0.09476090967655182
Step 299, loss: 0.08048875629901886
Step 300, loss: 0.0883999913930893
Step 301, loss: 0.10911105573177338
Step 302, loss: 0.11065614968538284
Step 303, loss: 0.08949830383062363
Step 304, loss: 0.08194184303283691
Step 305, loss: 0.07901506870985031
Step 306, loss: 0.09255432337522507
Step 307, loss: 0.10475584864616394
Step 308, loss: 0.09881763905286789
Step 337, loss: 0.10486156493425369
Step 338, loss: 0.09143216907978058
Step 309, loss: 0.09676898270845413
Step 310, loss: 0.09536278247833252
Step 311, loss: 0.08626880496740341
Step 312, loss: 0.09074317663908005
Step 313, loss: 0.0872684195637703
Step 314, loss: 0.09509485214948654
Step 315, loss: 0.08823093771934509
Step 316, loss: 0.09192297607660294
Step 317, loss: 0.09358572214841843
Step 318, loss: 0.09756763279438019
Step 319, loss: 0.0958862453699112
Step 320, loss: 0.09939354658126831
Step 321, loss: 0.09518513083457947
Step 322, loss: 0.09240536391735077
Step 323, loss: 0.10315807908773422
Step 324, loss: 0.09763992577791214
Step 325, loss: 0.08891928195953369
Step 326, loss: 0.09153340756893158
Step 327, loss: 0.1023201271891594
Step 328, loss: 0.08948364108800888
Step 329, loss: 0.08565863221883774
Step 330, loss: 0.09451939165592194
Step 331, loss: 0.09675204008817673
Step 332, loss: 0.10256344825029373
Step 333, loss: 0.10136643797159195
Step 334, loss: 0.0872085765004158
Step 335, loss: 0.10480964928865433
Step 336, loss: 0.09052358567714691
Step 339, loss: 0.07445961236953735
Step 340, loss: 0.0805797353386879
Step 341, loss: 0.09287401288747787
Step 342, loss: 0.09077334403991699
Step 343, loss: 0.09919575601816177
Step 344, loss: 0.08214805275201797
Step 345, loss: 0.10290668159723282
Step 346, loss: 0.09631843119859695
Step 347, loss: 0.08739935606718063
Step 348, loss: 0.10452226549386978
Step 349, loss: 0.08685916662216187
Step 350, loss: 0.089399553835392
Step 351, loss: 0.09517337381839752
Step 352, loss: 0.0884610190987587
Step 353, loss: 0.08564262837171555
Step 354, loss: 0.0826452225446701
Step 355, loss: 0.1019214615225792
Step 356, loss: 0.09059467166662216
Step 357, loss: 0.08375606685876846
Step 358, loss: 0.0884118601679802
Step 359, loss: 0.08985290676355362
Step 360, loss: 0.0931832417845726
Step 361, loss: 0.10271522402763367
Step 362, loss: 0.08568371087312698
Step 363, loss: 0.08449043333530426
Step 364, loss: 0.1045188307762146
Step 365, loss: 0.10840359330177307
Step 366, loss: 0.09198035299777985
Step 337, loss: 0.10486156493425369
Step 338, loss: 0.09143216907978058
Step 367, loss: 0.10256067663431168
Step 368, loss: 0.08583091199398041
Step 369, loss: 0.09691421687602997
Step 370, loss: 0.09554970264434814
Step 371, loss: 0.0964154377579689
Step 372, loss: 0.09535901993513107
Step 373, loss: 0.09217069298028946
Step 374, loss: 0.10800403356552124
Step 375, loss: 0.0908806174993515
Step 376, loss: 0.09458478540182114
Step 377, loss: 0.08760320395231247
Step 378, loss: 0.0933598205447197
Step 379, loss: 0.09907391667366028
Step 380, loss: 0.10368257761001587
Step 381, loss: 0.09667558968067169
Step 382, loss: 0.09884196519851685
Step 383, loss: 0.08062954246997833
Step 384, loss: 0.10893204063177109
Step 385, loss: 0.08792293071746826
Step 386, loss: 0.09512859582901001
Step 387, loss: 0.09335962682962418
Step 388, loss: 0.0782364085316658
Step 389, loss: 0.09548364579677582
Step 390, loss: 0.09789202362298965
Step 391, loss: 0.10904669016599655
Step 392, loss: 0.09689964354038239
Step 393, loss: 0.10324911028146744
Step 394, loss: 0.09940492361783981
Step 225, loss: 0.0915277972817421
Step 226, loss: 0.09734950959682465
Step 227, loss: 0.09756748378276825
Step 228, loss: 0.0834878459572792
Step 229, loss: 0.10312364250421524
Step 230, loss: 0.09948989003896713
Step 231, loss: 0.08834394067525864
Step 232, loss: 0.0942598208785057
Step 233, loss: 0.08965849876403809
Step 234, loss: 0.08935556560754776
Step 235, loss: 0.089423768222332
Step 236, loss: 0.10344216972589493
Step 237, loss: 0.07752181589603424
Step 238, loss: 0.09235274791717529
Step 239, loss: 0.09368637204170227
Step 240, loss: 0.09884591400623322
Step 241, loss: 0.08072476089000702
Step 242, loss: 0.11127684265375137
Step 243, loss: 0.10105646401643753
Step 244, loss: 0.10253992676734924
Step 245, loss: 0.11403459310531616
Step 246, loss: 0.10600597411394119
Step 247, loss: 0.09744757413864136
Step 248, loss: 0.10752671957015991
Step 249, loss: 0.09257692843675613
Step 250, loss: 0.10353343188762665
Step 251, loss: 0.09556671977043152
Step 252, loss: 0.10835393518209457
Step 395, loss: 0.08620208501815796
Step 396, loss: 0.0948353037238121
Step 397, loss: 0.09858851134777069
Step 398, loss: 0.08667661994695663
Step 399, loss: 0.07714290916919708
Step 400, loss: 0.09605338424444199
Step 401, loss: 0.08259181678295135
Step 402, loss: 0.10587847977876663
Step 403, loss: 0.10170406103134155
Step 404, loss: 0.09050613641738892
Step 405, loss: 0.0847139060497284
Step 406, loss: 0.08444532006978989
Step 407, loss: 0.102294921875
Step 408, loss: 0.09200237691402435
Step 409, loss: 0.1071396917104721
Step 410, loss: 0.08932185173034668
Step 411, loss: 0.09256329387426376
Step 412, loss: 0.08985482901334763
Step 413, loss: 0.09989946335554123
Step 414, loss: 0.09001356363296509
Step 415, loss: 0.08021972328424454
Step 416, loss: 0.09103049337863922
Step 417, loss: 0.0843631774187088
Step 418, loss: 0.10525093972682953
Step 419, loss: 0.09829505532979965
Step 420, loss: 0.1055281013250351
Step 421, loss: 0.0941363275051117
Step 422, loss: 0.10333140194416046
Step 253, loss: 0.099678635597229
Step 254, loss: 0.08157074451446533
Step 255, loss: 0.092997707426548
Step 256, loss: 0.10462192445993423
Step 257, loss: 0.08885577321052551
Step 258, loss: 0.09344454854726791
Step 259, loss: 0.08895348757505417
Step 260, loss: 0.08721040934324265
Step 261, loss: 0.1102367639541626
Step 262, loss: 0.09160476177930832
Step 263, loss: 0.08556541055440903
Step 264, loss: 0.0972244068980217
Step 265, loss: 0.09133235365152359
Step 266, loss: 0.09668958932161331
Step 267, loss: 0.09747679531574249
Step 268, loss: 0.09259546548128128
Step 269, loss: 0.10741567611694336
Step 270, loss: 0.10364018380641937
Step 271, loss: 0.09192583709955215
Step 272, loss: 0.0866183191537857
Step 273, loss: 0.0995253473520279
Step 274, loss: 0.08897469192743301
Step 275, loss: 0.10126527398824692
Step 276, loss: 0.11130071431398392
Step 277, loss: 0.07839206606149673
Step 278, loss: 0.08253416419029236
Step 279, loss: 0.10969360172748566
Step 280, loss: 0.07669182121753693
Step 423, loss: 0.08642852306365967
Step 424, loss: 0.09760038554668427
Step 425, loss: 0.08078045397996902
Step 426, loss: 0.08566514402627945
Step 427, loss: 0.09726865589618683
Step 428, loss: 0.08297593146562576
Step 429, loss: 0.1069452315568924
Step 430, loss: 0.08628760278224945
Step 431, loss: 0.08262874186038971
Step 432, loss: 0.08956742286682129
Step 433, loss: 0.08533556014299393
Step 434, loss: 0.0934138149023056
Step 435, loss: 0.10118114203214645
Step 436, loss: 0.08869095146656036
Step 437, loss: 0.08668148517608643
Step 438, loss: 0.09466224908828735
Step 439, loss: 0.0809057280421257
Step 440, loss: 0.08602495491504669
Step 441, loss: 0.10809214413166046
Step 442, loss: 0.08890371024608612
Step 443, loss: 0.10107913613319397
Step 444, loss: 0.09205890446901321
Step 445, loss: 0.08381463587284088
Step 446, loss: 0.10211075842380524
Step 447, loss: 0.09884621948003769
Step 448, loss: 0.0960962101817131
Step 449, loss: 0.08752726763486862
Step 450, loss: 0.09262149780988693
Step 281, loss: 0.08920909464359283
Step 282, loss: 0.0840398371219635
Step 283, loss: 0.10449270159006119
Step 284, loss: 0.07988334447145462
Step 285, loss: 0.0852045938372612
Step 286, loss: 0.09351927787065506
Step 287, loss: 0.10052774846553802
Step 288, loss: 0.09816160053014755
Step 289, loss: 0.08929572254419327
Step 290, loss: 0.08400838822126389
Step 291, loss: 0.09651218354701996
Step 292, loss: 0.11312537640333176
Step 293, loss: 0.09710629284381866
Step 294, loss: 0.10586521774530411
Step 295, loss: 0.08972751349210739
Step 296, loss: 0.09800779819488525
Step 297, loss: 0.09882067888975143
Step 298, loss: 0.09476090967655182
Step 299, loss: 0.08048875629901886
Step 300, loss: 0.0883999913930893
Step 301, loss: 0.10911105573177338
Step 302, loss: 0.11065614968538284
Step 303, loss: 0.08949830383062363
Step 304, loss: 0.08194184303283691
Step 305, loss: 0.07901506870985031
Step 306, loss: 0.09255432337522507
Step 307, loss: 0.10475584864616394
Step 308, loss: 0.09881763905286789
Step 451, loss: 0.09735321998596191
Step 452, loss: 0.08930101245641708
Step 309, loss: 0.09676898270845413
Step 310, loss: 0.09536278247833252
Step 311, loss: 0.08626880496740341
Step 312, loss: 0.09074317663908005
Step 313, loss: 0.0872684195637703
Step 314, loss: 0.09509485214948654
Step 315, loss: 0.08823093771934509
Step 316, loss: 0.09192297607660294
Step 317, loss: 0.09358572214841843
Step 318, loss: 0.09756763279438019
Step 319, loss: 0.0958862453699112
Step 320, loss: 0.09939354658126831
Step 321, loss: 0.09518513083457947
Step 322, loss: 0.09240536391735077
Step 323, loss: 0.10315807908773422
Step 324, loss: 0.09763992577791214
Step 325, loss: 0.08891928195953369
Step 326, loss: 0.09153340756893158
Step 327, loss: 0.1023201271891594
Step 328, loss: 0.08948364108800888
Step 329, loss: 0.08565863221883774
Step 330, loss: 0.09451939165592194
Step 331, loss: 0.09675204008817673
Step 332, loss: 0.10256344825029373
Step 333, loss: 0.10136643797159195
Step 334, loss: 0.0872085765004158
Step 335, loss: 0.10480964928865433
Step 336, loss: 0.09052358567714691
Step 453, loss: 0.08560588210821152
Step 337, loss: 0.10486156493425369
Step 338, loss: 0.09143216907978058
Step 225, loss: 0.0915277972817421
Step 226, loss: 0.09734950959682465
Step 227, loss: 0.09756748378276825
Step 228, loss: 0.0834878459572792
Step 229, loss: 0.10312364250421524
Step 230, loss: 0.09948989003896713
Step 231, loss: 0.08834394067525864
Step 232, loss: 0.0942598208785057
Step 233, loss: 0.08965849876403809
Step 234, loss: 0.08935556560754776
Step 235, loss: 0.089423768222332
Step 236, loss: 0.10344216972589493
Step 237, loss: 0.07752181589603424
Step 238, loss: 0.09235274791717529
Step 239, loss: 0.09368637204170227
Step 240, loss: 0.09884591400623322
Step 241, loss: 0.08072476089000702
Step 242, loss: 0.11127684265375137
Step 243, loss: 0.10105646401643753
Step 244, loss: 0.10253992676734924
Step 245, loss: 0.11403459310531616
Step 246, loss: 0.10600597411394119
Step 247, loss: 0.09744757413864136
Step 248, loss: 0.10752671957015991
Step 249, loss: 0.09257692843675613
Step 250, loss: 0.10353343188762665
Step 251, loss: 0.09556671977043152
Step 252, loss: 0.10835393518209457
Step 339, loss: 0.07445961236953735
Step 340, loss: 0.0805797353386879
Step 341, loss: 0.09287401288747787
Step 342, loss: 0.09077334403991699
Step 343, loss: 0.09919575601816177
Step 344, loss: 0.08214805275201797
Step 345, loss: 0.10290668159723282
Step 346, loss: 0.09631843119859695
Step 347, loss: 0.08739935606718063
Step 348, loss: 0.10452226549386978
Step 349, loss: 0.08685916662216187
Step 350, loss: 0.089399553835392
Step 351, loss: 0.09517337381839752
Step 352, loss: 0.0884610190987587
Step 353, loss: 0.08564262837171555
Step 354, loss: 0.0826452225446701
Step 355, loss: 0.1019214615225792
Step 356, loss: 0.09059467166662216
Step 357, loss: 0.08375606685876846
Step 358, loss: 0.0884118601679802
Step 359, loss: 0.08985290676355362
Step 360, loss: 0.0931832417845726
Step 361, loss: 0.10271522402763367
Step 362, loss: 0.08568371087312698
Step 363, loss: 0.08449043333530426
Step 364, loss: 0.1045188307762146
Step 365, loss: 0.10840359330177307
Step 366, loss: 0.09198035299777985
Step 253, loss: 0.099678635597229
Step 254, loss: 0.08157074451446533
Step 255, loss: 0.092997707426548
Step 256, loss: 0.10462192445993423
Step 257, loss: 0.08885577321052551
Step 258, loss: 0.09344454854726791
Step 259, loss: 0.08895348757505417
Step 260, loss: 0.08721040934324265
Step 261, loss: 0.1102367639541626
Step 262, loss: 0.09160476177930832
Step 263, loss: 0.08556541055440903
Step 264, loss: 0.0972244068980217
Step 265, loss: 0.09133235365152359
Step 266, loss: 0.09668958932161331
Step 267, loss: 0.09747679531574249
Step 268, loss: 0.09259546548128128
Step 269, loss: 0.10741567611694336
Step 270, loss: 0.10364018380641937
Step 271, loss: 0.09192583709955215
Step 272, loss: 0.0866183191537857
Step 273, loss: 0.0995253473520279
Step 274, loss: 0.08897469192743301
Step 275, loss: 0.10126527398824692
Step 276, loss: 0.11130071431398392
Step 277, loss: 0.07839206606149673
Step 278, loss: 0.08253416419029236
Step 279, loss: 0.10969360172748566
Step 280, loss: 0.07669182121753693
Step 367, loss: 0.10256067663431168
Step 368, loss: 0.08583091199398041
Step 369, loss: 0.09691421687602997
Step 370, loss: 0.09554970264434814
Step 371, loss: 0.0964154377579689
Step 372, loss: 0.09535901993513107
Step 373, loss: 0.09217069298028946
Step 374, loss: 0.10800403356552124
Step 375, loss: 0.0908806174993515
Step 376, loss: 0.09458478540182114
Step 377, loss: 0.08760320395231247
Step 378, loss: 0.0933598205447197
Step 379, loss: 0.09907391667366028
Step 380, loss: 0.10368257761001587
Step 381, loss: 0.09667558968067169
Step 382, loss: 0.09884196519851685
Step 383, loss: 0.08062954246997833
Step 384, loss: 0.10893204063177109
Step 385, loss: 0.08792293071746826
Step 386, loss: 0.09512859582901001
Step 387, loss: 0.09335962682962418
Step 388, loss: 0.0782364085316658
Step 389, loss: 0.09548364579677582
Step 390, loss: 0.09789202362298965
Step 391, loss: 0.10904669016599655
Step 392, loss: 0.09689964354038239
Step 393, loss: 0.10324911028146744
Step 394, loss: 0.09940492361783981
Step 281, loss: 0.08920909464359283
Step 282, loss: 0.0840398371219635
Step 283, loss: 0.10449270159006119
Step 284, loss: 0.07988334447145462
Step 285, loss: 0.0852045938372612
Step 286, loss: 0.09351927787065506
Step 287, loss: 0.10052774846553802
Step 288, loss: 0.09816160053014755
Step 289, loss: 0.08929572254419327
Step 290, loss: 0.08400838822126389
Step 291, loss: 0.09651218354701996
Step 292, loss: 0.11312537640333176
Step 293, loss: 0.09710629284381866
Step 294, loss: 0.10586521774530411
Step 295, loss: 0.08972751349210739
Step 296, loss: 0.09800779819488525
Step 297, loss: 0.09882067888975143
Step 298, loss: 0.09476090967655182
Step 299, loss: 0.08048875629901886
Step 300, loss: 0.0883999913930893
Step 301, loss: 0.10911105573177338
Step 302, loss: 0.11065614968538284
Step 303, loss: 0.08949830383062363
Step 304, loss: 0.08194184303283691
Step 305, loss: 0.07901506870985031
Step 306, loss: 0.09255432337522507
Step 307, loss: 0.10475584864616394
Step 308, loss: 0.09881763905286789
Step 395, loss: 0.08620208501815796
Step 396, loss: 0.0948353037238121
Step 397, loss: 0.09858851134777069
Step 398, loss: 0.08667661994695663
Step 399, loss: 0.07714290916919708
Step 400, loss: 0.09605338424444199
Step 401, loss: 0.08259181678295135
Step 402, loss: 0.10587847977876663
Step 403, loss: 0.10170406103134155
Step 404, loss: 0.09050613641738892
Step 405, loss: 0.0847139060497284
Step 406, loss: 0.08444532006978989
Step 407, loss: 0.102294921875
Step 408, loss: 0.09200237691402435
Step 409, loss: 0.1071396917104721
Step 410, loss: 0.08932185173034668
Step 411, loss: 0.09256329387426376
Step 412, loss: 0.08985482901334763
Step 413, loss: 0.09989946335554123
Step 414, loss: 0.09001356363296509
Step 415, loss: 0.08021972328424454
Step 416, loss: 0.09103049337863922
Step 417, loss: 0.0843631774187088
Step 418, loss: 0.10525093972682953
Step 419, loss: 0.09829505532979965
Step 420, loss: 0.1055281013250351
Step 421, loss: 0.0941363275051117
Step 422, loss: 0.10333140194416046
Step 309, loss: 0.09676898270845413
Step 310, loss: 0.09536278247833252
Step 311, loss: 0.08626880496740341
Step 312, loss: 0.09074317663908005
Step 313, loss: 0.0872684195637703
Step 314, loss: 0.09509485214948654
Step 315, loss: 0.08823093771934509
Step 316, loss: 0.09192297607660294
Step 317, loss: 0.09358572214841843
Step 318, loss: 0.09756763279438019
Step 319, loss: 0.0958862453699112
Step 320, loss: 0.09939354658126831
Step 321, loss: 0.09518513083457947
Step 322, loss: 0.09240536391735077
Step 323, loss: 0.10315807908773422
Step 324, loss: 0.09763992577791214
Step 325, loss: 0.08891928195953369
Step 326, loss: 0.09153340756893158
Step 327, loss: 0.1023201271891594
Step 328, loss: 0.08948364108800888
Step 329, loss: 0.08565863221883774
Step 330, loss: 0.09451939165592194
Step 331, loss: 0.09675204008817673
Step 332, loss: 0.10256344825029373
Step 333, loss: 0.10136643797159195
Step 334, loss: 0.0872085765004158
Step 335, loss: 0.10480964928865433
Step 336, loss: 0.09052358567714691
Step 423, loss: 0.08642852306365967
Step 424, loss: 0.09760038554668427
Step 425, loss: 0.08078045397996902
Step 426, loss: 0.08566514402627945
Step 427, loss: 0.09726865589618683
Step 428, loss: 0.08297593146562576
Step 429, loss: 0.1069452315568924
Step 430, loss: 0.08628760278224945
Step 431, loss: 0.08262874186038971
Step 432, loss: 0.08956742286682129
Step 433, loss: 0.08533556014299393
Step 434, loss: 0.0934138149023056
Step 435, loss: 0.10118114203214645
Step 436, loss: 0.08869095146656036
Step 437, loss: 0.08668148517608643
Step 438, loss: 0.09466224908828735
Step 439, loss: 0.0809057280421257
Step 440, loss: 0.08602495491504669
Step 441, loss: 0.10809214413166046
Step 442, loss: 0.08890371024608612
Step 443, loss: 0.10107913613319397
Step 444, loss: 0.09205890446901321
Step 445, loss: 0.08381463587284088
Step 446, loss: 0.10211075842380524
Step 447, loss: 0.09884621948003769
Step 448, loss: 0.0960962101817131
Step 449, loss: 0.08752726763486862
Step 450, loss: 0.09262149780988693
Step 337, loss: 0.10486156493425369
Step 338, loss: 0.09143216907978058
Step 451, loss: 0.09735321998596191
Step 452, loss: 0.08930101245641708
Step 339, loss: 0.07445961236953735
Step 340, loss: 0.0805797353386879
Step 341, loss: 0.09287401288747787
Step 342, loss: 0.09077334403991699
Step 343, loss: 0.09919575601816177
Step 344, loss: 0.08214805275201797
Step 345, loss: 0.10290668159723282
Step 346, loss: 0.09631843119859695
Step 347, loss: 0.08739935606718063
Step 348, loss: 0.10452226549386978
Step 349, loss: 0.08685916662216187
Step 350, loss: 0.089399553835392
Step 351, loss: 0.09517337381839752
Step 352, loss: 0.0884610190987587
Step 353, loss: 0.08564262837171555
Step 354, loss: 0.0826452225446701
Step 355, loss: 0.1019214615225792
Step 356, loss: 0.09059467166662216
Step 357, loss: 0.08375606685876846
Step 358, loss: 0.0884118601679802
Step 359, loss: 0.08985290676355362
Step 360, loss: 0.0931832417845726
Step 361, loss: 0.10271522402763367
Step 362, loss: 0.08568371087312698
Step 363, loss: 0.08449043333530426
Step 364, loss: 0.1045188307762146
Step 365, loss: 0.10840359330177307
Step 366, loss: 0.09198035299777985
Step 339, loss: 0.07445961236953735
Step 340, loss: 0.0805797353386879
Step 341, loss: 0.09287401288747787
Step 342, loss: 0.09077334403991699
Step 343, loss: 0.09919575601816177
Step 344, loss: 0.08214805275201797
Step 345, loss: 0.10290668159723282
Step 346, loss: 0.09631843119859695
Step 347, loss: 0.08739935606718063
Step 348, loss: 0.10452226549386978
Step 349, loss: 0.08685916662216187
Step 350, loss: 0.089399553835392
Step 351, loss: 0.09517337381839752
Step 352, loss: 0.0884610190987587
Step 353, loss: 0.08564262837171555
Step 354, loss: 0.0826452225446701
Step 355, loss: 0.1019214615225792
Step 356, loss: 0.09059467166662216
Step 357, loss: 0.08375606685876846
Step 358, loss: 0.0884118601679802
Step 359, loss: 0.08985290676355362
Step 360, loss: 0.0931832417845726
Step 361, loss: 0.10271522402763367
Step 362, loss: 0.08568371087312698
Step 363, loss: 0.08449043333530426
Step 364, loss: 0.1045188307762146
Step 365, loss: 0.10840359330177307
Step 366, loss: 0.09198035299777985
Step 367, loss: 0.10256067663431168
Step 368, loss: 0.08583091199398041
Step 369, loss: 0.09691421687602997
Step 370, loss: 0.09554970264434814
Step 371, loss: 0.0964154377579689
Step 372, loss: 0.09535901993513107
Step 373, loss: 0.09217069298028946
Step 374, loss: 0.10800403356552124
Step 375, loss: 0.0908806174993515
Step 376, loss: 0.09458478540182114
Step 377, loss: 0.08760320395231247
Step 378, loss: 0.0933598205447197
Step 379, loss: 0.09907391667366028
Step 380, loss: 0.10368257761001587
Step 381, loss: 0.09667558968067169
Step 382, loss: 0.09884196519851685
Step 383, loss: 0.08062954246997833
Step 384, loss: 0.10893204063177109
Step 385, loss: 0.08792293071746826
Step 386, loss: 0.09512859582901001
Step 387, loss: 0.09335962682962418
Step 388, loss: 0.0782364085316658
Step 389, loss: 0.09548364579677582
Step 390, loss: 0.09789202362298965
Step 391, loss: 0.10904669016599655
Step 392, loss: 0.09689964354038239
Step 393, loss: 0.10324911028146744
Step 394, loss: 0.09940492361783981
Step 367, loss: 0.10256067663431168
Step 368, loss: 0.08583091199398041
Step 369, loss: 0.09691421687602997
Step 370, loss: 0.09554970264434814
Step 371, loss: 0.0964154377579689
Step 372, loss: 0.09535901993513107
Step 373, loss: 0.09217069298028946
Step 374, loss: 0.10800403356552124
Step 375, loss: 0.0908806174993515
Step 376, loss: 0.09458478540182114
Step 377, loss: 0.08760320395231247
Step 378, loss: 0.0933598205447197
Step 379, loss: 0.09907391667366028
Step 380, loss: 0.10368257761001587
Step 381, loss: 0.09667558968067169
Step 382, loss: 0.09884196519851685
Step 383, loss: 0.08062954246997833
Step 384, loss: 0.10893204063177109
Step 385, loss: 0.08792293071746826
Step 386, loss: 0.09512859582901001
Step 387, loss: 0.09335962682962418
Step 388, loss: 0.0782364085316658
Step 389, loss: 0.09548364579677582
Step 390, loss: 0.09789202362298965
Step 391, loss: 0.10904669016599655
Step 392, loss: 0.09689964354038239
Step 393, loss: 0.10324911028146744
Step 394, loss: 0.09940492361783981
Step 395, loss: 0.08620208501815796
Step 396, loss: 0.0948353037238121
Step 397, loss: 0.09858851134777069
Step 398, loss: 0.08667661994695663
Step 399, loss: 0.07714290916919708
Step 400, loss: 0.09605338424444199
Step 401, loss: 0.08259181678295135
Step 402, loss: 0.10587847977876663
Step 403, loss: 0.10170406103134155
Step 404, loss: 0.09050613641738892
Step 405, loss: 0.0847139060497284
Step 406, loss: 0.08444532006978989
Step 407, loss: 0.102294921875
Step 408, loss: 0.09200237691402435
Step 409, loss: 0.1071396917104721
Step 410, loss: 0.08932185173034668
Step 411, loss: 0.09256329387426376
Step 412, loss: 0.08985482901334763
Step 413, loss: 0.09989946335554123
Step 414, loss: 0.09001356363296509
Step 415, loss: 0.08021972328424454
Step 416, loss: 0.09103049337863922
Step 417, loss: 0.0843631774187088
Step 418, loss: 0.10525093972682953
Step 419, loss: 0.09829505532979965
Step 420, loss: 0.1055281013250351
Step 421, loss: 0.0941363275051117
Step 422, loss: 0.10333140194416046
Step 395, loss: 0.08620208501815796
Step 396, loss: 0.0948353037238121
Step 397, loss: 0.09858851134777069
Step 398, loss: 0.08667661994695663
Step 399, loss: 0.07714290916919708
Step 400, loss: 0.09605338424444199
Step 401, loss: 0.08259181678295135
Step 402, loss: 0.10587847977876663
Step 403, loss: 0.10170406103134155
Step 404, loss: 0.09050613641738892
Step 405, loss: 0.0847139060497284
Step 406, loss: 0.08444532006978989
Step 407, loss: 0.102294921875
Step 408, loss: 0.09200237691402435
Step 409, loss: 0.1071396917104721
Step 410, loss: 0.08932185173034668
Step 411, loss: 0.09256329387426376
Step 412, loss: 0.08985482901334763
Step 413, loss: 0.09989946335554123
Step 414, loss: 0.09001356363296509
Step 415, loss: 0.08021972328424454
Step 416, loss: 0.09103049337863922
Step 417, loss: 0.0843631774187088
Step 418, loss: 0.10525093972682953
Step 419, loss: 0.09829505532979965
Step 420, loss: 0.1055281013250351
Step 421, loss: 0.0941363275051117
Step 422, loss: 0.10333140194416046
Step 423, loss: 0.08642852306365967
Step 424, loss: 0.09760038554668427
Step 425, loss: 0.08078045397996902
Step 426, loss: 0.08566514402627945
Step 427, loss: 0.09726865589618683
Step 428, loss: 0.08297593146562576
Step 429, loss: 0.1069452315568924
Step 430, loss: 0.08628760278224945
Step 431, loss: 0.08262874186038971
Step 432, loss: 0.08956742286682129
Step 433, loss: 0.08533556014299393
Step 434, loss: 0.0934138149023056
Step 435, loss: 0.10118114203214645
Step 436, loss: 0.08869095146656036
Step 437, loss: 0.08668148517608643
Step 438, loss: 0.09466224908828735
Step 439, loss: 0.0809057280421257
Step 440, loss: 0.08602495491504669
Step 441, loss: 0.10809214413166046
Step 442, loss: 0.08890371024608612
Step 443, loss: 0.10107913613319397
Step 444, loss: 0.09205890446901321
Step 445, loss: 0.08381463587284088
Step 446, loss: 0.10211075842380524
Step 447, loss: 0.09884621948003769
Step 448, loss: 0.0960962101817131
Step 449, loss: 0.08752726763486862
Step 450, loss: 0.09262149780988693
Step 423, loss: 0.08642852306365967
Step 424, loss: 0.09760038554668427
Step 425, loss: 0.08078045397996902
Step 426, loss: 0.08566514402627945
Step 427, loss: 0.09726865589618683
Step 428, loss: 0.08297593146562576
Step 429, loss: 0.1069452315568924
Step 430, loss: 0.08628760278224945
Step 431, loss: 0.08262874186038971
Step 432, loss: 0.08956742286682129
Step 433, loss: 0.08533556014299393
Step 434, loss: 0.0934138149023056
Step 435, loss: 0.10118114203214645
Step 436, loss: 0.08869095146656036
Step 437, loss: 0.08668148517608643
Step 438, loss: 0.09466224908828735
Step 439, loss: 0.0809057280421257
Step 440, loss: 0.08602495491504669
Step 441, loss: 0.10809214413166046
Step 442, loss: 0.08890371024608612
Step 443, loss: 0.10107913613319397
Step 444, loss: 0.09205890446901321
Step 445, loss: 0.08381463587284088
Step 446, loss: 0.10211075842380524
Step 447, loss: 0.09884621948003769
Step 448, loss: 0.0960962101817131
Step 449, loss: 0.08752726763486862
Step 450, loss: 0.09262149780988693
Step 451, loss: 0.09735321998596191
Step 452, loss: 0.08930101245641708
Step 451, loss: 0.09735321998596191
Step 452, loss: 0.08930101245641708
Step 453, loss: 0.08560588210821152
Step 453, loss: 0.08560588210821152
Step 225, loss: 0.0915277972817421
Step 226, loss: 0.09734950959682465
Step 227, loss: 0.09756748378276825
Step 228, loss: 0.0834878459572792
Step 229, loss: 0.10312364250421524
Step 230, loss: 0.09948989003896713
Step 231, loss: 0.08834394067525864
Step 232, loss: 0.0942598208785057
Step 233, loss: 0.08965849876403809
Step 234, loss: 0.08935556560754776
Step 235, loss: 0.089423768222332
Step 236, loss: 0.10344216972589493
Step 237, loss: 0.07752181589603424
Step 238, loss: 0.09235274791717529
Step 239, loss: 0.09368637204170227
Step 240, loss: 0.09884591400623322
Step 241, loss: 0.08072476089000702
Step 242, loss: 0.11127684265375137
Step 243, loss: 0.10105646401643753
Step 244, loss: 0.10253992676734924
Step 245, loss: 0.11403459310531616
Step 246, loss: 0.10600597411394119
Step 247, loss: 0.09744757413864136
Step 248, loss: 0.10752671957015991
Step 249, loss: 0.09257692843675613
Step 250, loss: 0.10353343188762665
Step 251, loss: 0.09556671977043152
Step 252, loss: 0.10835393518209457
Step 453, loss: 0.08560588210821152
Step 253, loss: 0.099678635597229
Step 254, loss: 0.08157074451446533
Step 255, loss: 0.092997707426548
Step 256, loss: 0.10462192445993423
Step 257, loss: 0.08885577321052551
Step 258, loss: 0.09344454854726791
Step 259, loss: 0.08895348757505417
Step 260, loss: 0.08721040934324265
Step 261, loss: 0.1102367639541626
Step 262, loss: 0.09160476177930832
Step 263, loss: 0.08556541055440903
Step 264, loss: 0.0972244068980217
Step 265, loss: 0.09133235365152359
Step 266, loss: 0.09668958932161331
Step 267, loss: 0.09747679531574249
Step 268, loss: 0.09259546548128128
Step 269, loss: 0.10741567611694336
Step 270, loss: 0.10364018380641937
Step 271, loss: 0.09192583709955215
Step 272, loss: 0.0866183191537857
Step 273, loss: 0.0995253473520279
Step 274, loss: 0.08897469192743301
Step 275, loss: 0.10126527398824692
Step 276, loss: 0.11130071431398392
Step 277, loss: 0.07839206606149673
Step 278, loss: 0.08253416419029236
Step 279, loss: 0.10969360172748566
Step 280, loss: 0.07669182121753693
Step 225, loss: 0.0915277972817421
Step 226, loss: 0.09734950959682465
Step 227, loss: 0.09756748378276825
Step 228, loss: 0.0834878459572792
Step 229, loss: 0.10312364250421524
Step 230, loss: 0.09948989003896713
Step 231, loss: 0.08834394067525864
Step 232, loss: 0.0942598208785057
Step 233, loss: 0.08965849876403809
Step 234, loss: 0.08935556560754776
Step 235, loss: 0.089423768222332
Step 236, loss: 0.10344216972589493
Step 237, loss: 0.07752181589603424
Step 238, loss: 0.09235274791717529
Step 239, loss: 0.09368637204170227
Step 240, loss: 0.09884591400623322
Step 241, loss: 0.08072476089000702
Step 242, loss: 0.11127684265375137
Step 243, loss: 0.10105646401643753
Step 244, loss: 0.10253992676734924
Step 245, loss: 0.11403459310531616
Step 246, loss: 0.10600597411394119
Step 247, loss: 0.09744757413864136
Step 248, loss: 0.10752671957015991
Step 249, loss: 0.09257692843675613
Step 250, loss: 0.10353343188762665
Step 251, loss: 0.09556671977043152
Step 252, loss: 0.10835393518209457
Step 281, loss: 0.08920909464359283
Step 282, loss: 0.0840398371219635
Step 283, loss: 0.10449270159006119
Step 284, loss: 0.07988334447145462
Step 285, loss: 0.0852045938372612
Step 286, loss: 0.09351927787065506
Step 287, loss: 0.10052774846553802
Step 288, loss: 0.09816160053014755
Step 289, loss: 0.08929572254419327
Step 290, loss: 0.08400838822126389
Step 291, loss: 0.09651218354701996
Step 292, loss: 0.11312537640333176
Step 293, loss: 0.09710629284381866
Step 294, loss: 0.10586521774530411
Step 295, loss: 0.08972751349210739
Step 296, loss: 0.09800779819488525
Step 297, loss: 0.09882067888975143
Step 298, loss: 0.09476090967655182
Step 299, loss: 0.08048875629901886
Step 300, loss: 0.0883999913930893
Step 301, loss: 0.10911105573177338
Step 302, loss: 0.11065614968538284
Step 303, loss: 0.08949830383062363
Step 304, loss: 0.08194184303283691
Step 305, loss: 0.07901506870985031
Step 306, loss: 0.09255432337522507
Step 307, loss: 0.10475584864616394
Step 308, loss: 0.09881763905286789
Step 253, loss: 0.099678635597229
Step 254, loss: 0.08157074451446533
Step 255, loss: 0.092997707426548
Step 256, loss: 0.10462192445993423
Step 257, loss: 0.08885577321052551
Step 258, loss: 0.09344454854726791
Step 259, loss: 0.08895348757505417
Step 260, loss: 0.08721040934324265
Step 261, loss: 0.1102367639541626
Step 262, loss: 0.09160476177930832
Step 263, loss: 0.08556541055440903
Step 264, loss: 0.0972244068980217
Step 265, loss: 0.09133235365152359
Step 266, loss: 0.09668958932161331
Step 267, loss: 0.09747679531574249
Step 268, loss: 0.09259546548128128
Step 269, loss: 0.10741567611694336
Step 270, loss: 0.10364018380641937
Step 271, loss: 0.09192583709955215
Step 272, loss: 0.0866183191537857
Step 273, loss: 0.0995253473520279
Step 274, loss: 0.08897469192743301
Step 275, loss: 0.10126527398824692
Step 276, loss: 0.11130071431398392
Step 277, loss: 0.07839206606149673
Step 278, loss: 0.08253416419029236
Step 279, loss: 0.10969360172748566
Step 280, loss: 0.07669182121753693
Step 309, loss: 0.09676898270845413
Step 310, loss: 0.09536278247833252
Step 311, loss: 0.08626880496740341
Step 312, loss: 0.09074317663908005
Step 313, loss: 0.0872684195637703
Step 314, loss: 0.09509485214948654
Step 315, loss: 0.08823093771934509
Step 316, loss: 0.09192297607660294
Step 317, loss: 0.09358572214841843
Step 318, loss: 0.09756763279438019
Step 319, loss: 0.0958862453699112
Step 320, loss: 0.09939354658126831
Step 321, loss: 0.09518513083457947
Step 322, loss: 0.09240536391735077
Step 323, loss: 0.10315807908773422
Step 324, loss: 0.09763992577791214
Step 325, loss: 0.08891928195953369
Step 326, loss: 0.09153340756893158
Step 327, loss: 0.1023201271891594
Step 328, loss: 0.08948364108800888
Step 329, loss: 0.08565863221883774
Step 330, loss: 0.09451939165592194
Step 331, loss: 0.09675204008817673
Step 332, loss: 0.10256344825029373
Step 333, loss: 0.10136643797159195
Step 334, loss: 0.0872085765004158
Step 335, loss: 0.10480964928865433
Step 336, loss: 0.09052358567714691
Step 281, loss: 0.08920909464359283
Step 282, loss: 0.0840398371219635
Step 283, loss: 0.10449270159006119
Step 284, loss: 0.07988334447145462
Step 285, loss: 0.0852045938372612
Step 286, loss: 0.09351927787065506
Step 287, loss: 0.10052774846553802
Step 288, loss: 0.09816160053014755
Step 289, loss: 0.08929572254419327
Step 290, loss: 0.08400838822126389
Step 291, loss: 0.09651218354701996
Step 292, loss: 0.11312537640333176
Step 293, loss: 0.09710629284381866
Step 294, loss: 0.10586521774530411
Step 295, loss: 0.08972751349210739
Step 296, loss: 0.09800779819488525
Step 297, loss: 0.09882067888975143
Step 298, loss: 0.09476090967655182
Step 299, loss: 0.08048875629901886
Step 300, loss: 0.0883999913930893
Step 301, loss: 0.10911105573177338
Step 302, loss: 0.11065614968538284
Step 303, loss: 0.08949830383062363
Step 304, loss: 0.08194184303283691
Step 305, loss: 0.07901506870985031
Step 306, loss: 0.09255432337522507
Step 307, loss: 0.10475584864616394
Step 308, loss: 0.09881763905286789
Step 337, loss: 0.10486156493425369
Step 338, loss: 0.09143216907978058
Step 309, loss: 0.09676898270845413
Step 310, loss: 0.09536278247833252
Step 311, loss: 0.08626880496740341
Step 312, loss: 0.09074317663908005
Step 313, loss: 0.0872684195637703
Step 314, loss: 0.09509485214948654
Step 315, loss: 0.08823093771934509
Step 316, loss: 0.09192297607660294
Step 317, loss: 0.09358572214841843
Step 318, loss: 0.09756763279438019
Step 319, loss: 0.0958862453699112
Step 320, loss: 0.09939354658126831
Step 321, loss: 0.09518513083457947
Step 322, loss: 0.09240536391735077
Step 323, loss: 0.10315807908773422
Step 324, loss: 0.09763992577791214
Step 325, loss: 0.08891928195953369
Step 326, loss: 0.09153340756893158
Step 327, loss: 0.1023201271891594
Step 328, loss: 0.08948364108800888
Step 329, loss: 0.08565863221883774
Step 330, loss: 0.09451939165592194
Step 331, loss: 0.09675204008817673
Step 332, loss: 0.10256344825029373
Step 333, loss: 0.10136643797159195
Step 334, loss: 0.0872085765004158
Step 335, loss: 0.10480964928865433
Step 336, loss: 0.09052358567714691
Step 339, loss: 0.07445961236953735
Step 340, loss: 0.0805797353386879
Step 341, loss: 0.09287401288747787
Step 342, loss: 0.09077334403991699
Step 343, loss: 0.09919575601816177
Step 344, loss: 0.08214805275201797
Step 345, loss: 0.10290668159723282
Step 346, loss: 0.09631843119859695
Step 347, loss: 0.08739935606718063
Step 348, loss: 0.10452226549386978
Step 349, loss: 0.08685916662216187
Step 350, loss: 0.089399553835392
Step 351, loss: 0.09517337381839752
Step 352, loss: 0.0884610190987587
Step 353, loss: 0.08564262837171555
Step 354, loss: 0.0826452225446701
Step 355, loss: 0.1019214615225792
Step 356, loss: 0.09059467166662216
Step 357, loss: 0.08375606685876846
Step 358, loss: 0.0884118601679802
Step 359, loss: 0.08985290676355362
Step 360, loss: 0.0931832417845726
Step 361, loss: 0.10271522402763367
Step 362, loss: 0.08568371087312698
Step 363, loss: 0.08449043333530426
Step 364, loss: 0.1045188307762146
Step 365, loss: 0.10840359330177307
Step 366, loss: 0.09198035299777985
Step 337, loss: 0.10486156493425369
Step 338, loss: 0.09143216907978058
Step 367, loss: 0.10256067663431168
Step 368, loss: 0.08583091199398041
Step 369, loss: 0.09691421687602997
Step 370, loss: 0.09554970264434814
Step 371, loss: 0.0964154377579689
Step 372, loss: 0.09535901993513107
Step 373, loss: 0.09217069298028946
Step 374, loss: 0.10800403356552124
Step 375, loss: 0.0908806174993515
Step 376, loss: 0.09458478540182114
Step 377, loss: 0.08760320395231247
Step 378, loss: 0.0933598205447197
Step 379, loss: 0.09907391667366028
Step 380, loss: 0.10368257761001587
Step 381, loss: 0.09667558968067169
Step 382, loss: 0.09884196519851685
Step 383, loss: 0.08062954246997833
Step 384, loss: 0.10893204063177109
Step 385, loss: 0.08792293071746826
Step 386, loss: 0.09512859582901001
Step 387, loss: 0.09335962682962418
Step 388, loss: 0.0782364085316658
Step 389, loss: 0.09548364579677582
Step 390, loss: 0.09789202362298965
Step 391, loss: 0.10904669016599655
Step 392, loss: 0.09689964354038239
Step 393, loss: 0.10324911028146744
Step 394, loss: 0.09940492361783981
Step 339, loss: 0.07445961236953735
Step 340, loss: 0.0805797353386879
Step 341, loss: 0.09287401288747787
Step 342, loss: 0.09077334403991699
Step 343, loss: 0.09919575601816177
Step 344, loss: 0.08214805275201797
Step 345, loss: 0.10290668159723282
Step 346, loss: 0.09631843119859695
Step 347, loss: 0.08739935606718063
Step 348, loss: 0.10452226549386978
Step 349, loss: 0.08685916662216187
Step 350, loss: 0.089399553835392
Step 351, loss: 0.09517337381839752
Step 352, loss: 0.0884610190987587
Step 353, loss: 0.08564262837171555
Step 354, loss: 0.0826452225446701
Step 355, loss: 0.1019214615225792
Step 356, loss: 0.09059467166662216
Step 357, loss: 0.08375606685876846
Step 358, loss: 0.0884118601679802
Step 359, loss: 0.08985290676355362
Step 360, loss: 0.0931832417845726
Step 361, loss: 0.10271522402763367
Step 362, loss: 0.08568371087312698
Step 363, loss: 0.08449043333530426
Step 364, loss: 0.1045188307762146
Step 365, loss: 0.10840359330177307
Step 366, loss: 0.09198035299777985
Step 395, loss: 0.08620208501815796
Step 396, loss: 0.0948353037238121
Step 397, loss: 0.09858851134777069
Step 398, loss: 0.08667661994695663
Step 399, loss: 0.07714290916919708
Step 400, loss: 0.09605338424444199
Step 401, loss: 0.08259181678295135
Step 402, loss: 0.10587847977876663
Step 403, loss: 0.10170406103134155
Step 404, loss: 0.09050613641738892
Step 405, loss: 0.0847139060497284
Step 406, loss: 0.08444532006978989
Step 407, loss: 0.102294921875
Step 408, loss: 0.09200237691402435
Step 409, loss: 0.1071396917104721
Step 410, loss: 0.08932185173034668
Step 411, loss: 0.09256329387426376
Step 412, loss: 0.08985482901334763
Step 413, loss: 0.09989946335554123
Step 414, loss: 0.09001356363296509
Step 415, loss: 0.08021972328424454
Step 416, loss: 0.09103049337863922
Step 417, loss: 0.0843631774187088
Step 418, loss: 0.10525093972682953
Step 419, loss: 0.09829505532979965
Step 420, loss: 0.1055281013250351
Step 421, loss: 0.0941363275051117
Step 422, loss: 0.10333140194416046
Step 367, loss: 0.10256067663431168
Step 368, loss: 0.08583091199398041
Step 369, loss: 0.09691421687602997
Step 370, loss: 0.09554970264434814
Step 371, loss: 0.0964154377579689
Step 372, loss: 0.09535901993513107
Step 373, loss: 0.09217069298028946
Step 374, loss: 0.10800403356552124
Step 375, loss: 0.0908806174993515
Step 376, loss: 0.09458478540182114
Step 377, loss: 0.08760320395231247
Step 378, loss: 0.0933598205447197
Step 379, loss: 0.09907391667366028
Step 380, loss: 0.10368257761001587
Step 381, loss: 0.09667558968067169
Step 382, loss: 0.09884196519851685
Step 383, loss: 0.08062954246997833
Step 384, loss: 0.10893204063177109
Step 385, loss: 0.08792293071746826
Step 386, loss: 0.09512859582901001
Step 387, loss: 0.09335962682962418
Step 388, loss: 0.0782364085316658
Step 389, loss: 0.09548364579677582
Step 390, loss: 0.09789202362298965
Step 391, loss: 0.10904669016599655
Step 392, loss: 0.09689964354038239
Step 393, loss: 0.10324911028146744
Step 394, loss: 0.09940492361783981
Step 423, loss: 0.08642852306365967
Step 424, loss: 0.09760038554668427
Step 425, loss: 0.08078045397996902
Step 426, loss: 0.08566514402627945
Step 427, loss: 0.09726865589618683
Step 428, loss: 0.08297593146562576
Step 429, loss: 0.1069452315568924
Step 430, loss: 0.08628760278224945
Step 431, loss: 0.08262874186038971
Step 432, loss: 0.08956742286682129
Step 433, loss: 0.08533556014299393
Step 434, loss: 0.0934138149023056
Step 435, loss: 0.10118114203214645
Step 436, loss: 0.08869095146656036
Step 437, loss: 0.08668148517608643
Step 438, loss: 0.09466224908828735
Step 439, loss: 0.0809057280421257
Step 440, loss: 0.08602495491504669
Step 441, loss: 0.10809214413166046
Step 442, loss: 0.08890371024608612
Step 443, loss: 0.10107913613319397
Step 444, loss: 0.09205890446901321
Step 445, loss: 0.08381463587284088
Step 446, loss: 0.10211075842380524
Step 447, loss: 0.09884621948003769
Step 448, loss: 0.0960962101817131
Step 449, loss: 0.08752726763486862
Step 450, loss: 0.09262149780988693
Step 395, loss: 0.08620208501815796
Step 396, loss: 0.0948353037238121
Step 397, loss: 0.09858851134777069
Step 398, loss: 0.08667661994695663
Step 399, loss: 0.07714290916919708
Step 400, loss: 0.09605338424444199
Step 401, loss: 0.08259181678295135
Step 402, loss: 0.10587847977876663
Step 403, loss: 0.10170406103134155
Step 404, loss: 0.09050613641738892
Step 405, loss: 0.0847139060497284
Step 406, loss: 0.08444532006978989
Step 407, loss: 0.102294921875
Step 408, loss: 0.09200237691402435
Step 409, loss: 0.1071396917104721
Step 410, loss: 0.08932185173034668
Step 411, loss: 0.09256329387426376
Step 412, loss: 0.08985482901334763
Step 413, loss: 0.09989946335554123
Step 414, loss: 0.09001356363296509
Step 415, loss: 0.08021972328424454
Step 416, loss: 0.09103049337863922
Step 417, loss: 0.0843631774187088
Step 418, loss: 0.10525093972682953
Step 419, loss: 0.09829505532979965
Step 420, loss: 0.1055281013250351
Step 421, loss: 0.0941363275051117
Step 422, loss: 0.10333140194416046
Step 451, loss: 0.09735321998596191
Step 452, loss: 0.08930101245641708
Step 423, loss: 0.08642852306365967
Step 424, loss: 0.09760038554668427
Step 425, loss: 0.08078045397996902
Step 426, loss: 0.08566514402627945
Step 427, loss: 0.09726865589618683
Step 428, loss: 0.08297593146562576
Step 429, loss: 0.1069452315568924
Step 430, loss: 0.08628760278224945
Step 431, loss: 0.08262874186038971
Step 432, loss: 0.08956742286682129
Step 433, loss: 0.08533556014299393
Step 434, loss: 0.0934138149023056
Step 435, loss: 0.10118114203214645
Step 436, loss: 0.08869095146656036
Step 437, loss: 0.08668148517608643
Step 438, loss: 0.09466224908828735
Step 439, loss: 0.0809057280421257
Step 440, loss: 0.08602495491504669
Step 441, loss: 0.10809214413166046
Step 442, loss: 0.08890371024608612
Step 443, loss: 0.10107913613319397
Step 444, loss: 0.09205890446901321
Step 445, loss: 0.08381463587284088
Step 446, loss: 0.10211075842380524
Step 447, loss: 0.09884621948003769
Step 448, loss: 0.0960962101817131
Step 449, loss: 0.08752726763486862
Step 450, loss: 0.09262149780988693
Step 453, loss: 0.08560588210821152
Step 451, loss: 0.09735321998596191
Step 452, loss: 0.08930101245641708
Step 453, loss: 0.08560588210821152
Step 454, loss: 0.08738715201616287
Step 455, loss: 0.08078156411647797
Step 456, loss: 0.1025397926568985
Step 457, loss: 0.09246592223644257
Step 458, loss: 0.09283715486526489
Step 459, loss: 0.08990254253149033
Step 460, loss: 0.09983637928962708
Step 461, loss: 0.10962655395269394
Step 462, loss: 0.08536367118358612
Step 463, loss: 0.08930333703756332
Step 464, loss: 0.09678693860769272
Step 465, loss: 0.09519515931606293
Step 466, loss: 0.11151231080293655
Step 467, loss: 0.11232274770736694
Step 468, loss: 0.08937115967273712
Step 469, loss: 0.09148123115301132
Step 470, loss: 0.0892455130815506
Step 471, loss: 0.10928546637296677
Step 472, loss: 0.0883624404668808
Step 473, loss: 0.09327864646911621
Step 474, loss: 0.08831151574850082
Step 475, loss: 0.09559176117181778
Step 476, loss: 0.10287821292877197
Step 477, loss: 0.10371410101652145
Step 478, loss: 0.09492306411266327
Step 479, loss: 0.08908714354038239
Step 480, loss: 0.09100113809108734
Step 481, loss: 0.08331077545881271
Step 482, loss: 0.10054487735033035
Step 483, loss: 0.09636945277452469
Step 484, loss: 0.09950514882802963
Step 485, loss: 0.09211280941963196
Step 486, loss: 0.09582772105932236
Step 487, loss: 0.08931293338537216
Step 488, loss: 0.08673063665628433
Step 489, loss: 0.09423231333494186
Step 490, loss: 0.09473028779029846
Step 491, loss: 0.08363447338342667
Step 492, loss: 0.0886383056640625
Step 493, loss: 0.08623740077018738
Step 494, loss: 0.09342889487743378
Step 495, loss: 0.09426616132259369
Step 496, loss: 0.08552157133817673
Step 497, loss: 0.10631301254034042
Step 498, loss: 0.08671450614929199
Step 499, loss: 0.0800226703286171
Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 159, in wait_for_base_dir_creation
Step 454, loss: 0.08738715201616287
Step 455, loss: 0.08078156411647797
Step 456, loss: 0.1025397926568985
Step 457, loss: 0.09246592223644257
Step 458, loss: 0.09283715486526489
Step 459, loss: 0.08990254253149033
Step 460, loss: 0.09983637928962708
Step 461, loss: 0.10962655395269394
Step 462, loss: 0.08536367118358612
Step 463, loss: 0.08930333703756332
Step 464, loss: 0.09678693860769272
Step 465, loss: 0.09519515931606293
Step 466, loss: 0.11151231080293655
Step 467, loss: 0.11232274770736694
Step 468, loss: 0.08937115967273712
Step 469, loss: 0.09148123115301132
Step 470, loss: 0.0892455130815506
Step 471, loss: 0.10928546637296677
Step 472, loss: 0.0883624404668808
Step 473, loss: 0.09327864646911621
Step 474, loss: 0.08831151574850082
Step 475, loss: 0.09559176117181778
Step 476, loss: 0.10287821292877197
Step 477, loss: 0.10371410101652145
Step 478, loss: 0.09492306411266327
Step 479, loss: 0.08908714354038239
Step 480, loss: 0.09100113809108734
Step 481, loss: 0.08331077545881271
Step 482, loss: 0.10054487735033035
Step 483, loss: 0.09636945277452469
Step 484, loss: 0.09950514882802963
Step 485, loss: 0.09211280941963196
Step 486, loss: 0.09582772105932236
Step 487, loss: 0.08931293338537216
Step 488, loss: 0.08673063665628433
Step 489, loss: 0.09423231333494186
Step 490, loss: 0.09473028779029846
Step 491, loss: 0.08363447338342667
Step 492, loss: 0.0886383056640625
Step 493, loss: 0.08623740077018738
Step 494, loss: 0.09342889487743378
Step 495, loss: 0.09426616132259369
Step 496, loss: 0.08552157133817673
Step 497, loss: 0.10631301254034042
Step 498, loss: 0.08671450614929199
Step 499, loss: 0.0800226703286171
Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 159, in wait_for_base_dir_creation
Step 454, loss: 0.08738715201616287
Step 455, loss: 0.08078156411647797
Step 456, loss: 0.1025397926568985
Step 457, loss: 0.09246592223644257
Step 458, loss: 0.09283715486526489
Step 459, loss: 0.08990254253149033
Step 460, loss: 0.09983637928962708
Step 461, loss: 0.10962655395269394
Step 462, loss: 0.08536367118358612
Step 463, loss: 0.08930333703756332
Step 464, loss: 0.09678693860769272
Step 465, loss: 0.09519515931606293
Step 466, loss: 0.11151231080293655
Step 467, loss: 0.11232274770736694
Step 468, loss: 0.08937115967273712
Step 469, loss: 0.09148123115301132
Step 470, loss: 0.0892455130815506
Step 471, loss: 0.10928546637296677
Step 472, loss: 0.0883624404668808
Step 473, loss: 0.09327864646911621
Step 474, loss: 0.08831151574850082
Step 475, loss: 0.09559176117181778
Step 476, loss: 0.10287821292877197
Step 477, loss: 0.10371410101652145
Step 478, loss: 0.09492306411266327
Step 479, loss: 0.08908714354038239
Step 480, loss: 0.09100113809108734
Step 481, loss: 0.08331077545881271
Step 482, loss: 0.10054487735033035
Step 483, loss: 0.09636945277452469
Step 484, loss: 0.09950514882802963
Step 485, loss: 0.09211280941963196
Step 486, loss: 0.09582772105932236
Step 487, loss: 0.08931293338537216
Step 488, loss: 0.08673063665628433
Step 489, loss: 0.09423231333494186
Step 490, loss: 0.09473028779029846
Step 491, loss: 0.08363447338342667
Step 492, loss: 0.0886383056640625
Step 493, loss: 0.08623740077018738
Step 494, loss: 0.09342889487743378
Step 495, loss: 0.09426616132259369
Step 496, loss: 0.08552157133817673
Step 497, loss: 0.10631301254034042
Step 498, loss: 0.08671450614929199
Step 499, loss: 0.0800226703286171
Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 159, in wait_for_base_dir_creation
Step 454, loss: 0.08738715201616287
Step 455, loss: 0.08078156411647797
Step 456, loss: 0.1025397926568985
Step 457, loss: 0.09246592223644257
Step 458, loss: 0.09283715486526489
Step 459, loss: 0.08990254253149033
Step 460, loss: 0.09983637928962708
Step 461, loss: 0.10962655395269394
Step 462, loss: 0.08536367118358612
Step 463, loss: 0.08930333703756332
Step 464, loss: 0.09678693860769272
Step 465, loss: 0.09519515931606293
Step 466, loss: 0.11151231080293655
Step 467, loss: 0.11232274770736694
Step 468, loss: 0.08937115967273712
Step 469, loss: 0.09148123115301132
Step 470, loss: 0.0892455130815506
Step 471, loss: 0.10928546637296677
Step 472, loss: 0.0883624404668808
Step 473, loss: 0.09327864646911621
Step 474, loss: 0.08831151574850082
Step 475, loss: 0.09559176117181778
Step 476, loss: 0.10287821292877197
Step 477, loss: 0.10371410101652145
Step 478, loss: 0.09492306411266327
Step 479, loss: 0.08908714354038239
Step 480, loss: 0.09100113809108734
Step 481, loss: 0.08331077545881271
Step 482, loss: 0.10054487735033035
Step 483, loss: 0.09636945277452469
Step 484, loss: 0.09950514882802963
Step 485, loss: 0.09211280941963196
Step 486, loss: 0.09582772105932236
Step 487, loss: 0.08931293338537216
Step 488, loss: 0.08673063665628433
Step 489, loss: 0.09423231333494186
Step 490, loss: 0.09473028779029846
Step 491, loss: 0.08363447338342667
Step 492, loss: 0.0886383056640625
Step 493, loss: 0.08623740077018738
Step 494, loss: 0.09342889487743378
Step 495, loss: 0.09426616132259369
Step 496, loss: 0.08552157133817673
Step 497, loss: 0.10631301254034042
Step 498, loss: 0.08671450614929199
Step 499, loss: 0.0800226703286171
Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 159, in wait_for_base_dir_creation
    await asyncio.sleep(0.25)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 605, in sleep
    await asyncio.sleep(0.25)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 605, in sleep
    await asyncio.sleep(0.25)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 605, in sleep
    await asyncio.sleep(0.25)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 605, in sleep
    return await future
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return await future
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return await future
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return await future
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 162, in _maybe_create_base_dir
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 162, in _maybe_create_base_dir
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 162, in _maybe_create_base_dir
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 162, in _maybe_create_base_dir
    await asyncio.wait_for(
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    await asyncio.wait_for(
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    await asyncio.wait_for(
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    await asyncio.wait_for(
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar_jobs/train_tokenizer.py", line 270, in <module>
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar_jobs/train_tokenizer.py", line 270, in <module>
    orbax_checkpointer.save(
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/checkpointers/checkpointer.py", line 255, in save
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar_jobs/train_tokenizer.py", line 270, in <module>
    orbax_checkpointer.save(
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/checkpointers/checkpointer.py", line 255, in save
    orbax_checkpointer.save(
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/checkpointers/checkpointer.py", line 255, in save
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar_jobs/train_tokenizer.py", line 270, in <module>
    orbax_checkpointer.save(
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/checkpointers/checkpointer.py", line 255, in save
    self._handler.save(tmpdir.get(), args=ckpt_args)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/pytree_checkpoint_handler.py", line 594, in save
    self._handler.save(tmpdir.get(), args=ckpt_args)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/pytree_checkpoint_handler.py", line 594, in save
    self._handler.save(tmpdir.get(), args=ckpt_args)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/pytree_checkpoint_handler.py", line 594, in save
    self._handler.save(tmpdir.get(), args=ckpt_args)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/pytree_checkpoint_handler.py", line 594, in save
    self._handler_impl.save(directory, args=args)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/base_pytree_checkpoint_handler.py", line 554, in save
    self._handler_impl.save(directory, args=args)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/base_pytree_checkpoint_handler.py", line 554, in save
    self._handler_impl.save(directory, args=args)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/base_pytree_checkpoint_handler.py", line 554, in save
    self._handler_impl.save(directory, args=args)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/base_pytree_checkpoint_handler.py", line 554, in save
    asyncio_utils.run_sync(async_save(directory, *args, **kwargs))
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/asyncio_utils.py", line 50, in run_sync
    asyncio_utils.run_sync(async_save(directory, *args, **kwargs))
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/asyncio_utils.py", line 50, in run_sync
    asyncio_utils.run_sync(async_save(directory, *args, **kwargs))
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/asyncio_utils.py", line 50, in run_sync
    asyncio_utils.run_sync(async_save(directory, *args, **kwargs))
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/asyncio_utils.py", line 50, in run_sync
    return asyncio.run(coro)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/runners.py", line 44, in run
    return asyncio.run(coro)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/runners.py", line 44, in run
    return asyncio.run(coro)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/runners.py", line 44, in run
    return asyncio.run(coro)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return loop.run_until_complete(main)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return loop.run_until_complete(main)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return loop.run_until_complete(main)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/base_pytree_checkpoint_handler.py", line 552, in async_save
    return future.result()
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/base_pytree_checkpoint_handler.py", line 552, in async_save
    return future.result()
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/base_pytree_checkpoint_handler.py", line 552, in async_save
    return future.result()
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/base_pytree_checkpoint_handler.py", line 552, in async_save
    f.result()  # Block on result.
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 174, in result
    f.result()  # Block on result.
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 174, in result
    f.result()  # Block on result.
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 174, in result
    f.result()  # Block on result.
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 174, in result
    f.result(timeout=time_remaining)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 397, in result
    f.result(timeout=time_remaining)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 397, in result
    f.result(timeout=time_remaining)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 397, in result
    f.result(timeout=time_remaining)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 397, in result
    return self._f.result(timeout=timeout)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 348, in result
    return self._f.result(timeout=timeout)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 348, in result
    return self._f.result(timeout=timeout)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 348, in result
    return self._f.result(timeout=timeout)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 348, in result
    return self._t.result(timeout=timeout)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 297, in result
    return self._t.result(timeout=timeout)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 297, in result
    return self._t.result(timeout=timeout)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 297, in result
    self.join(timeout=timeout)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 294, in join
    return self._t.result(timeout=timeout)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 297, in result
    self.join(timeout=timeout)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 294, in join
    raise self._exception
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 281, in run
    self.join(timeout=timeout)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 294, in join
    self.join(timeout=timeout)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 294, in join
    super().run()
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/threading.py", line 953, in run
    raise self._exception
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 281, in run
    raise self._exception
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 281, in run
    super().run()
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/threading.py", line 953, in run
    raise self._exception
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 281, in run
    super().run()
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/threading.py", line 953, in run
    super().run()
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 230, in _target_setting_result
    self._target(*self._args, **self._kwargs)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 230, in _target_setting_result
    self._result = target()
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 341, in <lambda>
    self._result = target()
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 341, in <lambda>
    self._target(*self._args, **self._kwargs)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 230, in _target_setting_result
    self._target(*self._args, **self._kwargs)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 230, in _target_setting_result
    target=lambda: asyncio_utils.run_sync(coro),
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/asyncio_utils.py", line 50, in run_sync
    target=lambda: asyncio_utils.run_sync(coro),
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/asyncio_utils.py", line 50, in run_sync
    self._result = target()
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 341, in <lambda>
    return asyncio.run(coro)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/runners.py", line 44, in run
    self._result = target()
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/futures/future.py", line 341, in <lambda>
    return asyncio.run(coro)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return loop.run_until_complete(main)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    target=lambda: asyncio_utils.run_sync(coro),
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/asyncio_utils.py", line 50, in run_sync
    target=lambda: asyncio_utils.run_sync(coro),
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/asyncio_utils.py", line 50, in run_sync
    return future.result()
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py", line 1105, in _background_serialize
    return future.result()
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py", line 1105, in _background_serialize
    return asyncio.run(coro)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/runners.py", line 44, in run
    return asyncio.run(coro)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return loop.run_until_complete(main)
  File "/home/hk-project-p0023960/tum_cte0515/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py", line 1105, in _background_serialize
    return future.result()
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py", line 1105, in _background_serialize
    await asyncio.gather(*write_coros)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 194, in write
    await asyncio.gather(*write_coros)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 194, in write
    await self._maybe_create_base_dir(file_path.parent)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 169, in _maybe_create_base_dir
    await self._maybe_create_base_dir(file_path.parent)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 169, in _maybe_create_base_dir
    await asyncio.gather(*write_coros)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 194, in write
    await asyncio.gather(*write_coros)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 194, in write
    raise ValueError(
ValueError: [process_index=5] Timed out waiting for array_metadatas base directory creation: /hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/checkpoints/3313565/tokenizer_1751512102_500.orbax-checkpoint-tmp-0/array_metadatas. timeout=600 seconds. primary_process=0
    raise ValueError(
ValueError: [process_index=7] Timed out waiting for array_metadatas base directory creation: /hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/checkpoints/3313565/tokenizer_1751512102_500.orbax-checkpoint-tmp-0/array_metadatas. timeout=600 seconds. primary_process=0
    await self._maybe_create_base_dir(file_path.parent)
    await self._maybe_create_base_dir(file_path.parent)
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 169, in _maybe_create_base_dir
  File "/hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/metadata/array_metadata_store.py", line 169, in _maybe_create_base_dir
    raise ValueError(
ValueError: [process_index=4] Timed out waiting for array_metadatas base directory creation: /hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/checkpoints/3313565/tokenizer_1751512102_500.orbax-checkpoint-tmp-0/array_metadatas. timeout=600 seconds. primary_process=0
    raise ValueError(
ValueError: [process_index=6] Timed out waiting for array_metadatas base directory creation: /hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/checkpoints/3313565/tokenizer_1751512102_500.orbax-checkpoint-tmp-0/array_metadatas. timeout=600 seconds. primary_process=0
2025-07-03 05:45:46.564504: E external/xla/xla/tsl/distributed_runtime/coordination/coordination_service.cc:1689] Shutdown barrier in coordination service has failed:
DEADLINE_EXCEEDED: Barrier timed out. Id: Shutdown::11800344298275656134::0. This usually happens because a task triggered the barrier too early or too slowly. Please look at the task logs (both timed out and first task) to debug further.
# of tasks that reached the barrier: 4/8.
The first task at the barrier: /job:jax_worker/replica:0/task:6. Some timed out task names:
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:2
 [type.googleapis.com/tensorflow.CoordinationServiceError=''] [type.googleapis.com/tensorflow.BarrierError='
\x1eShutdown::11800344298275656134']
This suggests that the workers are out of sync. Either at least one worker (a) crashed early due to program error or scheduler events (e.g. preemption, eviction), (b) was too fast in its execution, or (c) too slow / hanging. Check the logs (both the program and scheduler events) for an earlier error to identify the root cause.
2025-07-03 05:45:46.564533: E external/xla/xla/tsl/distributed_runtime/coordination/coordination_service.cc:1730] Use error polling to propagate the following error to all tasks: INTERNAL: Shutdown barrier has failed.
Barrier result: 'DEADLINE_EXCEEDED: Barrier timed out. Id: Shutdown::11800344298275656134::0. This usually happens because a task triggered the barrier too early or too slowly. Please look at the task logs (both timed out and first task) to debug further.
# of tasks that reached the barrier: 4/8.
The first task at the barrier: /job:jax_worker/replica:0/task:6. Some timed out task names:
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:2
 [type.googleapis.com/tensorflow.CoordinationServiceError=''] [type.googleapis.com/tensorflow.BarrierError='
\x1eShutdown::11800344298275656134'] [type.googleapis.com/tensorflow.CoordinationServiceError='']
2025-07-03 05:45:46.564763: E external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc:427] Polled an error from coordination service (this can be an error from this or another task).
2025-07-03 05:45:46.564823: E external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc:427] Polled an error from coordination service (this can be an error from this or another task).
2025-07-03 05:45:46.564865: E external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc:427] Polled an error from coordination service (this can be an error from this or another task).
2025-07-03 05:45:46.564928: F external/xla/xla/pjrt/distributed/client.h:80] Terminating process because the JAX distributed service detected fatal errors. This most likely indicates that another task died; see the other task logs for more details. Disable Python buffering, i.e. `python -u`, to be sure to see all the previous output. absl::Status: INTERNAL: Shutdown barrier has failed.
Barrier result: 'DEADLINE_EXCEEDED: Barrier timed out. Id: Shutdown::11800344298275656134::0. This usually happens because a task triggered the barrier too early or too slowly. Please look at the task logs (both timed out and first task) to debug further.
# of tasks that reached the barrier: 4/8.
The first task at the barrier: /job:jax_worker/replica:0/task:6. Some timed out task names:
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:2
 [type.googleapis.com/tensorflow.CoordinationServiceError=''] [type.googleapis.com/tensorflow.BarrierError='
\x1eShutdown::11800344298275656134']

RPC: /tensorflow.CoordinationService/PollForError [type.googleapis.com/tensorflow.CoordinationServiceError='']
2025-07-03 05:45:46.564907: E external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc:427] Polled an error from coordination service (this can be an error from this or another task).
2025-07-03 05:45:46.564948: F external/xla/xla/pjrt/distributed/client.h:80] Terminating process because the JAX distributed service detected fatal errors. This most likely indicates that another task died; see the other task logs for more details. Disable Python buffering, i.e. `python -u`, to be sure to see all the previous output. absl::Status: INTERNAL: Shutdown barrier has failed.
Barrier result: 'DEADLINE_EXCEEDED: Barrier timed out. Id: Shutdown::11800344298275656134::0. This usually happens because a task triggered the barrier too early or too slowly. Please look at the task logs (both timed out and first task) to debug further.
# of tasks that reached the barrier: 4/8.
The first task at the barrier: /job:jax_worker/replica:0/task:6. Some timed out task names:
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:2
 [type.googleapis.com/tensorflow.CoordinationServiceError=''] [type.googleapis.com/tensorflow.BarrierError='
\x1eShutdown::11800344298275656134']

RPC: /tensorflow.CoordinationService/PollForError [type.googleapis.com/tensorflow.CoordinationServiceError='']
2025-07-03 05:45:46.565002: F external/xla/xla/pjrt/distributed/client.h:80] Terminating process because the JAX distributed service detected fatal errors. This most likely indicates that another task died; see the other task logs for more details. Disable Python buffering, i.e. `python -u`, to be sure to see all the previous output. absl::Status: INTERNAL: Shutdown barrier has failed.
Barrier result: 'DEADLINE_EXCEEDED: Barrier timed out. Id: Shutdown::11800344298275656134::0. This usually happens because a task triggered the barrier too early or too slowly. Please look at the task logs (both timed out and first task) to debug further.
# of tasks that reached the barrier: 4/8.
The first task at the barrier: /job:jax_worker/replica:0/task:6. Some timed out task names:
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:2
 [type.googleapis.com/tensorflow.CoordinationServiceError=''] [type.googleapis.com/tensorflow.BarrierError='
\x1eShutdown::11800344298275656134']

RPC: /tensorflow.CoordinationService/PollForError [type.googleapis.com/tensorflow.CoordinationServiceError='']
2025-07-03 05:45:46.565068: F external/xla/xla/pjrt/distributed/client.h:80] Terminating process because the JAX distributed service detected fatal errors. This most likely indicates that another task died; see the other task logs for more details. Disable Python buffering, i.e. `python -u`, to be sure to see all the previous output. absl::Status: INTERNAL: Shutdown barrier has failed.
Barrier result: 'DEADLINE_EXCEEDED: Barrier timed out. Id: Shutdown::11800344298275656134::0. This usually happens because a task triggered the barrier too early or too slowly. Please look at the task logs (both timed out and first task) to debug further.
# of tasks that reached the barrier: 4/8.
The first task at the barrier: /job:jax_worker/replica:0/task:6. Some timed out task names:
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:2
 [type.googleapis.com/tensorflow.CoordinationServiceError=''] [type.googleapis.com/tensorflow.BarrierError='
\x1eShutdown::11800344298275656134']

RPC: /tensorflow.CoordinationService/PollForError [type.googleapis.com/tensorflow.CoordinationServiceError='']
2025-07-03 05:45:46.569576: E external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc:427] Polled an error from coordination service (this can be an error from this or another task).
2025-07-03 05:45:46.569603: E external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc:427] Polled an error from coordination service (this can be an error from this or another task).
2025-07-03 05:45:46.569592: E external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc:427] Polled an error from coordination service (this can be an error from this or another task).
2025-07-03 05:45:46.569702: E external/xla/xla/tsl/distributed_runtime/coordination/coordination_service_agent.cc:427] Polled an error from coordination service (this can be an error from this or another task).
2025-07-03 05:45:46.576720: F external/xla/xla/pjrt/distributed/client.h:80] Terminating process because the JAX distributed service detected fatal errors. This most likely indicates that another task died; see the other task logs for more details. Disable Python buffering, i.e. `python -u`, to be sure to see all the previous output. absl::Status: INTERNAL: Shutdown barrier has failed.
Barrier result: 'DEADLINE_EXCEEDED: Barrier timed out. Id: Shutdown::11800344298275656134::0. This usually happens because a task triggered the barrier too early or too slowly. Please look at the task logs (both timed out and first task) to debug further.
# of tasks that reached the barrier: 4/8.
The first task at the barrier: /job:jax_worker/replica:0/task:6. Some timed out task names:
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:2
 [type.googleapis.com/tensorflow.CoordinationServiceError=''] [type.googleapis.com/tensorflow.BarrierError='
\x1eShutdown::11800344298275656134']

RPC: /tensorflow.CoordinationService/PollForError [type.googleapis.com/tensorflow.CoordinationServiceError='']
2025-07-03 05:45:46.576853: F external/xla/xla/pjrt/distributed/client.h:80] Terminating process because the JAX distributed service detected fatal errors. This most likely indicates that another task died; see the other task logs for more details. Disable Python buffering, i.e. `python -u`, to be sure to see all the previous output. absl::Status: INTERNAL: Shutdown barrier has failed.
Barrier result: 'DEADLINE_EXCEEDED: Barrier timed out. Id: Shutdown::11800344298275656134::0. This usually happens because a task triggered the barrier too early or too slowly. Please look at the task logs (both timed out and first task) to debug further.
# of tasks that reached the barrier: 4/8.
The first task at the barrier: /job:jax_worker/replica:0/task:6. Some timed out task names:
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:2
 [type.googleapis.com/tensorflow.CoordinationServiceError=''] [type.googleapis.com/tensorflow.BarrierError='
\x1eShutdown::11800344298275656134']

RPC: /tensorflow.CoordinationService/PollForError [type.googleapis.com/tensorflow.CoordinationServiceError='']
2025-07-03 05:45:46.576835: F external/xla/xla/pjrt/distributed/client.h:80] Terminating process because the JAX distributed service detected fatal errors. This most likely indicates that another task died; see the other task logs for more details. Disable Python buffering, i.e. `python -u`, to be sure to see all the previous output. absl::Status: INTERNAL: Shutdown barrier has failed.
Barrier result: 'DEADLINE_EXCEEDED: Barrier timed out. Id: Shutdown::11800344298275656134::0. This usually happens because a task triggered the barrier too early or too slowly. Please look at the task logs (both timed out and first task) to debug further.
# of tasks that reached the barrier: 4/8.
The first task at the barrier: /job:jax_worker/replica:0/task:6. Some timed out task names:
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:2
 [type.googleapis.com/tensorflow.CoordinationServiceError=''] [type.googleapis.com/tensorflow.BarrierError='
\x1eShutdown::11800344298275656134']

RPC: /tensorflow.CoordinationService/PollForError [type.googleapis.com/tensorflow.CoordinationServiceError='']
2025-07-03 05:45:46.576791: F external/xla/xla/pjrt/distributed/client.h:80] Terminating process because the JAX distributed service detected fatal errors. This most likely indicates that another task died; see the other task logs for more details. Disable Python buffering, i.e. `python -u`, to be sure to see all the previous output. absl::Status: INTERNAL: Shutdown barrier has failed.
Barrier result: 'DEADLINE_EXCEEDED: Barrier timed out. Id: Shutdown::11800344298275656134::0. This usually happens because a task triggered the barrier too early or too slowly. Please look at the task logs (both timed out and first task) to debug further.
# of tasks that reached the barrier: 4/8.
The first task at the barrier: /job:jax_worker/replica:0/task:6. Some timed out task names:
/job:jax_worker/replica:0/task:0
/job:jax_worker/replica:0/task:2
 [type.googleapis.com/tensorflow.CoordinationServiceError=''] [type.googleapis.com/tensorflow.BarrierError='
\x1eShutdown::11800344298275656134']

RPC: /tensorflow.CoordinationService/PollForError [type.googleapis.com/tensorflow.CoordinationServiceError='']
srun: error: hkn0427: task 2: Aborted (core dumped)
srun: error: hkn0512: tasks 5-6: Aborted (core dumped)
srun: error: hkn0427: tasks 1,3: Aborted (core dumped)
srun: error: hkn0427: task 0: Aborted (core dumped)
srun: error: hkn0512: task 7: Aborted (core dumped)
srun: error: hkn0512: task 4: Aborted (core dumped)

============================= JOB FEEDBACK =============================

Job ID: 3313565
Cluster: hk
User/Group: tum_cte0515/hk-project-p0023960
Account: hk-project-p0023960
State: FAILED (exit code 134)
Partition: accelerated
Nodes: 2
Cores per node: 24
Nodelist: hkn[0427,0512]
CPU Utilized: 03:20:37
CPU Efficiency: 11.00% of 1-06:23:12 core-walltime
Job Wall-clock time: 00:37:59
Starttime: Thu Jul  3 05:07:48 2025
Endtime: Thu Jul  3 05:45:47 2025
Memory Utilized: 74.71 GB (estimated maximum)
Memory Efficiency: 0.00% of 0.00 MB (0.00 MB/node)
Energy Consumed: 5210256 Joule / 1447.29333333333 Watthours
Average node power draw: 2286.20272049144 Watt