[terminal_command] TERMINAL ()
salloc --gpus=2 --ntasks-per-node=2 --cpus-per-task=1 --mem=100Gsalloc: Granted job allocation 14977salloc: Waiting for resource configurationsalloc: Nodes hai004 are ready for jobRunning inside SLURM, Job ID 14977.[franz.srambical@hai004.haicore.berlin:~/jafar] $ bash experiments/tokenizer_grain_checkpointing.sh Running on 2 devices.Running on 2 devices.Counting all components: ['decoder', 'encoder', 'vq']Parameter counts:{'decoder': 16858736, 'encoder': 16858752, 'vq': 32768, 'total': 33750256}Counting all components: ['decoder', 'encoder', 'vq']Parameter counts:{'decoder': 16858736, 'encoder': 16858752, 'vq': 32768, 'total': 33750256}WARNING:absl:Dropping 1 examples of 5 examples (shard 2).WARNING:absl:Dropping 1 examples of 5 examples (shard 2).Traceback (most recent call last):  File "/fast/home/franz.srambical/jafar/train_tokenizer.py", line 303, in <module>    restored = checkpoint_manager.restore(  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/checkpoint_manager.py", line 1647, in restore    restored = self._checkpointer.restore(restore_directory, args=args)  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/checkpointers/async_checkpointer.py", line 562, in restore    return super().restore(directory, *args, **kwargs)  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/checkpointers/checkpointer.py", line 304, in restore    restored = self._restore(directory, args=ckpt_args)  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/checkpointers/checkpointer.py", line 323, in _restore    return self._handler.restore(directory, args=args)  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/composite_checkpoint_handler.py", line 859, in restore    restored[item_name] = handler.restore(  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/grain/_src/python/checkpoint_handlers.py", line 62, in restore    raise ValueError(f"File {filename} does not exist.")ValueError: File /home/franz.srambical/jafar/checkpoints/tokenizer_openai_grain_checkpointing/000015/dataloader_state/process_1-of-2.json does not exist.Traceback (most recent call last):  File "/fast/home/franz.srambical/jafar/train_tokenizer.py", line 303, in <module>    restored = checkpoint_manager.restore(  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/checkpoint_manager.py", line 1647, in restore    restored = self._checkpointer.restore(restore_directory, args=args)  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/checkpointers/async_checkpointer.py", line 562, in restore    return super().restore(directory, *args, **kwargs)  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/checkpointers/checkpointer.py", line 304, in restore    restored = self._restore(directory, args=ckpt_args)  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/checkpointers/checkpointer.py", line 323, in _restore    return self._handler.restore(directory, args=args)  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/handlers/composite_checkpoint_handler.py", line 859, in restore    restored[item_name] = handler.restore(  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/grain/_src/python/checkpoint_handlers.py", line 62, in restore    raise ValueError(f"File {filename} does not exist.")ValueError: File /home/franz.srambical/jafar/checkpoints/tokenizer_openai_grain_checkpointing/000015/dataloader_state/process_0-of-2.json does not exist.srun: error: hai004: tasks 0-1: Exited with exit code 1[franz.srambical@hai004.haicore.berlin:~/jafar] $ salloc: Job allocation 14977 has been revoked.                                                                                                srun: Job step aborted: Waiting up to 32 seconds for job step to finish.[2025-08-05T15:57:49.689] error: *** STEP 14977.interactive ON hai004 CANCELLED AT 2025-08-05T15:57:49 DUE to SIGNAL Terminated ***srun: error: hai004: task 0: Killed