[terminal_command] TERMINAL ()
salloc --gpus=1 --ntasks-per-node=1 --cpus-per-task=10 --mem=100Gsalloc: Granted job allocation 29889salloc: Waiting for resource configurationsalloc: Nodes hai004 are ready for jobRunning inside SLURM, Job ID 29889.[franz.srambical@hai004.haicore.berlin:~/jafar] $ bash slurm/dev/franz/berlin/coinrun/sample/maskgit/sample_dynamics_from_fully_trained_tokenizer.sh Sampling from checkpoint: /fast/project/HFMI_SynergyUnit/jafar_ws/checkpoints/coinrun/dynamics/dynamics_coinrun_500m_dataset_w_eval_ff/29851/fast/home/franz.srambical/jafar/.venv/lib/python3.13/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py:1269: UserWarning: Sharding info not provided when restoring. Populating sharding info from sharding file. Please note restoration time will be slightly increased due to reading from file. Note also that this option is unsafe when restoring on a different topology than the checkpoint was saved with.  warnings.warn(Per-frame SSIM: [0.57319623 0.54733205 0.5227049  0.48956802 0.5013784  0.46939224 0.4892099  0.47639948 0.44199222 0.46024764 0.49088758 0.44623953]Per-frame PSNR: [17.134462 16.30024  15.468311 15.132342 15.414605 15.832775 14.787997 14.059711 14.086777 14.237678 14.854488 14.458799]SSIM: 0.4923790395259857PSNR: 15.14734935760498W1002 10:23:15.877250 4186569 pjrt_client.cc:1469] WatchJobStateAsync failed for task goo.gle/debugproto   job_name: "jax_worker": CANCELLED: CANCELLEDAdditional GRPC error information from remote target coordination_service while calling /tensorflow.CoordinationService/WatchJobState::UNKNOWN:Error received from peer  {grpc_status:1, grpc_message:"CANCELLED"} [type.googleapis.com/tensorflow.DerivedStatus=''][franz.srambical@hai004.haicore.berlin:~/jafar] $ bash slurm/dev/franz/berlin/coinrun/sample/maskgit/sample_dynamics_from_fully_trained_tokenizer.sh Sampling from checkpoint: /fast/project/HFMI_SynergyUnit/jafar_ws/checkpoints/coinrun/dynamics/dynamics_coinrun_500m_dataset/29519/fast/home/franz.srambical/jafar/.venv/lib/python3.13/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py:1269: UserWarning: Sharding info not provided when restoring. Populating sharding info from sharding file. Please note restoration time will be slightly increased due to reading from file. Note also that this option is unsafe when restoring on a different topology than the checkpoint was saved with.  warnings.warn(Per-frame SSIM: [0.5646278  0.6038726  0.5279748  0.5055892  0.49214277 0.47423407 0.45490086 0.4486615  0.39024955 0.4109726  0.44856626 0.43377078]Per-frame PSNR: [18.160048 16.95538  15.633233 15.407247 15.042919 14.729964 14.036336 13.592623 13.456531 13.609724 14.187572 14.492484]SSIM: 0.4796302318572998PSNR: 14.942004203796387W1002 10:28:07.966836  135844 pjrt_client.cc:1469] WatchJobStateAsync failed for task goo.gle/debugonly  job_name: "jax_worker": UNAVAILABLE: failed to connect to all addresses; last error: UNKNOWN: ipv4:10.86.2.40:62657: Failed to connect to remote host: Connection refusedAdditional GRPC error information from remote target coordination_service while calling /tensorflow.CoordinationService/WatchJobState::UNKNOWN:Error received from peer  {grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.86.2.40:62657: Failed to connect to remote host: Connection refused"}[franz.srambical@hai004.haicore.berlin:~/jafar] $ source /home/franz.srambical/jafar/data/.venv/bin/activate(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ squeue  JOBID       USER PARTITION NODES  CPUS ST         SUBMIT_TIME          START_TIME     TIME TIME_LIMIT NODELIST(REASON)  29889 franz.sram interacti     1    20  R 2025-10-02T10:22:13 2025-10-02T10:22:13  1:37:21 1-00:00:00 hai004  29888   xiao.liu interacti     1   128  R 2025-10-02T07:43:35 2025-10-02T07:43:35  4:15:59   23:59:00 hai002  29884   xiao.liu interacti     1    64  R 2025-10-02T04:52:43 2025-10-02T04:52:43  7:06:51   23:59:00 hai008  29883   xiao.liu interacti     1   128  R 2025-10-02T04:51:41 2025-10-02T04:51:41  7:07:53   23:59:00 hai007  29880   xiao.liu interacti     1    64  R 2025-10-02T02:20:07 2025-10-02T02:20:07  9:39:27   23:59:00 hai006  29890  ali.bekar  standard     2    64 PD 2025-10-02T10:34:28                 N/A     0:00 1-00:00:00 (Resources)  29774 franz.sram  standard     1    16  R 2025-10-02T10:14:13 2025-10-02T10:16:41  1:42:53 1-00:00:00 hai004  29875 alfred.ngu  standard     1    16  R 2025-10-02T01:05:33 2025-10-02T01:05:33 10:54:01 1-00:00:00 hai001  29874 alfred.ngu  standard     1    16  R 2025-10-02T01:03:32 2025-10-02T01:03:32 10:56:02 1-00:00:00 hai008  29873 alfred.ngu  standard     1    16  R 2025-10-02T01:02:01 2025-10-02T01:02:03 10:57:31 1-00:00:00 hai001  29868 alfred.ngu  standard     1    16  R 2025-10-02T00:47:51 2025-10-02T00:47:51 11:11:43 1-00:00:00 hai001  29866 alfred.ngu  standard     1    16  R 2025-10-02T00:46:04 2025-10-02T00:46:15 11:13:19 1-00:00:00 hai006  29849 nishant.ku  standard     2   384  R 2025-10-01T19:37:07 2025-10-02T00:28:10 11:31:24 1-00:00:00 hai[003,005]  29852 alfred.ngu  standard     1    16  R 2025-10-01T23:11:55 2025-10-01T23:12:24 12:47:10 1-00:00:00 hai001  29851 alfred.ngu  standard     1    16  R 2025-10-01T22:58:03 2025-10-01T22:58:24 13:01:10 1-00:00:00 hai001  29834 alfred.ngu  standard     1    24  R 2025-10-01T18:24:33 2025-10-01T18:24:33 17:35:01 1-00:00:00 hai004(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ fqueuebash: fqueue: command not found(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ sinfo 29834PARTITION   AVAIL  TIMELIMIT  NODES  STATE NODELISTstandard*      up 1-00:00:00      8    mix hai[001-008]interactive    up 1-00:00:00      8    mix hai[001-008](jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ squeue -o 298342983429834298342983429834298342983429834298342983429834298342983429834298342983429834(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ squeue -osqueue: option requires an argument -- 'o'Try "squeue --help" for more information(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ watch -n 1 "squeue -o "%.10i %.16P %.30j %.8u %.8T %.10M %.9l %.6D %R""(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ watch -n 1 "squeue -o '%.10i %.16P %.30j %.8u %.8T %.10M %.9l %.6D %R'"(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ ls /fast/project/HFMI_SynergyUnit/jafar_ws/datadata/              data_uncompressed/ (jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ ls /fast/project/HFMI_SynergyUnit/jafar_ws/data/.cache/           .gitattributes    atari/            breakout/         climber/          coinrun/          minatar_breakout/ pong/             starpilot/        tars/             (jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ ls /fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records/                          array_records_10k/                      array_records_250M_npy_arr_rec/         array_records_500m_no_distinct_seeding/ array_records_single/                   array_records_100k_npy_arr_rec/         array_records_10k_npy_arr_rec/          array_records_500M_npy_arr_rec/         array_records_500m_seed_w_increment/    dataset_investigation/                  array_records_10M_arr_rec_no_seeding/   array_records_10k_w_seeding/            array_records_500k_seed_w_increment/    array_records_big/                      dataset_investigation_seeded/           array_records_10M_npy_arr_rec/          array_records_10m/                      array_records_500m/                     array_records_overfit/                  npy_records_10m/                        (jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ ls /fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/array_record  npy(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ cd abash: cd: a: No such file or directory(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ ls' '      LICENSE     __pycache__   data               experiments                        generation_1754228185.703112.gif    generation_1754233948.4598346.gif   jasmine   pyproject.toml   slurm             tensorboard   third_party   wandb'$PWD'   README.md   checkpoints   data_arrayrecord   generation_1754227585.831618.gif   generation_1754228581.3146486.gif   gifs                                outputs   runs             slurm-29284.out   tests         uv.lock(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ cd /fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec] $ lsarray_record  npy(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec] $ cd array_record/(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/array_record] $ lstrain(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/array_record] $ cd train/(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/array_record/train] $ ls^C(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/array_record/train] $ ^C(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/array_record/train] $ ^C(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/array_record/train] $ ls ^C(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/array_record/train] $ ^C(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/array_record/train] $ ^C(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/array_record/train] $ ^C(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/array_record/train] $ ls -l | wc -l190160(jasmine-data) [franz.srambical@hai004.haicore.berlin:/fast/project/HFMI_SynergyUnit/jafar_ws/data/coinrun/array_records_250M_npy_arr_rec/array_record/train] $ cd(jasmine-data) [franz.srambical@hai004.haicore.berlin:~] $ cd jafar/(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ cd slurm(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar/slurm] $ git commit -m "gt actions concat"On branch mainYour branch is behind 'origin/main' by 6 commits, and can be fast-forwarded.  (use "git pull" to update your local branch)Changes not staged for commit:  (use "git add/rm <file>..." to update what will be committed)  (use "git restore <file>..." to discard changes in working directory)        modified:   dev/franz/berlin/coinrun/sample/maskgit/sample_dynamics_from_fully_trained_tokenizer.sh        deleted:    jobs/franz/berlin/coinrun/mila_submission/coinrun_dynamics_base.sh        deleted:    jobs/franz/berlin/coinrun/mila_submission/coinrun_lam_base.sh        deleted:    jobs/franz/berlin/coinrun/mila_submission/coinrun_lam_no_flash_attention.sh        deleted:    jobs/franz/berlin/coinrun/mila_submission/coinrun_tokenizer_base.sh        deleted:    jobs/franz/berlin/coinrun/submission_debug/coinrun_dynamics_base_patch_size_16_action_prepend.sh        deleted:    jobs/franz/berlin/coinrun/submission_debug/coinrun_dynamics_base_patch_size_16_action_prepend_cos.sh        deleted:    jobs/franz/berlin/coinrun/submission_debug/coinrun_dynamics_base_patch_size_16_noise-branch.sh        deleted:    jobs/franz/berlin/coinrun/submission_debug/coinrun_dynamics_base_patch_size_4_action_prepend.sh        deleted:    jobs/franz/berlin/coinrun/submission_debug/coinrun_dynamics_base_patch_size_4_main.sh        deleted:    jobs/franz/berlin/coinrun/submission_debug/coinrun_dynamics_base_patch_size_4_noise-branch.sh        deleted:    jobs/franz/berlin/coinrun/submission_debug/coinrun_tokenizer_patch_size_4.shUntracked files:  (use "git add <file>..." to include in what will be committed)        jobs/franz/berlin/coinrun/mila_submission/legacy_runs_50M_dataset/no changes added to commit (use "git add" and/or "git commit -a")(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar/slurm] $ git add *(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar/slurm] $ git commit -m "gt actions concat"[main f80b015] gt actions concat 13 files changed, 78 insertions(+), 3 deletions(-) rename jobs/franz/berlin/coinrun/mila_submission/{ => legacy_runs_50M_dataset}/coinrun_dynamics_base.sh (100%) rename jobs/franz/berlin/coinrun/mila_submission/{ => legacy_runs_50M_dataset}/coinrun_lam_base.sh (100%) rename jobs/franz/berlin/coinrun/mila_submission/{ => legacy_runs_50M_dataset}/coinrun_lam_no_flash_attention.sh (100%) rename jobs/franz/berlin/coinrun/mila_submission/{ => legacy_runs_50M_dataset}/coinrun_tokenizer_base.sh (100%) rename jobs/franz/berlin/coinrun/{ => mila_submission/legacy_runs_50M_dataset}/submission_debug/coinrun_dynamics_base_patch_size_16_action_prepend.sh (100%) rename jobs/franz/berlin/coinrun/{ => mila_submission/legacy_runs_50M_dataset}/submission_debug/coinrun_dynamics_base_patch_size_16_action_prepend_cos.sh (100%) rename jobs/franz/berlin/coinrun/{ => mila_submission/legacy_runs_50M_dataset}/submission_debug/coinrun_dynamics_base_patch_size_16_noise-branch.sh (100%) rename jobs/franz/berlin/coinrun/{ => mila_submission/legacy_runs_50M_dataset}/submission_debug/coinrun_dynamics_base_patch_size_4_action_prepend.sh (100%) rename jobs/franz/berlin/coinrun/{ => mila_submission/legacy_runs_50M_dataset}/submission_debug/coinrun_dynamics_base_patch_size_4_main.sh (100%) rename jobs/franz/berlin/coinrun/{ => mila_submission/legacy_runs_50M_dataset}/submission_debug/coinrun_dynamics_base_patch_size_4_noise-branch.sh (100%) create mode 100644 jobs/franz/berlin/coinrun/mila_submission/legacy_runs_50M_dataset/submission_debug/coinrun_dynamics_gt_actions_concat_branch.sh rename jobs/franz/berlin/coinrun/{ => mila_submission/legacy_runs_50M_dataset}/submission_debug/coinrun_tokenizer_patch_size_4.sh (100%)(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar/slurm] $ git pushTo github.com:p-doom/slurm.git ! [rejected]        main -> main (non-fast-forward)error: failed to push some refs to 'github.com:p-doom/slurm.git'hint: Updates were rejected because the tip of your current branch is behindhint: its remote counterpart. If you want to integrate the remote changes,hint: use 'git pull' before pushing again.hint: See the 'Note about fast-forwards' in 'git push --help' for details.(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar/slurm] $ git pullMerge made by the 'ort' strategy. dev/alfred/berlin/test_exposure_bias/dynamics_full_prec copy.sbatch                                  |  82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ dev/alfred/berlin/test_exposure_bias/dynamics_maskgit_full_prec.sbatch                               |  82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ dev/alfred/berlin/test_exposure_bias/dynamics_maskgit_mix_prec.sbatch                                |  82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ dev/alfred/berlin/test_exposure_bias/dynamics_nan_fix.sbatch                                         |  80 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ jobs/alfred/berlin/coinrun/coinrun_data/generate_data_500mio_no_distinct_seeding.sbatch              |  35 ++++++++++++++++++++++++++++++ jobs/alfred/berlin/coinrun/overfitting_test/500m_dataset/dynamics_w_eval_ff.sbatch                   |  82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ jobs/alfred/berlin/coinrun/overfitting_test/500m_dataset/dynamics_w_eval_ff_from_current_main.sbatch |  82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ jobs/alfred/berlin/coinrun/w_val/tokenizer_baseline_debug_default_jasming_old_commit.sbatch          | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ jobs/alfred/berlin/workshop/case_study_vanilla_genie/debug/lam.sh                                    |  70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ jobs/alfred/berlin/workshop/case_study_vanilla_genie/debug/tokenizer.sh                              |  72 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ jobs/alfred/berlin/workshop/case_study_vanilla_genie/tokenizer.sh                                    |  72 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ jobs/alfred/berlin/workshop/confirm_mix_prec_speedup/dynamics.sbatch                                 |  84 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ jobs/alfred/berlin/workshop/jafar_default/coinrun_dynamics_base.sh                                   |  70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ jobs/alfred/berlin/workshop/mix_prec_ablation/coinrun_dynamics_full_prec.sh                          |  71 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ jobs/alfred/berlin/workshop/mix_prec_ablation/coinrun_dynamics_mix_precision.sh                      |  70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ jobs/alfred/berlin/workshop/mix_prec_ablation/coinrun_dynamics_mix_precision_no_flash_attention.sh   |  71 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 16 files changed, 1209 insertions(+) create mode 100644 dev/alfred/berlin/test_exposure_bias/dynamics_full_prec copy.sbatch create mode 100644 dev/alfred/berlin/test_exposure_bias/dynamics_maskgit_full_prec.sbatch create mode 100644 dev/alfred/berlin/test_exposure_bias/dynamics_maskgit_mix_prec.sbatch create mode 100644 dev/alfred/berlin/test_exposure_bias/dynamics_nan_fix.sbatch create mode 100644 jobs/alfred/berlin/coinrun/coinrun_data/generate_data_500mio_no_distinct_seeding.sbatch create mode 100644 jobs/alfred/berlin/coinrun/overfitting_test/500m_dataset/dynamics_w_eval_ff.sbatch create mode 100644 jobs/alfred/berlin/coinrun/overfitting_test/500m_dataset/dynamics_w_eval_ff_from_current_main.sbatch create mode 100644 jobs/alfred/berlin/coinrun/w_val/tokenizer_baseline_debug_default_jasming_old_commit.sbatch create mode 100644 jobs/alfred/berlin/workshop/case_study_vanilla_genie/debug/lam.sh create mode 100644 jobs/alfred/berlin/workshop/case_study_vanilla_genie/debug/tokenizer.sh create mode 100644 jobs/alfred/berlin/workshop/case_study_vanilla_genie/tokenizer.sh create mode 100644 jobs/alfred/berlin/workshop/confirm_mix_prec_speedup/dynamics.sbatch create mode 100644 jobs/alfred/berlin/workshop/jafar_default/coinrun_dynamics_base.sh create mode 100644 jobs/alfred/berlin/workshop/mix_prec_ablation/coinrun_dynamics_full_prec.sh create mode 100644 jobs/alfred/berlin/workshop/mix_prec_ablation/coinrun_dynamics_mix_precision.sh create mode 100644 jobs/alfred/berlin/workshop/mix_prec_ablation/coinrun_dynamics_mix_precision_no_flash_attention.sh(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar/slurm] $ git pushEnumerating objects: 50, done.Counting objects: 100% (42/42), done.Delta compression using up to 224 threadsCompressing objects: 100% (25/25), done.Writing objects: 100% (28/28), 4.21 KiB | 307.00 KiB/s, done.Total 28 (delta 9), reused 0 (delta 0), pack-reused 0remote: Resolving deltas: 100% (9/9), completed with 2 local objects.To github.com:p-doom/slurm.git   bca718c..f59954f  main -> main(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar/slurm] $ cd(jasmine-data) [franz.srambical@hai004.haicore.berlin:~] $ cd jafar/(jasmine-data) [franz.srambical@hai004.haicore.berlin:~/jafar] $ deactivate [franz.srambical@hai004.haicore.berlin:~/jafar] $ sbatch slurm/jobs/franz/berlin/coinrun/mila_submission/legacy_runs_50M_dataset/submission_debug/coinrun_dynamics_gt_actions_concat_branch.shSubmitted batch job 29898[franz.srambical@hai004.haicore.berlin:~/jafar] $ watch squeue[franz.srambical@hai004.haicore.berlin:~/jafar] $ scalcbash: scalc: command not found[franz.srambical@hai004.haicore.berlin:~/jafar] $ scancel 29889salloc: Job allocation 29889 has been revoked.                                              [franz.srambical@hai004.haicore.berlin:~/jafar] $ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.[2025-10-02T12:17:33.983] error: *** STEP 29889.interactive ON hai004 CANCELLED AT 2025-10-02T12:17:33 DUE to SIGNAL Terminated ***^C[franz.srambical@hai004.haicore.berlin:~/jafar] $ watch squeue(reverse-i-search)`log': less /fast/project/HFMI_SynergyUnit/jafar_ws/logs/franz/coinrun/dynamics/dynamics_coinrun_mila_submission_patch_size_4_action_prepend_branch_29804.srun: error: hai004: task 0: Killed