[terminal_command] TERMINAL ()
salloc --gpus=1 --ntasks-per-node=1 --cpus-per-task=64 --mem=100Gsalloc: Granted job allocation 29666salloc: Nodes hai005 are ready for jobRunning inside SLURM, Job ID 29666.[franz.srambical@hai005.haicore.berlin:~/jafar/data] $ uv run jasmine_data/__init__.py                atari/                     minatar/                   pngs_to_array_records.py   video_to_array_records.py  __pycache__/               coinrun/                   minecraft/                 utils.py                   [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ uv run jasmine_data/atari/visualize_array_record.py --input data/atari_episodes/train/data_00 --output_dir=test_output data_0000.array_record  data_0006.array_record  data_0012.array_record  data_0018.array_record  data_0024.array_record  data_0030.array_record  data_0036.array_record  data_0042.array_recorddata_0001.array_record  data_0007.array_record  data_0013.array_record  data_0019.array_record  data_0025.array_record  data_0031.array_record  data_0037.array_record  data_0043.array_recorddata_0002.array_record  data_0008.array_record  data_0014.array_record  data_0020.array_record  data_0026.array_record  data_0032.array_record  data_0038.array_record  data_0003.array_record  data_0009.array_record  data_0015.array_record  data_0021.array_record  data_0027.array_record  data_0033.array_record  data_0039.array_record  data_0004.array_record  data_0010.array_record  data_0016.array_record  data_0022.array_record  data_0028.array_record  data_0034.array_record  data_0040.array_record  data_0005.array_record  data_0011.array_record  data_0017.array_record  data_0023.array_record  data_0029.array_record  data_0035.array_record  data_0041.array_record  [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ uv run jasmine_data/atari/visualize_array_record.py --input data/atari_episodes/train/data_0041.array_record --output_dir=test_output       Built dopamine-rl @ file:///fast/home/franz.srambical/jafar/third_party/dopamineUninstalled 1 package in 41msInstalled 1 package in 28msLoaded record with sequence_length=160, actions_present=True/fast/home/franz.srambical/jafar/data/jasmine_data/atari/visualize_array_record.py:111: DeprecationWarning: 'mode' parameter is deprecated and will be removed in Pillow 13 (2026-10-15)  img = Image.fromarray(img_np_rgb, mode="RGB")Saved GIF to test_output/sequence.gif with 160 frames (H=84, W=84, C=3, fps=10).[franz.srambical@hai005.haicore.berlin:~/jafar/data] $ ls__pycache__  data  jasmine_data  pyproject.toml  runs  test_output  uv.lock[franz.srambical@hai005.haicore.berlin:~/jafar/data] $ squeue  JOBID       USER PARTITION NODES  CPUS ST         SUBMIT_TIME          START_TIME     TIME TIME_LIMIT NODELIST(REASON)  29666 franz.sram interacti     1   128  R 2025-09-29T20:08:03 2025-09-29T20:08:03     6:39 1-00:00:00 hai005  29664 franz.sram interacti     1   128  R 2025-09-29T19:33:16 2025-09-29T19:33:16    41:26 1-00:00:00 hai004  29653   xiao.liu interacti     1    64  R 2025-09-29T15:54:10 2025-09-29T15:54:10  4:20:32   23:59:00 hai002  29652   xiao.liu interacti     1    64  R 2025-09-29T15:53:48 2025-09-29T15:53:48  4:20:54   23:59:00 hai006  29651   xiao.liu interacti     1    64  R 2025-09-29T15:53:13 2025-09-29T15:53:13  4:21:29   23:59:00 hai007  29665 johannmich  standard     1    48  R 2025-09-29T19:51:12 2025-09-29T19:51:12    23:30    8:00:00 hai003  29624 alfred.ngu  standard     1    16  R 2025-09-29T17:00:52 2025-09-29T17:02:56  3:11:46 1-00:00:00 hai003  29660 alfred.ngu  standard     1    16  R 2025-09-29T16:14:02 2025-09-29T16:14:18  4:00:24 1-00:00:00 hai001  29659 alfred.ngu  standard     1    16  R 2025-09-29T16:13:44 2025-09-29T16:13:55  4:00:47 1-00:00:00 hai008  29658 alfred.ngu  standard     1    16  R 2025-09-29T16:08:31 2025-09-29T16:08:31  4:06:11 1-00:00:00 hai001  29657 alfred.ngu  standard     1    16  R 2025-09-29T16:07:47 2025-09-29T16:07:47  4:06:55 1-00:00:00 hai005[franz.srambical@hai005.haicore.berlin:~/jafar/data] $ scancel 29664[franz.srambical@hai005.haicore.berlin:~/jafar/data] $ squeue  JOBID       USER PARTITION NODES  CPUS ST         SUBMIT_TIME          START_TIME     TIME TIME_LIMIT NODELIST(REASON)  29666 franz.sram interacti     1   128  R 2025-09-29T20:08:03 2025-09-29T20:08:03     7:25 1-00:00:00 hai005  29653   xiao.liu interacti     1    64  R 2025-09-29T15:54:10 2025-09-29T15:54:10  4:21:18   23:59:00 hai002  29652   xiao.liu interacti     1    64  R 2025-09-29T15:53:48 2025-09-29T15:53:48  4:21:40   23:59:00 hai006  29651   xiao.liu interacti     1    64  R 2025-09-29T15:53:13 2025-09-29T15:53:13  4:22:15   23:59:00 hai007  29665 johannmich  standard     1    48  R 2025-09-29T19:51:12 2025-09-29T19:51:12    24:16    8:00:00 hai003  29624 alfred.ngu  standard     1    16  R 2025-09-29T17:00:52 2025-09-29T17:02:56  3:12:32 1-00:00:00 hai003  29660 alfred.ngu  standard     1    16  R 2025-09-29T16:14:02 2025-09-29T16:14:18  4:01:10 1-00:00:00 hai001  29659 alfred.ngu  standard     1    16  R 2025-09-29T16:13:44 2025-09-29T16:13:55  4:01:33 1-00:00:00 hai008  29658 alfred.ngu  standard     1    16  R 2025-09-29T16:08:31 2025-09-29T16:08:31  4:06:57 1-00:00:00 hai001  29657 alfred.ngu  standard     1    16  R 2025-09-29T16:07:47 2025-09-29T16:07:47  4:07:41 1-00:00:00 hai005[franz.srambical@hai005.haicore.berlin:~/jafar/data] $ source /home/franz.srambical/jafar/data/.venv/bin/activate(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ squeue  JOBID       USER PARTITION NODES  CPUS ST         SUBMIT_TIME          START_TIME     TIME TIME_LIMIT NODELIST(REASON)  29666 franz.sram interacti     1   128  R 2025-09-29T20:08:03 2025-09-29T20:08:03  1:13:15 1-00:00:00 hai005  29653   xiao.liu interacti     1    64  R 2025-09-29T15:54:10 2025-09-29T15:54:10  5:27:08   23:59:00 hai002  29652   xiao.liu interacti     1    64  R 2025-09-29T15:53:48 2025-09-29T15:53:48  5:27:30   23:59:00 hai006  29651   xiao.liu interacti     1    64  R 2025-09-29T15:53:13 2025-09-29T15:53:13  5:28:05   23:59:00 hai007  29669 alfred.ngu  standard     1    10  R 2025-09-29T20:56:04 2025-09-29T20:56:04    25:14 1-00:00:00 hai008  29668 alfred.ngu  standard     1    10  R 2025-09-29T20:53:33 2025-09-29T20:53:33    27:45 1-00:00:00 hai004  29667 alfred.ngu  standard     1    10  R 2025-09-29T20:52:44 2025-09-29T20:52:45    28:33 1-00:00:00 hai004  29665 johannmich  standard     1    48  R 2025-09-29T19:51:12 2025-09-29T19:51:12  1:30:06    8:00:00 hai003  29624 alfred.ngu  standard     1    16  R 2025-09-29T17:00:52 2025-09-29T17:02:56  4:18:22 1-00:00:00 hai003  29660 alfred.ngu  standard     1    16  R 2025-09-29T16:14:02 2025-09-29T16:14:18  5:07:00 1-00:00:00 hai001  29659 alfred.ngu  standard     1    16  R 2025-09-29T16:13:44 2025-09-29T16:13:55  5:07:23 1-00:00:00 hai008  29658 alfred.ngu  standard     1    16  R 2025-09-29T16:08:31 2025-09-29T16:08:31  5:12:47 1-00:00:00 hai001  29657 alfred.ngu  standard     1    16  R 2025-09-29T16:07:47 2025-09-29T16:07:47  5:13:31 1-00:00:00 hai005(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ nvidia-smi Mon Sep 29 22:19:47 2025       +-----------------------------------------------------------------------------------------+| NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     ||-----------------------------------------+------------------------+----------------------+| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC || Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. ||                                         |                        |               MIG M. ||=========================================+========================+======================||   0  NVIDIA H100 80GB HBM3          On  |   00000000:3B:00.0 Off |                    0 || N/A   37C    P0             71W /  700W |       1MiB /  81559MiB |      0%      Default ||                                         |                        |             Disabled |+-----------------------------------------+------------------------+----------------------+                                                                                         +-----------------------------------------------------------------------------------------+| Processes:                                                                              ||  GPU   GI   CI              PID   Type   Process name                        GPU Memory ||        ID   ID                                                               Usage      ||=========================================================================================||  No running processes found                                                             |+-----------------------------------------------------------------------------------------+(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ uv run jasmine_data/atari/visualize_array_record.py --input data/atari_episodes/train/data_0041.array_record --output_dir=test_output Loaded record with sequence_length=160, actions_present=True/fast/home/franz.srambical/jafar/data/jasmine_data/atari/visualize_array_record.py:111: DeprecationWarning: 'mode' parameter is deprecated and will be removed in Pillow 13 (2026-10-15)  img = Image.fromarray(img_np_rgb, mode="RGB")Saved GIF to test_output/sequence.gif with 160 frames (H=84, W=84, C=3, fps=10).(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ uv run python jasmine_data/atari/generate_atari_dataset.py --output_dir=test_outputGym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.2025-09-29 22:20:39.321760: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.2025-09-29 22:20:40.506333: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags./fast/home/franz.srambical/jafar/data/.venv/lib/python3.10/site-packages/tensorflow_probability/python/__init__.py:57: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.  if (distutils.version.LooseVersion(tf.__version__) </fast/home/franz.srambical/jafar/data/.venv/lib/python3.10/site-packages/tensorflow_probability/python/internal/backend/jax/ops.py:681: DeprecationWarning: jax.interpreters.xla.pytype_aval_mappings is deprecated.  jax.interpreters.xla.pytype_aval_mappings[onp.ndarray])Traceback (most recent call last):  File "/fast/home/franz.srambical/jafar/data/jasmine_data/atari/generate_atari_dataset.py", line 37, in <module>    assert gin_files, "No gin files provided"(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ uv run python jasmine_data/atari/generate_atari_dataset.py --output_dir=test_output --gin_files=jasmine_data/atari/pong_no_sticky_actions.ginGym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.2025-09-29 22:24:58.105373: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.2025-09-29 22:24:58.160201: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags./fast/home/franz.srambical/jafar/data/.venv/lib/python3.10/site-packages/tensorflow_probability/python/__init__.py:57: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.  if (distutils.version.LooseVersion(tf.__version__) </fast/home/franz.srambical/jafar/data/.venv/lib/python3.10/site-packages/tensorflow_probability/python/internal/backend/jax/ops.py:681: DeprecationWarning: jax.interpreters.xla.pytype_aval_mappings is deprecated.  jax.interpreters.xla.pytype_aval_mappings[onp.ndarray])A.L.E: Arcade Learning Environment (version 0.11.2+ecc1138)[Powered by Stella]> /fast/home/franz.srambical/jafar/data/jasmine_data/atari/generate_atari_dataset.py(100)<module>()-> if frame.ndim == 2:(Pdb) framearray([[[ 64],        [ 64],        [ 64],        ...,        [107],        [107],        [107]],       [[107],        [107],        [107],        ...,        [107],        [107],        [107]],       [[107],        [107],        [107],        ...,        [107],        [107],        [107]],       ...,       [[ 74],        [ 74],        [ 74],        ...,        [ 74],        [ 74],        [ 74]],       [[ 74],        [ 74],        [ 74],        ...,        [ 74],        [ 74],        [ 74]],       [[ 74],        [ 74],        [ 74],        ...,        [ 74],        [ 74],        [ 74]]], dtype=uint8)(Pdb) frame.shape(84, 84, 1)(Pdb) frame.ndim3(Pdb) action.shape()(Pdb) quit()Traceback (most recent call last):  File "/fast/home/franz.srambical/jafar/data/jasmine_data/atari/generate_atari_dataset.py", line 100, in <module>    f"Frame has {frame.ndim} dimensions, expected a trailing singleton dimension"  File "/fast/home/franz.srambical/jafar/data/jasmine_data/atari/generate_atari_dataset.py", line 100, in <module>    f"Frame has {frame.ndim} dimensions, expected a trailing singleton dimension"  File "/home/franz.srambical/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/bdb.py", line 90, in trace_dispatch    return self.dispatch_line(frame)  File "/home/franz.srambical/.local/share/uv/python/cpython-3.10.18-linux-x86_64-gnu/lib/python3.10/bdb.py", line 115, in dispatch_line    if self.quitting: raise BdbQuitbdb.BdbQuit(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ cd ..(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar] $ cd slurm/jobs/franz/berlin/atari/(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/slurm/jobs/franz/berlin/atari] $ mv atari_breakout_* legmv: target 'leg' is not a directory(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/slurm/jobs/franz/berlin/atari] $ mv atari_breakout_* legacy_cleanrl_data_generation/(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/slurm/jobs/franz/berlin/atari] $ lsdynamics  lam  legacy_cleanrl_data_generation  tokenizer(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/slurm/jobs/franz/berlin/atari] $ squeue  JOBID       USER PARTITION NODES  CPUS ST         SUBMIT_TIME          START_TIME     TIME TIME_LIMIT NODELIST(REASON)  29666 franz.sram interacti     1   128  R 2025-09-29T20:08:03 2025-09-29T20:08:03  2:32:24 1-00:00:00 hai005  29653   xiao.liu interacti     1    64  R 2025-09-29T15:54:10 2025-09-29T15:54:10  6:46:17   23:59:00 hai002  29652   xiao.liu interacti     1    64  R 2025-09-29T15:53:48 2025-09-29T15:53:48  6:46:39   23:59:00 hai006  29651   xiao.liu interacti     1    64  R 2025-09-29T15:53:13 2025-09-29T15:53:13  6:47:14   23:59:00 hai007  29670 alfred.ngu  standard     1    10  R 2025-09-29T22:07:04 2025-09-29T22:07:04    33:23 1-00:00:00 hai004  29669 alfred.ngu  standard     1    10  R 2025-09-29T20:56:04 2025-09-29T20:56:04  1:44:23 1-00:00:00 hai008  29668 alfred.ngu  standard     1    10  R 2025-09-29T20:53:33 2025-09-29T20:53:33  1:46:54 1-00:00:00 hai004  29667 alfred.ngu  standard     1    10  R 2025-09-29T20:52:44 2025-09-29T20:52:45  1:47:42 1-00:00:00 hai004  29665 johannmich  standard     1    48  R 2025-09-29T19:51:12 2025-09-29T19:51:12  2:49:15    8:00:00 hai003  29624 alfred.ngu  standard     1    16  R 2025-09-29T17:00:52 2025-09-29T17:02:56  5:37:31 1-00:00:00 hai003  29660 alfred.ngu  standard     1    16  R 2025-09-29T16:14:02 2025-09-29T16:14:18  6:26:09 1-00:00:00 hai001  29659 alfred.ngu  standard     1    16  R 2025-09-29T16:13:44 2025-09-29T16:13:55  6:26:32 1-00:00:00 hai008  29658 alfred.ngu  standard     1    16  R 2025-09-29T16:08:31 2025-09-29T16:08:31  6:31:56 1-00:00:00 hai001  29657 alfred.ngu  standard     1    16  R 2025-09-29T16:07:47 2025-09-29T16:07:47  6:32:40 1-00:00:00 hai005(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/slurm/jobs/franz/berlin/atari] $ cd(jasmine-data) [franz.srambical@hai005.haicore.berlin:~] $ cd jafar/data(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ cd ..(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar] $ sbatch slurm/jobs/franz/berlin/atari/data_generation/atari_breakout_no_sticky_actions.shSubmitted batch job 29671(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar] $ sbatch slurm/jobs/franz/berlin/atari/data_generation/atari_pong_no_sticky_actions.shSubmitted batch job 29672(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar] $ squeue  JOBID       USER PARTITION NODES  CPUS ST         SUBMIT_TIME          START_TIME     TIME TIME_LIMIT NODELIST(REASON)  29666 franz.sram interacti     1   128  R 2025-09-29T20:08:03 2025-09-29T20:08:03  2:43:40 1-00:00:00 hai005  29653   xiao.liu interacti     1    64  R 2025-09-29T15:54:10 2025-09-29T15:54:10  6:57:33   23:59:00 hai002  29652   xiao.liu interacti     1    64  R 2025-09-29T15:53:48 2025-09-29T15:53:48  6:57:55   23:59:00 hai006  29651   xiao.liu interacti     1    64  R 2025-09-29T15:53:13 2025-09-29T15:53:13  6:58:30   23:59:00 hai007  29672 franz.sram  standard     1   128  R 2025-09-29T22:50:28 2025-09-29T22:50:37     1:06 1-00:00:00 hai004  29671 franz.sram  standard     1   128  R 2025-09-29T22:50:22 2025-09-29T22:50:22     1:21 1-00:00:00 hai008  29670 alfred.ngu  standard     1    10  R 2025-09-29T22:07:04 2025-09-29T22:07:04    44:39 1-00:00:00 hai004  29669 alfred.ngu  standard     1    10  R 2025-09-29T20:56:04 2025-09-29T20:56:04  1:55:39 1-00:00:00 hai008  29668 alfred.ngu  standard     1    10  R 2025-09-29T20:53:33 2025-09-29T20:53:33  1:58:10 1-00:00:00 hai004  29667 alfred.ngu  standard     1    10  R 2025-09-29T20:52:44 2025-09-29T20:52:45  1:58:58 1-00:00:00 hai004  29665 johannmich  standard     1    48  R 2025-09-29T19:51:12 2025-09-29T19:51:12  3:00:31    8:00:00 hai003  29624 alfred.ngu  standard     1    16  R 2025-09-29T17:00:52 2025-09-29T17:02:56  5:48:47 1-00:00:00 hai003  29660 alfred.ngu  standard     1    16  R 2025-09-29T16:14:02 2025-09-29T16:14:18  6:37:25 1-00:00:00 hai001  29659 alfred.ngu  standard     1    16  R 2025-09-29T16:13:44 2025-09-29T16:13:55  6:37:48 1-00:00:00 hai008  29658 alfred.ngu  standard     1    16  R 2025-09-29T16:08:31 2025-09-29T16:08:31  6:43:12 1-00:00:00 hai001  29657 alfred.ngu  standard     1    16  R 2025-09-29T16:07:47 2025-09-29T16:07:47  6:43:56 1-00:00:00 hai005(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar] $ squeue  JOBID       USER PARTITION NODES  CPUS ST         SUBMIT_TIME          START_TIME     TIME TIME_LIMIT NODELIST(REASON)  29666 franz.sram interacti     1   128  R 2025-09-29T20:08:03 2025-09-29T20:08:03  2:43:43 1-00:00:00 hai005  29653   xiao.liu interacti     1    64  R 2025-09-29T15:54:10 2025-09-29T15:54:10  6:57:36   23:59:00 hai002  29652   xiao.liu interacti     1    64  R 2025-09-29T15:53:48 2025-09-29T15:53:48  6:57:58   23:59:00 hai006  29651   xiao.liu interacti     1    64  R 2025-09-29T15:53:13 2025-09-29T15:53:13  6:58:33   23:59:00 hai007  29672 franz.sram  standard     1   128  R 2025-09-29T22:50:28 2025-09-29T22:50:37     1:09 1-00:00:00 hai004  29671 franz.sram  standard     1   128  R 2025-09-29T22:50:22 2025-09-29T22:50:22     1:24 1-00:00:00 hai008  29670 alfred.ngu  standard     1    10  R 2025-09-29T22:07:04 2025-09-29T22:07:04    44:42 1-00:00:00 hai004  29669 alfred.ngu  standard     1    10  R 2025-09-29T20:56:04 2025-09-29T20:56:04  1:55:42 1-00:00:00 hai008  29668 alfred.ngu  standard     1    10  R 2025-09-29T20:53:33 2025-09-29T20:53:33  1:58:13 1-00:00:00 hai004  29667 alfred.ngu  standard     1    10  R 2025-09-29T20:52:44 2025-09-29T20:52:45  1:59:01 1-00:00:00 hai004  29665 johannmich  standard     1    48  R 2025-09-29T19:51:12 2025-09-29T19:51:12  3:00:34    8:00:00 hai003  29624 alfred.ngu  standard     1    16  R 2025-09-29T17:00:52 2025-09-29T17:02:56  5:48:50 1-00:00:00 hai003  29660 alfred.ngu  standard     1    16  R 2025-09-29T16:14:02 2025-09-29T16:14:18  6:37:28 1-00:00:00 hai001  29659 alfred.ngu  standard     1    16  R 2025-09-29T16:13:44 2025-09-29T16:13:55  6:37:51 1-00:00:00 hai008  29658 alfred.ngu  standard     1    16  R 2025-09-29T16:08:31 2025-09-29T16:08:31  6:43:15 1-00:00:00 hai001  29657 alfred.ngu  standard     1    16  R 2025-09-29T16:07:47 2025-09-29T16:07:47  6:43:59 1-00:00:00 hai005(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar] $ squeue  JOBID       USER PARTITION NODES  CPUS ST         SUBMIT_TIME          START_TIME     TIME TIME_LIMIT NODELIST(REASON)  29666 franz.sram interacti     1   128  R 2025-09-29T20:08:03 2025-09-29T20:08:03  2:46:32 1-00:00:00 hai005  29653   xiao.liu interacti     1    64  R 2025-09-29T15:54:10 2025-09-29T15:54:10  7:00:25   23:59:00 hai002  29652   xiao.liu interacti     1    64  R 2025-09-29T15:53:48 2025-09-29T15:53:48  7:00:47   23:59:00 hai006  29651   xiao.liu interacti     1    64  R 2025-09-29T15:53:13 2025-09-29T15:53:13  7:01:22   23:59:00 hai007  29672 franz.sram  standard     1   128  R 2025-09-29T22:50:28 2025-09-29T22:50:37     3:58 1-00:00:00 hai004  29671 franz.sram  standard     1   128  R 2025-09-29T22:50:22 2025-09-29T22:50:22     4:13 1-00:00:00 hai008  29670 alfred.ngu  standard     1    10  R 2025-09-29T22:07:04 2025-09-29T22:07:04    47:31 1-00:00:00 hai004  29669 alfred.ngu  standard     1    10  R 2025-09-29T20:56:04 2025-09-29T20:56:04  1:58:31 1-00:00:00 hai008  29668 alfred.ngu  standard     1    10  R 2025-09-29T20:53:33 2025-09-29T20:53:33  2:01:02 1-00:00:00 hai004  29667 alfred.ngu  standard     1    10  R 2025-09-29T20:52:44 2025-09-29T20:52:45  2:01:50 1-00:00:00 hai004  29665 johannmich  standard     1    48  R 2025-09-29T19:51:12 2025-09-29T19:51:12  3:03:23    8:00:00 hai003  29624 alfred.ngu  standard     1    16  R 2025-09-29T17:00:52 2025-09-29T17:02:56  5:51:39 1-00:00:00 hai003  29660 alfred.ngu  standard     1    16  R 2025-09-29T16:14:02 2025-09-29T16:14:18  6:40:17 1-00:00:00 hai001  29659 alfred.ngu  standard     1    16  R 2025-09-29T16:13:44 2025-09-29T16:13:55  6:40:40 1-00:00:00 hai008  29658 alfred.ngu  standard     1    16  R 2025-09-29T16:08:31 2025-09-29T16:08:31  6:46:04 1-00:00:00 hai001  29657 alfred.ngu  standard     1    16  R 2025-09-29T16:07:47 2025-09-29T16:07:47  6:46:48 1-00:00:00 hai005(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar] $ squeue  JOBID       USER PARTITION NODES  CPUS ST         SUBMIT_TIME          START_TIME     TIME TIME_LIMIT NODELIST(REASON)  29666 franz.sram interacti     1   128  R 2025-09-29T20:08:03 2025-09-29T20:08:03  2:46:33 1-00:00:00 hai005  29653   xiao.liu interacti     1    64  R 2025-09-29T15:54:10 2025-09-29T15:54:10  7:00:26   23:59:00 hai002  29652   xiao.liu interacti     1    64  R 2025-09-29T15:53:48 2025-09-29T15:53:48  7:00:48   23:59:00 hai006  29651   xiao.liu interacti     1    64  R 2025-09-29T15:53:13 2025-09-29T15:53:13  7:01:23   23:59:00 hai007  29672 franz.sram  standard     1   128  R 2025-09-29T22:50:28 2025-09-29T22:50:37     3:59 1-00:00:00 hai004  29671 franz.sram  standard     1   128  R 2025-09-29T22:50:22 2025-09-29T22:50:22     4:14 1-00:00:00 hai008  29670 alfred.ngu  standard     1    10  R 2025-09-29T22:07:04 2025-09-29T22:07:04    47:32 1-00:00:00 hai004  29669 alfred.ngu  standard     1    10  R 2025-09-29T20:56:04 2025-09-29T20:56:04  1:58:32 1-00:00:00 hai008  29668 alfred.ngu  standard     1    10  R 2025-09-29T20:53:33 2025-09-29T20:53:33  2:01:03 1-00:00:00 hai004  29667 alfred.ngu  standard     1    10  R 2025-09-29T20:52:44 2025-09-29T20:52:45  2:01:51 1-00:00:00 hai004  29665 johannmich  standard     1    48  R 2025-09-29T19:51:12 2025-09-29T19:51:12  3:03:24    8:00:00 hai003  29624 alfred.ngu  standard     1    16  R 2025-09-29T17:00:52 2025-09-29T17:02:56  5:51:40 1-00:00:00 hai003  29660 alfred.ngu  standard     1    16  R 2025-09-29T16:14:02 2025-09-29T16:14:18  6:40:18 1-00:00:00 hai001  29659 alfred.ngu  standard     1    16  R 2025-09-29T16:13:44 2025-09-29T16:13:55  6:40:41 1-00:00:00 hai008  29658 alfred.ngu  standard     1    16  R 2025-09-29T16:08:31 2025-09-29T16:08:31  6:46:05 1-00:00:00 hai001  29657 alfred.ngu  standard     1    16  R 2025-09-29T16:07:47 2025-09-29T16:07:47  6:46:49 1-00:00:00 hai005(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar] $ cd data(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ ls__pycache__  jasmine_data  pyproject.toml  runs  uv.lock(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ uv run jasmine_data/atari/generate_atari_dataset.py --output_dir test)bash: syntax error near unexpected token `)'(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ uv run jasmine_data/atari/generate_atari_dataset.py --output_dir test_output --gin_files=jasmine_data/atari/rainbow_no_sticky_actions.gin --gin_bindings="atari_lib.create_atari_environment.game_name = 'Breakout'"Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.2025-09-29 22:57:51.921501: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.2025-09-29 22:57:51.976174: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags./fast/home/franz.srambical/jafar/data/.venv/lib/python3.10/site-packages/tensorflow_probability/python/__init__.py:57: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.  if (distutils.version.LooseVersion(tf.__version__) </fast/home/franz.srambical/jafar/data/.venv/lib/python3.10/site-packages/tensorflow_probability/python/internal/backend/jax/ops.py:681: DeprecationWarning: jax.interpreters.xla.pytype_aval_mappings is deprecated.  jax.interpreters.xla.pytype_aval_mappings[onp.ndarray])A.L.E: Arcade Learning Environment (version 0.11.2+ecc1138)[Powered by Stella]Created test_output/train/data_0000.array_record with 100 video chunks2025-09-29 22:58:41.744087: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.2025-09-29 22:58:41.744119: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.2025-09-29 22:58:41.744186: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.2025-09-29 22:58:41.744199: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.Created test_output/train/data_0001.array_record with 100 video chunks^CTraceback (most recent call last):  File "/fast/home/franz.srambical/jafar/data/jasmine_data/atari/generate_atari_dataset.py", line 112, in <module>    action = agent.step(float(reward), next_obs)  File "/fast/home/franz.srambical/jafar/third_party/dopamine/dopamine/jax/agents/rainbow/rainbow_agent.py", line 387, in step    self._record_observation(observation)  File "/fast/home/franz.srambical/jafar/third_party/dopamine/dopamine/jax/agents/dqn/dqn_agent.py", line 457, in _record_observation    self.state = onp.roll(self.state, -1, axis=-1)  File "/fast/home/franz.srambical/jafar/data/.venv/lib/python3.10/site-packages/numpy/core/numeric.py", line 1231, in roll    result[res_index] = a[arr_index]KeyboardInterrupt(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ ^C(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ ^C(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ uv run --input=test_output/train_data_ --output_dir=test_viserror: unexpected argument '--input' foundUsage: uv run [OPTIONS] [COMMAND]For more information, try '--help'.(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ (jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ (jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ uv run jasmine_data/atari/visualize_array_record.py --input=test_output/train/data_0001.array_record --output_dir=test_visLoaded record with sequence_length=24, actions_present=True/fast/home/franz.srambical/jafar/data/jasmine_data/atari/visualize_array_record.py:111: DeprecationWarning: 'mode' parameter is deprecated and will be removed in Pillow 13 (2026-10-15)  img = Image.fromarray(img_np_rgb, mode="RGB")Saved GIF to test_vis/sequence.gif with 24 frames (H=84, W=84, C=3, fps=10).(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ ls test_vis/actions.txt   sequence.gif  (jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ ls__pycache__  jasmine_data  pyproject.toml  runs  test_output  test_vis  uv.lock(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ rm -r test_output/(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ uv run jasmine_data/atari/generate_atari_dataset.py --output_dir test_output --gin_files=jasmine_data/atari/rainbow_no_sticky_actions.gin --gin_bindings="atari_lib.create_atari_environment.game_name = 'Boxing'"Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.2025-09-29 23:04:32.754532: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.2025-09-29 23:04:32.809014: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX512_FP16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags./fast/home/franz.srambical/jafar/data/.venv/lib/python3.10/site-packages/tensorflow_probability/python/__init__.py:57: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.  if (distutils.version.LooseVersion(tf.__version__) </fast/home/franz.srambical/jafar/data/.venv/lib/python3.10/site-packages/tensorflow_probability/python/internal/backend/jax/ops.py:681: DeprecationWarning: jax.interpreters.xla.pytype_aval_mappings is deprecated.  jax.interpreters.xla.pytype_aval_mappings[onp.ndarray])A.L.E: Arcade Learning Environment (version 0.11.2+ecc1138)[Powered by Stella]Created test_output/train/data_0000.array_record with 100 video chunks2025-09-29 23:05:24.148723: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.2025-09-29 23:05:24.148762: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.^C2025-09-29 23:05:24.234350: F external/xla/xla/service/gpu/autotuning/gemm_fusion_autotuner.cc:1136] Non-OK-status: executable.status()Status: INTERNAL: ptxas exited with non-zero error code 2, output:  - Failure occured when compiling fusion gemm_fusion_dot.413 with config 'reference (cublas)'Fused HLO computation:%gemm_fusion_dot.413_computation (parameter_0: f32[32,11,11,64], parameter_1: f32[7744,512]) -> f32[32,512] {  %parameter_1 = f32[7744,512]{1,0} parameter(1)  %parameter_0 = f32[32,11,11,64]{3,2,1,0} parameter(0)  %constant.34 = f32[] constant(0)  %broadcast.165 = f32[32,11,11,64]{3,2,1,0} broadcast(%constant.34), dimensions={}  %maximum.29 = f32[32,11,11,64]{3,2,1,0} maximum(%parameter_0, %broadcast.165), metadata={op_name="jit(train)/jit(main)/vmap(RainbowNetwork)/jit(relu)/max" source_file="/fast/home/franz.srambical/jafar/third_party/dopamine/dopamine/jax/networks.py" source_line=319}  %bitcast.298 = f32[32,7744]{1,0} bitcast(%maximum.29), metadata={op_name="jit(train)/jit(main)/vmap(RainbowNetwork)/jit(relu)/max" source_file="/fast/home/franz.srambical/jafar/third_party/dopamine/dopamine/jax/networks.py" source_line=319}  %dot.10 = f32[512,32]{0,1} dot(%parameter_1, %bitcast.298), lhs_contracting_dims={0}, rhs_contracting_dims={1}, metadata={op_name="jit(train)/jit(main)/jvp(vmap(RainbowNetwork))/Dense_0/dot_general" source_file="/fast/home/franz.srambical/jafar/data/.venv/lib/python3.10/site-packages/flax/linen/linear.py" source_line=288}  ROOT %bitcast.304 = f32[32,512]{1,0} bitcast(%dot.10), metadata={op_name="jit(train)/jit(main)/jvp(vmap(RainbowNetwork))/Dense_0/dot_general" source_file="/fast/home/franz.srambical/jafar/data/.venv/lib/python3.10/site-packages/flax/linen/linear.py" source_line=288}}2025-09-29 23:05:24.234515: F external/xla/xla/service/gpu/autotuning/gemm_fusion_autotuner.cc:1136] Non-OK-status: executable.status()Status: INTERNAL: ptxas exited with non-zero error code 2, output:  - Failure occured when compiling fusion gemm_fusion_dot with config 'reference (cublas)'Fused HLO computation:%gemm_fusion_dot_computation (parameter_0.6: f32[32,11,11,64], parameter_1.6: f32[32,512]) -> f32[7744,512] {  %parameter_0.6 = f32[32,11,11,64]{3,2,1,0} parameter(0)  %constant.37 = f32[] constant(0)  %broadcast.175 = f32[32,11,11,64]{3,2,1,0} broadcast(%constant.37), dimensions={}  %maximum.32 = f32[32,11,11,64]{3,2,1,0} maximum(%parameter_0.6, %broadcast.175), metadata={op_name="jit(train)/jit(main)/vmap(RainbowNetwork)/jit(relu)/max" source_file="/fast/home/franz.srambical/jafar/third_party/dopamine/dopamine/jax/networks.py" source_line=319}  %bitcast.302 = f32[32,7744]{1,0} bitcast(%maximum.32), metadata={op_name="jit(train)/jit(main)/vmap(RainbowNetwork)/jit(relu)/max" source_file="/fast/home/franz.srambical/jafar/third_party/dopamine/dopamine/jax/networks.py" source_line=319}  %parameter_1.6 = f32[32,512]{1,0} parameter(1)  ROOT %dot.8 = f32[7744,512]{1,0} dot(%bitcast.302, %parameter_1.6), lhs_contracting_dims={0}, rhs_contracting_dims={0}, metadata={op_name="jit(train)/jit(main)/transpose(jvp(vmap(RainbowNetwork)))/Dense_0/transpose" source_file="/fast/home/franz.srambical/jafar/data/.venv/lib/python3.10/site-packages/flax/linen/linear.py" source_line=288}}^C^C^C^C^C(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafauv run jasmine_data/atari/visualize_array_record.py --input=test_output/train/data_0000.array_record --output_dir=test_visLoaded record with sequence_length=160, actions_present=True/fast/home/franz.srambical/jafar/data/jasmine_data/atari/visualize_array_record.py:111: DeprecationWarning: 'mode' parameter is deprecated and will be removed in Pillow 13 (2026-10-15)  img = Image.fromarray(img_np_rgb, mode="RGB")Saved GIF to test_vis/sequence.gif with 160 frames (H=84, W=84, C=3, fps=10).(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ rm -r test_output/(jasmine-data) [franz.srambical@hai005.haicore.berlin:~/jafar/data] $ exitexitsalloc: Relinquishing job allocation 29666salloc: Job allocation 29666 has been revoked.