p-doom/crowd-code-dataset-0.2 huggingface dataset:
[terminal_command] TERMINAL ()
(jafar) [franz.srambical@hai001.haicore.berlin:~/jafar] $ squeue --me  JOBID       USER PARTITION NODES  CPUS ST         SUBMIT_TIME          START_TIME     TIME TIME_LIMIT NODELIST(REASON)  13981 franz.sram interacti     1     1  R 2025-07-26T20:21:50 2025-07-26T20:21:50    53:59 1-00:00:00 hai001  13980 franz.sram interacti     1     1  R 2025-07-26T18:22:32 2025-07-26T18:22:32  2:53:17 1-00:00:00 hai006(jafar) [franz.srambical@hai001.haicore.berlin:~/jafar] $ scancel --mesalloc: Job allocation 13981 has been revoked.                                              (jafar) [franz.srambical@hai001.haicore.berlin:~/jafar] $ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.[2025-07-26T21:15:50.653] error: *** STEP 13981.interactive ON hai001 CANCELLED AT 2025-07-26T21:15:50 DUE to SIGNAL Terminated ***^C(jafar) [franz.srambical@hai001.haicore.berlin:~/jafar] $ squeue --me  JOBID       USER PARTITION NODES  CPUS ST         SUBMIT_TIME          START_TIME     TIME TIME_LIMIT NODELIST(REASON)  13981 franz.sram interacti     1     1 CG 2025-07-26T20:21:50 2025-07-26T20:21:50    54:00 1-00:00:00 hai001  13980 franz.sram interacti     1     1 CG 2025-07-26T18:22:32 2025-07-26T18:22:32  2:53:18 1-00:00:00 hai006(jafar) [franz.srambical@hai001.haicore.berlin:~/jafar] $ srun: error: hai001: task 0: Killed
[terminal_command] TERMINAL ()
ine 167, in _autoreg_sample    generated_vid = _sampling_fn(jasmine, batch)  File "/fast/home/franz.srambical/jafar/sample.py", line 154, in _sampling_fn    return model.sample_causal(  File "/fast/home/franz.srambical/jafar/jasmine.py", line 378, in sample_causal    final_carry, _ = jax.lax.scan(  File "/fast/home/franz.srambical/jafar/jasmine.py", line 351, in token_step_fn[franz.srambical@hai006.haicore.berlin:~/jafar] $ bash experiments/sample.sh /fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py:1256: UserWarning: Sharding info not provided when restoring. Populating sharding info from sharding file. Please note restoration time will be slightly increased due to reading from file. Note also that this option is unsafe when restoring on a different topology than the checkpoint was saved with.  warnings.warn(2025-07-26 20:37:43.714633: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.[franz.srambical@hai006.haicore.berlin:~/jafar] $ bash experiments/sample.sh /fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py:1256: UserWarning: Sharding info not provided when restoring. Populating sharding info from sharding file. Please note restoration time will be slightly increased due to reading from file. Note also that this option is unsafe when restoring on a different topology than the checkpoint was saved with.  warnings.warn(2025-07-26 20:40:35.591428: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.jax.errors.SimplifiedTraceback: For simplicity, JAX has removed its internal frames from the traceback of the following exception. Set JAX_TRACEBACK_FILTERING=off to include these.The above exception was the direct cause of the following exception:Traceback (most recent call last):  File "/fast/home/franz.srambical/jafar/sample.py", line 225, in <module>(jafar) [franz.srambical@hai006.haicore.berlin:~/jafar] $ salloc --gpus=1 --ntasks-per-node=1 --cpus-per-task=1 --mem=100Gsalloc: Granted job allocation 13982salloc: Nodes hai002 are ready for jobRunning inside SLURM, Job ID 13982.[franz.srambical@hai002.haicore.berlin:~/jafar] $ srun: error: hai006: task 0: Killed