[terminal_command] TERMINAL ()
Per-frame SSIM: [0.7894105  0.8011795  0.7812164  0.7577461  0.728574   0.74053216 0.73998666 0.725984   0.72455955 0.73330045 0.7074413  0.70861423 0.7424905  0.75625813 0.77523166]Per-frame PSNR: [23.77471  23.183779 24.317377 23.290848 21.04387  22.084274 21.25311 21.139128 21.253815 21.054768 21.114222 20.891474 22.124496 22.375664 22.179096]SSIM: 0.7475016713142395PSNR: 22.07204246520996W1001 15:09:04.743067 1107441 pjrt_client.cc:1469] WatchJobStateAsync failed for task goo.gle/debugproto    job_name: "jax_worker": UNAVAILABLE: Cancelling all callsAdditional GRPC error information from remote target coordination_service while calling /tensorflow.CoordinationService/WatchJobState::UNKNOWN:Error received from peer  {grpc_message:"Cancelling all calls", grpc_status:14}[franz.srambical@hai004.haicore.berlin:~/jafar] $ ls gils: cannot access 'gi': No such file or directory[franz.srambical@hai004.haicore.berlin:~/jafar] $ ls -lh gifs/total 3.5M-rw-r--r-- 1 franz.srambical franz.srambical 276K Sep 30 16:01 generation_1759240868.7708697.gif-rw-r--r-- 1 franz.srambical franz.srambical 1.8M Sep 30 17:42 generation_1759246964.255436.gif-rw-r--r-- 1 franz.srambical franz.srambical 301K Oct  1 15:08 generation_1759324138.4886644.gif[franz.srambical@hai004.haicore.berlin:~/jafar] $ bash slurm/dev/franz/berlin/coinrun/sample/maskgit/sample_dynamics_from_fully_trained_tokenizer.sh slurm/dev/franz/berlin/coinrun/sample/maskgit/sample_dynamics_from_fully_trained_tokenizer.sh: line 19: --patch_size=16: command not foundSampling from checkpoint: /fast/project/HFMI_SynergyUnit/jafar_ws/checkpoints/coinrun/dynamics/dynamics_coinrun_mila_submission/29766/fast/home/franz.srambical/jafar/.venv/lib/python3.13/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py:1269: UserWarning: Sharding info not provided when restoring. Populating sharding info from sharding file. Please note restoration time will be slightly increased due to reading from file. Note also that this option is unsafe when restoring on a different topology than the checkpoint was saved with.  warnings.warn(Per-frame SSIM: [0.7336701  0.70509785 0.6653772  0.587384   0.578086   0.5687523 0.5169649  0.48689753 0.45125365 0.42753714 0.4135151  0.38086024 0.36199933 0.36020482 0.39111507]Per-frame PSNR: [23.142338  21.925184  18.687675  17.46246   16.556253  15.815784 14.4365425 13.295241  11.938238  11.266008  10.866945  10.585852 10.548854  10.88524   11.48111  ]SSIM: 0.5085810422897339PSNR: 14.592915534973145W1001 15:12:17.376076 1192731 pjrt_client.cc:1469] WatchJobStateAsync failed for task goo.gle/debugproto    job_name: "jax_worker": UNAVAILABLE: Cancelling all callsAdditional GRPC error information from remote target coordination_service while calling /tensorflow.CoordinationService/WatchJobState::UNKNOWN:Error received from peer  {grpc_message:"Cancelling all calls", grpc_status:14}[franz.srambical@hai004.haicore.berlin:~/jafar] $ bash slurm/dev/franz/berlin/coinrun/sample/maskgit/sample_dynamics_from_fully_trained_tokenizer.sh slurm/dev/franz/berlin/coinrun/sample/maskgit/sample_dynamics_from_fully_trained_tokenizer.sh: line 19: --patch_size=16: command not foundSampling from checkpoint: /fast/project/HFMI_SynergyUnit/jafar_ws/checkpoints/coinrun/dynamics/dynamics_coinrun_mila_submission/29766/fast/home/franz.srambical/jafar/.venv/lib/python3.13/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py:1269: UserWarning: Sharding info not provided when restoring. Populating sharding info from sharding file. Please note restoration time will be slightly increased due to reading from file. Note also that this option is unsafe when restoring on a different topology than the checkpoint was saved with.  warnings.warn(Per-frame SSIM: [0.64136463 0.61766297 0.6466296  0.55556166 0.5162969  0.46530294 0.42861173 0.40834588 0.38667423 0.3559933  0.33581945 0.36400473]Per-frame PSNR: [19.535015 17.944286 18.107334 15.631359 13.560915 12.775366 11.972359 11.608855 11.128835 11.007093 10.982116 11.374056]SSIM: 0.47685569524765015PSNR: 13.802299499511719W1001 15:15:19.043631 1256171 pjrt_client.cc:1469] WatchJobStateAsync failed for task goo.gle/debugonly  job_name: "jax_worker": CANCELLED: CANCELLEDAdditional GRPC error information from remote target coordination_service while calling /tensorflow.CoordinationService/WatchJobState::UNKNOWN:Error received from peer  {grpc_message:"CANCELLED", grpc_status:1} [type.googleapis.com/tensorflow.DerivedStatus=''][franz.srambical@hai004.haicore.berlin:~/jafar] $ bash slurm/dev/franz/berlin/coinrun/sample/maskgit/sample_dynamics_from_fully_trained_tokenizer.sh slurm/dev/franz/berlin/coinrun/sample/maskgit/sample_dynamics_from_fully_trained_tokenizer.sh: line 19: --patch_size=16: command not foundSampling from checkpoint: /fast/project/HFMI_SynergyUnit/jafar_ws/checkpoints/coinrun/dynamics/dynamics_coinrun_mila_submission_patch_size_16_noise_branch/29773/fast/home/franz.srambical/jafar/.venv/lib/python3.13/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py:1269: UserWarning: Sharding info not provided when restoring. Populating sharding info from sharding file. Please note restoration time will be slightly increased due to reading from file. Note also that this option is unsafe when restoring on a different topology than the checkpoint was saved with.  warnings.warn(Per-frame SSIM: [0.8667842  0.8247647  0.8204562  0.79344803 0.79645336 0.7789147 0.784372   0.77086884 0.77146655 0.82577467 0.8720072  0.7734161 ]Per-frame PSNR: [27.080088 25.9544   25.563364 24.374443 23.817513 23.909765 23.384094 23.65361  23.901028 25.247204 25.978325 21.526476]SSIM: 0.8065605163574219PSNR: 24.532527923583984W1001 15:19:30.807462 1362352 pjrt_client.cc:1469] WatchJobStateAsync failed for task goo.gle/debugproto    job_name: "jax_worker": CANCELLED: CANCELLEDAdditional GRPC error information from remote target coordination_service while calling /tensorflow.CoordinationService/WatchJobState::UNKNOWN:Error received from peer  {grpc_status:1, grpc_message:"CANCELLED"} [type.googleapis.com/tensorflow.DerivedStatus=''][franz.srambical@hai004.haicore.berlin:~/jafar] $ sbatch slurm/jobs/franz/berlin/coinrun/submission_debug/coinrun_dynamics_base_patch_size_4_action_prepend.shSubmitted batch job 29804[franz.srambical@hai004.haicore.berlin:~/jafar] $ sbatch slurm/jobs/franz/berlin/coinrun/submission_debug/coinrun_dynamics_base_patch_size_16_action_prepend.shSubmitted batch job 29805[franz.srambical@hai004.haicore.berlin:~/jafar] $ watch squeue[franz.srambical@hai004.haicore.berlin:~/jafar] $ less /fast/project/HFMI_SynergyUnit/jafar_ws/logs/franz/coinrun/dynamics/dynamics_coinrun_mila_submission_29764.log                                                 dynamics_coinrun_mila_submission_patch_size_16_noise_branch_29773.logdynamics_coinrun_mila_submission_29765.log                                                 dynamics_coinrun_mila_submission_patch_size_4_action_prepend_branch_29804.logdynamics_coinrun_mila_submission_29766.log                                                 dynamics_coinrun_mila_submission_patch_size_4_main_alfred_tokenizer_old_29771.logdynamics_coinrun_mila_submission_causal_29762.log                                          dynamics_coinrun_mila_submission_patch_size_4_noise_branch_alfred_tokenizer_old_29772.logdynamics_coinrun_mila_submission_ffn_dim_ablation_29759.log                                train_dynamics_fp32_adam_moments_coinrun_20070.logdynamics_coinrun_mila_submission_gt_actions_29760.log                                      train_dynamics_fp32_layernorm_coinrun_20054.logdynamics_coinrun_mila_submission_no_cotraining_29761.log                                   train_dynamics_fp32_layernorm_coinrun_20058.logdynamics_coinrun_mila_submission_no_flash_attn_29763.log                                   train_dynamics_fp32_layernorm_coinrun_20060.logdynamics_coinrun_mila_submission_patch_size_16_action_prepend_branch_29805.log             train_dynamics_fp32_layernorm_coinrun_20066.log[franz.srambical@hai004.haicore.berlin:~/jafar] $ less /fast/project/HFMI_SynergyUnit/jafar_ws/logs/franz/coinrun/dynamics/dynamics_coinrun_mila_submission_patch_size_4_action_prepend_branch_29804.log [franz.srambical@hai004.haicore.berlin:~/jafar] $ exitexitsalloc: Relinquishing job allocation 29800(jasmine-data) [franz.srambical@hai-login1.haicore.berlin:~/jafar] $ deactivate [franz.srambical@hai-login1.haicore.berlin:~/jafar] $ sbatch slurm/jobs/franz/berlin/coinrun/submission_debug/coinrun_dynamics_base_patch_size_4_action_prepend.shSubmitted batch job 29807[franz.srambical@hai-login1.haicore.berlin:~/jafar] $ sbatch slurm/jobs/franz/berlin/coinrun/submission_debug/coinrun_dynamics_base_patch_size_16_action_prepend.shSubmitted batch job 29808[franz.srambical@hai-login1.haicore.berlin:~/jafar] $ watch squeue