[terminal_command] TERMINAL ()
Step 765, loss: 0.0Step 766, loss: 0.0Step 767, loss: 0.0Step 768, loss: 0.0Step 769, loss: 0.0Step 770, loss: 0.0Step 771, loss: 0.0Step 772, loss: 0.0Step 773, loss: 0.0Step 774, loss: 0.0Step 775, loss: 0.0Step 776, loss: 0.0Step 777, loss: 0.0Step 778, loss: 0.0Step 779, loss: 0.0Step 780, loss: 0.0Step 781, loss: 0.0Step 782, loss: 0.0Step 783, loss: 0.0Step 784, loss: 0.0Step 785, loss: 0.0Step 786, loss: 0.0Step 787, loss: 0.0Step 788, loss: 0.0Step 789, loss: 0.0Step 790, loss: 0.0Step 791, loss: 0.0Step 792, loss: 0.0Step 793, loss: 0.0Step 794, loss: 0.0Step 795, loss: 0.0Step 796, loss: 0.0Step 797, loss: 0.0Step 798, loss: 0.0Step 799, loss: 0.0Step 800, loss: 0.0Step 801, loss: 0.0Step 802, loss: 0.0Step 803, loss: 0.0Step 804, loss: 0.0Step 805, loss: 0.0Step 806, loss: 0.0Step 807, loss: 0.0Step 808, loss: 0.0Step 809, loss: 0.0Step 810, loss: 0.0Step 811, loss: 0.0Step 812, loss: 0.0Step 813, loss: 0.0Step 814, loss: 0.0Step 815, loss: 0.0Step 816, loss: 0.0Step 817, loss: 0.0Step 818, loss: 0.0Step 819, loss: 0.0Step 820, loss: 0.0Step 821, loss: 0.0Step 822, loss: 0.0Step 823, loss: 0.0Step 824, loss: 0.0Step 825, loss: 0.0Step 826, loss: 0.0Step 827, loss: 0.0Step 828, loss: 0.0Step 829, loss: 0.0Step 830, loss: 0.0Step 831, loss: 0.0Step 832, loss: 0.0Step 833, loss: 0.0Step 834, loss: 0.0Step 835, loss: 0.0Step 836, loss: 0.0Step 837, loss: 0.0Step 838, loss: 0.0Step 839, loss: 0.0Step 840, loss: 0.0Step 841, loss: 0.0Step 842, loss: 0.0Step 843, loss: 0.0Step 844, loss: 0.0Step 845, loss: 0.0Step 846, loss: 0.0Step 847, loss: 0.0Step 848, loss: 0.0Step 849, loss: 0.0Step 850, loss: 0.0Step 851, loss: 0.0Step 852, loss: 0.0Step 853, loss: 0.0Step 854, loss: 0.0Step 855, loss: 0.0Step 856, loss: 0.0Step 857, loss: 0.0Step 858, loss: 0.0Step 859, loss: 0.0Step 860, loss: 0.0Step 861, loss: 0.0Step 862, loss: 0.0Step 863, loss: 0.0Step 864, loss: 0.0Step 865, loss: 0.0Step 866, loss: 0.0Step 867, loss: 0.0Step 868, loss: 0.0Step 869, loss: 0.0Step 870, loss: 0.0Step 871, loss: 0.0Step 872, loss: 0.0Step 873, loss: 0.0Step 874, loss: 0.0Step 875, loss: 0.0Step 876, loss: 0.0Step 877, loss: 0.0Step 878, loss: 0.0Step 879, loss: 0.0Step 880, loss: 0.0Step 881, loss: 0.0Step 882, loss: 0.0Step 883, loss: 0.0Step 884, loss: 0.0Step 885, loss: 0.0Step 886, loss: 0.0Step 887, loss: 0.0Step 888, loss: 0.0Step 889, loss: 0.0Step 890, loss: 0.0Step 891, loss: 0.0Step 892, loss: 0.0Step 893, loss: 0.0Step 894, loss: 0.0Step 895, loss: 0.0Step 896, loss: 0.0Step 897, loss: 0.0Step 898, loss: 0.0Step 899, loss: 0.0Step 900, loss: 0.0Step 901, loss: 0.0Step 902, loss: 0.0Step 903, loss: 0.0Step 904, loss: 0.0Step 905, loss: 0.0Step 906, loss: 0.0Step 907, loss: 0.0Step 908, loss: 0.0Step 909, loss: 0.0Step 910, loss: 0.0Step 911, loss: 0.0Step 912, loss: 0.0Step 913, loss: 0.0Step 914, loss: 0.0Step 915, loss: 0.0Step 916, loss: 0.0Step 917, loss: 0.0Step 918, loss: 0.0Step 919, loss: 0.0Step 920, loss: 0.0Step 921, loss: 0.0Step 922, loss: 0.0Step 923, loss: 0.0Step 924, loss: 0.0Step 925, loss: 0.0Step 926, loss: 0.0Step 927, loss: 0.0Step 928, loss: 0.0Step 929, loss: 0.0Step 930, loss: 0.0Step 931, loss: 0.0Step 932, loss: 0.0Step 933, loss: 0.0Step 934, loss: 0.0Step 935, loss: 0.0Step 936, loss: 0.0Step 937, loss: 0.0Step 938, loss: 0.0Step 939, loss: 0.0Step 940, loss: 0.0Step 941, loss: 0.0Step 942, loss: 0.0Step 943, loss: 0.0Step 944, loss: 0.0Step 945, loss: 0.0Step 946, loss: 0.0Step 947, loss: 0.0Step 948, loss: 0.0Step 949, loss: 0.0Step 950, loss: 0.0Step 951, loss: 0.0Step 952, loss: 0.0Step 953, loss: 0.0Step 954, loss: 0.0Step 955, loss: 0.0Step 956, loss: 0.0Step 957, loss: 0.0Step 958, loss: 0.0Step 959, loss: 0.0Step 960, loss: 0.0Step 961, loss: 0.0Step 962, loss: 0.0Step 963, loss: 0.0Step 964, loss: 0.0Step 965, loss: 0.0Step 966, loss: 0.0Step 967, loss: 0.0Step 968, loss: 0.0Step 969, loss: 0.0Step 970, loss: 0.0Step 971, loss: 0.0Step 972, loss: 0.0Step 973, loss: 0.0Step 974, loss: 0.0Step 975, loss: 0.0Step 976, loss: 0.0Step 977, loss: 0.0Step 978, loss: 0.0Step 979, loss: 0.0Step 980, loss: 0.0Step 981, loss: 0.0Step 982, loss: 0.0Step 983, loss: 0.0Step 984, loss: 0.0Step 985, loss: 0.0Step 986, loss: 0.0Step 987, loss: 0.0Step 988, loss: 0.0Step 989, loss: 0.0Step 990, loss: 0.0Step 991, loss: 0.0Step 992, loss: 0.0Step 993, loss: 0.0Step 994, loss: 0.0Step 995, loss: 0.0Step 996, loss: 0.0Step 997, loss: 0.0Step 998, loss: 0.0Step 999, loss: 0.0Step 1000, loss: 0.0Step 1001, loss: 0.0Step 1002, loss: 0.0Step 1003, loss: 0.0Step 1004, loss: 0.0Step 1005, loss: 0.0Step 1006, loss: 0.0Step 1007, loss: 0.0Step 1008, loss: 0.0Step 1009, loss: 0.0Step 1010, loss: 0.0Step 1011, loss: 0.0Step 1012, loss: 0.0Step 1013, loss: 0.0Step 1014, loss: 0.0Step 1015, loss: 0.0Step 1016, loss: 0.0^Csrun: interrupt (one more within 1 sec to abort)srun: StepId=17482.1 task 0: running^Csrun: sending Ctrl-C to StepId=17482.1srun: forcing job terminationsrun: Job step aborted: Waiting up to 32 seconds for job step to finish.[2025-08-12T11:25:38.717] error: *** STEP 17482.1 ON hai004 CANCELLED AT 2025-08-12T11:25:38 DUE to SIGNAL Killed ***^Csrun: sending Ctrl-C to StepId=17482.1srun: job abort in progress^Csrun: sending Ctrl-C to StepId=17482.1srun: job abort in progress^Csrun: sending Ctrl-C to StepId=17482.1^Csrun: sending Ctrl-C to StepId=17482.1^Csrun: sending Ctrl-C to StepId=17482.1^Csrun: sending Ctrl-C to StepId=17482.1^Csrun: sending Ctrl-C to StepId=17482.1[franz.srambical@hai004.haicore.berlin:~/jafar] $ ^C[franz.srambical@hai004.haicore.berlin:~/jafar] $ ^C[franz.srambical@hai004.haicore.berlin:~/jafar] $ bash experiments/dynamics_grain_tok_restore.sh Running on 1 devices.Counting all components: ['dynamics', 'lam', 'tokenizer']Parameter counts:{'dynamics': 26555392, 'lam': 35115232, 'tokenizer': 33750256, 'total': 95420880}WARNING:absl:Metadata file does not exist: /home/franz.srambical/jafar/checkpoints/causal_dynamics_openai_grain_tok_restore/000290/_CHECKPOINT_METADATA/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/orbax/checkpoint/_src/serialization/type_handlers.py:1256: UserWarning: Sharding info not provided when restoring. Populating sharding info from sharding file. Please note restoration time will be slightly increased due to reading from file. Note also that this option is unsafe when restoring on a different topology than the checkpoint was saved with.  warnings.warn(Starting training from step 0...2025-08-12 11:27:44.315452: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registeredWARNING: All log messages before absl::InitializeLog() is called are written to STDERRE0000 00:00:1754990864.329760 2510141 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registeredE0000 00:00:1754990864.334144 2510141 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registeredW0000 00:00:1754990864.345866 2510141 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.W0000 00:00:1754990864.345883 2510141 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.W0000 00:00:1754990864.345885 2510141 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.W0000 00:00:1754990864.345887 2510141 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.2025-08-12 11:27:57.551531: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.2025-08-12 11:27:57.552034: W external/xla/xla/service/gpu/autotuning/dot_search_space.cc:200] All configs were filtered out because none of them sufficiently match the hints. Maybe the hints set does not contain a good representative set of valid configs?Working around this by using the full hints set instead.Step 0, loss: 16.592443466186523Step 1, loss: 2.1201940398896113e-05Step 2, loss: 4.415465468809998e-07Step 3, loss: 3.7130384100692027e-08Step 4, loss: 5.685093285023868e-09[franz.srambical@hai004.haicore.berlin:~/jafar] $ tensorboard --logdir tensorboard/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tensorboard/default.py:30: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.  import pkg_resources^C^CTraceback (most recent call last):  File "/fast/home/franz.srambical/jafar/.venv/bin/tensorboard", line 4, in <module>    from tensorboard.main import run_main  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tensorboard/main.py", line 27, in <module>    from tensorboard import default  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tensorboard/default.py", line 33, in <module>    from tensorboard.plugins.core import core_plugin  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tensorboard/plugins/core/core_plugin.py", line 32, in <module>    from tensorboard.util import grpc_util  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/tensorboard/util/grpc_util.py", line 24, in <module>    import grpc  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/grpc/__init__.py", line 2346, in <module>    from grpc import aio  # pylint: disable=ungrouped-imports  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/grpc/aio/__init__.py", line 44, in <module>    from ._call import AioRpcError  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked  File "<frozen importlib._bootstrap_external>", line 879, in exec_module  File "<frozen importlib._bootstrap_external>", line 975, in get_code  File "<frozen importlib._bootstrap_external>", line 1074, in get_dataKeyboardInterrupt[franz.srambical@hai004.haicore.berlin:~/jafar] $ ^C[franz.srambical@hai004.haicore.berlin:~/jafar] $ python test/test.py Traceback (most recent call last):  File "/fast/home/franz.srambical/jafar/test/test.py", line 1, in <module>    import jaxModuleNotFoundError: No module named 'jax'[franz.srambical@hai004.haicore.berlin:~/jafar] $ source /home/franz.srambical/jafar/.venv/bin/activate(jafar) [franz.srambical@hai004.haicore.berlin:~/jafar] $ python test/test.py 2025-08-12 11:48:08.397707: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registeredWARNING: All log messages before absl::InitializeLog() is called are written to STDERRE0000 00:00:1754992088.414957 3322448 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registeredE0000 00:00:1754992088.420368 3322448 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registeredW0000 00:00:1754992088.434449 3322448 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.W0000 00:00:1754992088.434472 3322448 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.W0000 00:00:1754992088.434474 3322448 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.W0000 00:00:1754992088.434477 3322448 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.(jafar) [franz.srambical@hai004.haicore.berlin:~/jafar] $ NVIDIA_TF32_OVERRIDE=0 python test/test.py 2025-08-12 12:10:35.696958: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registeredWARNING: All log messages before absl::InitializeLog() is called are written to STDERRE0000 00:00:1754993435.714043   13384 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registeredE0000 00:00:1754993435.719420   13384 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registeredW0000 00:00:1754993435.733478   13384 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.W0000 00:00:1754993435.733502   13384 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.W0000 00:00:1754993435.733505   13384 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.W0000 00:00:1754993435.733509   13384 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.(jafar) [franz.srambical@hai004.haicore.berlin:~/jafar] $ exitexitsalloc: Relinquishing job allocation 17482salloc: Job allocation 17482 has been revoked.