[terminal_command] TERMINAL ()
ents/tokenizer_grain_checkpointing.sh t size: 0.4 GB, Host temp size: 0.0 GB.[franz.srambical@hai008.haicore.berlin:~/jafar] $ bash experiments/tokenizer_grain_checkpointing.sh Running on 1 devices.Counting all components: ['decoder', 'encoder', 'vq']Parameter counts:{'decoder': 16858736, 'encoder': 16858752, 'vq': 32768, 'total': 33750256}Total memory size: 8.8 GB, Output size: 0.4 GB, Temp size: 8.3 GB, Argument size: 0.4 GB, Host temp size: 0.0 GB.FLOPs: 4.481e+13, Bytes: 3.947e+11 (367.6 GB), Intensity: 113.5 FLOPs/byteStarting training from step 0...Memstats: After params initialized:        Using (GB) 0.44 / 59.39 (0.740865%) on cuda:0Step 0, loss: 0.25430572032928467^Csrun: interrupt (one more within 1 sec to abort)srun: StepId=23180.2 task 0: running^Csrun: sending Ctrl-C to StepId=23180.2srun: forcing job terminationsrun: Job step aborted: Waiting up to 32 seconds for job step to finish.[2025-08-23T10:14:09.728] error: *** STEP 23180.2 ON hai008 CANCELLED AT 2025-08-23T10:14:09 DUE to SIGNAL Killed ***[franz.srambical@hai008.haicore.berlin:~/jafar] $ bash experiments/tokenizer_grain_checkpointing.sh Running on 1 devices.Counting all components: ['decoder', 'encoder', 'vq']Parameter counts:{'decoder': 16858736, 'encoder': 16858752, 'vq': 32768, 'total': 33750256}Total memory size: 8.8 GB, Output size: 0.4 GB, Temp size: 8.3 GB, Argument size: 0.4 GB, Host temp size: 0.0 GB.FLOPs: 4.481e+13, Bytes: 3.947e+11 (367.6 GB), Intensity: 113.5 FLOPs/byteStarting training from step 0...Memstats: Before train step:        Using (GB) 0.41 / 59.39 (0.690352%) on cuda:0Memstats: After params initialized:        Using (GB) 0.44 / 59.39 (0.740865%) on cuda:0Step 0, loss: 0.25430572032928467Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 1, loss: 0.26582303643226624Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 2, loss: 0.27628234028816223Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 3, loss: 0.2548494338989258Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 4, loss: 0.23908862471580505Saved checkpoint at step 5Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 5, loss: 0.22121138870716095Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 6, loss: 0.21052992343902588Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 7, loss: 0.202899768948555Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 8, loss: 0.20022976398468018Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 9, loss: 0.1893770396709442Saved checkpoint at step 10Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 10, loss: 0.18337929248809814Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 11, loss: 0.1806625872850418Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 12, loss: 0.1746610850095749Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 13, loss: 0.1692516952753067Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 14, loss: 0.16082949936389923Saved checkpoint at step 15Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 15, loss: 0.15853473544120789Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 16, loss: 0.15587542951107025Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 17, loss: 0.1504763513803482Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 18, loss: 0.14223776757717133Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 19, loss: 0.13885925710201263Saved checkpoint at step 20Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 20, loss: 0.1371721774339676Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 21, loss: 0.13666988909244537Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 22, loss: 0.1344180405139923Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 23, loss: 0.13085807859897614Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 24, loss: 0.12745551764965057Saved checkpoint at step 25Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 25, loss: 0.12578138709068298Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 26, loss: 0.12494925409555435Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 27, loss: 0.12306758016347885Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 28, loss: 0.12121306359767914Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 29, loss: 0.12009277939796448Saved checkpoint at step 30Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 30, loss: 0.11934643238782883Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 31, loss: 0.11683395504951477Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 32, loss: 0.11460163444280624Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 33, loss: 0.11413212865591049Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 34, loss: 0.11470935493707657Saved checkpoint at step 35Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 35, loss: 0.11339044570922852Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 36, loss: 0.10993002355098724Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 37, loss: 0.10971338301897049Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 38, loss: 0.11098533868789673Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 39, loss: 0.11039561033248901Saved checkpoint at step 40Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 40, loss: 0.10747850686311722Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 41, loss: 0.10731696337461472Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 42, loss: 0.10980496555566788Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 43, loss: 0.10946691036224365Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 44, loss: 0.10786790400743484Saved checkpoint at step 45Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 45, loss: 0.11114661395549774Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 46, loss: 0.11685499548912048Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 47, loss: 0.126074880361557Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 48, loss: 0.20033998787403107Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 49, loss: 0.15269383788108826Saved checkpoint at step 50Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 50, loss: 0.1695154458284378Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 51, loss: 0.15434326231479645Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 52, loss: 0.13523367047309875Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 53, loss: 0.14987485110759735Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 54, loss: 0.15986254811286926Saved checkpoint at step 55Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 55, loss: 0.15688151121139526Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 56, loss: 0.15056319534778595Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 57, loss: 0.15178322792053223Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 58, loss: 0.15561023354530334Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 59, loss: 0.15891970694065094Saved checkpoint at step 60Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 60, loss: 0.15841172635555267Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 61, loss: 0.15717166662216187Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 62, loss: 0.15419942140579224Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 63, loss: 0.15271219611167908Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 64, loss: 0.15140581130981445Saved checkpoint at step 65Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 65, loss: 0.14739863574504852Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 66, loss: 0.1438044309616089Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 67, loss: 0.1426621675491333Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 68, loss: 0.1428421586751938Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 69, loss: 0.14141756296157837Saved checkpoint at step 70Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 70, loss: 0.13802580535411835Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 71, loss: 0.13680499792099Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 72, loss: 0.1375885009765625Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 73, loss: 0.13673445582389832Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 74, loss: 0.13431914150714874Saved checkpoint at step 75Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 75, loss: 0.132796972990036Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 76, loss: 0.13311852514743805Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 77, loss: 0.13267678022384644Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 78, loss: 0.1310112476348877Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 79, loss: 0.12956343591213226Saved checkpoint at step 80Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 80, loss: 0.12952426075935364Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 81, loss: 0.1292293518781662Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 82, loss: 0.12763074040412903Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 83, loss: 0.12680737674236298Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 84, loss: 0.12729512155056Saved checkpoint at step 85Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 85, loss: 0.12675176560878754Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 86, loss: 0.12530091404914856Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 87, loss: 0.12498505413532257Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 88, loss: 0.12463594228029251Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 89, loss: 0.1234278753399849Saved checkpoint at step 90Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 90, loss: 0.12300451099872589Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 91, loss: 0.12320985645055771Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 92, loss: 0.12246612459421158Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 93, loss: 0.12153832614421844Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 94, loss: 0.1217721626162529Saved checkpoint at step 95Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 95, loss: 0.12184974551200867Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 96, loss: 0.12149906903505325Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 97, loss: 0.12133416533470154Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 98, loss: 0.12058915942907333Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 99, loss: 0.12018893659114838Saved checkpoint at step 100Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 100, loss: 0.12015579640865326Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 101, loss: 0.11978670209646225Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 102, loss: 0.11968912184238434Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 103, loss: 0.11912395060062408Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 104, loss: 0.11879576742649078Saved checkpoint at step 105Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 105, loss: 0.11857516318559647Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 106, loss: 0.1181713193655014Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 107, loss: 0.11805718392133713Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 108, loss: 0.11768493056297302Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 109, loss: 0.11750298738479614Saved checkpoint at step 110Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 110, loss: 0.11747319251298904Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 111, loss: 0.11770090460777283Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 112, loss: 0.11815155297517776Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 113, loss: 0.11762141436338425Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 114, loss: 0.1159559041261673Saved checkpoint at step 115Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 115, loss: 0.11445591598749161Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 116, loss: 0.1140182614326477Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 117, loss: 0.1150854155421257Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 118, loss: 0.11450567096471786Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 119, loss: 0.11200319230556488Saved checkpoint at step 120Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 120, loss: 0.113042913377285Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 121, loss: 0.11355871707201004Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 122, loss: 0.11193477362394333Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 123, loss: 0.11168619990348816Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 124, loss: 0.11242271214723587Saved checkpoint at step 125Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 125, loss: 0.11165618896484375Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 126, loss: 0.1103954017162323Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 127, loss: 0.11149464547634125Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 128, loss: 0.11111971735954285Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 129, loss: 0.10990496724843979Saved checkpoint at step 130Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 130, loss: 0.11086277663707733Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 131, loss: 0.11044728755950928Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 132, loss: 0.10974788665771484Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 133, loss: 0.11053556203842163Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 134, loss: 0.11079050600528717Saved checkpoint at step 135Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 135, loss: 0.10945716500282288Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 136, loss: 0.10983554273843765Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 137, loss: 0.10979827493429184Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 138, loss: 0.10966945439577103Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 139, loss: 0.10938902944326401Saved checkpoint at step 140Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 140, loss: 0.10906355082988739Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 141, loss: 0.10878872126340866Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 142, loss: 0.10881690680980682Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 143, loss: 0.10846255719661713Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 144, loss: 0.10837554931640625Saved checkpoint at step 145Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 145, loss: 0.10852308571338654Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 146, loss: 0.1080116406083107Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 147, loss: 0.10853998363018036Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 148, loss: 0.1083887591958046Memstats: Before train step:        Using (GB) 0.46 / 59.39 (0.774541%) on cuda:0Step 149, loss: 0.10816454887390137Saved checkpoint at step 150Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 150, loss: 0.10826939344406128Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 151, loss: 0.1078299880027771Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 152, loss: 0.10766081511974335Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0Step 153, loss: 0.10789129137992859Memstats: Before train step:        Using (GB) 0.45 / 59.39 (0.757703%) on cuda:0^Csrun: interrupt (one more within 1 sec to abort)srun: StepId=23180.3 task 0: running^Csrun: sending Ctrl-C to StepId=23180.3srun: forcing job terminationsrun: Job step aborted: Waiting up to 32 seconds for job step to finish.[2025-08-23T14:56:36.424] error: *** STEP 23180.3 ON hai008 CANCELLED AT 2025-08-23T14:56:36 DUE to SIGNAL Killed ***^Csrun: sending Ctrl-C to StepId=23180.3srun: job abort in progress[franz.srambical@hai008.haicore.berlin:~/jafar] $ c^C[franz.srambical@hai008.haicore.berlin:~/jafar] $ ^C[franz.srambical@hai008.haicore.berlin:~/jafar] $ bash experiments/tokenizer_grain_checkpointing.sh Traceback (most recent call last):  File "/fast/home/franz.srambical/jafar/train_tokenizer.py", line 30, in <module>    jax.config.update("xla_python_client_mem_fraction", 0.98)  File "/fast/home/franz.srambical/jafar/.venv/lib/python3.10/site-packages/jax/_src/config.py", line 111, in update    raise AttributeError(f"Unrecognized config option: {name}")AttributeError: Unrecognized config option: xla_python_client_mem_fractionsrun: error: hai008: task 0: Exited with exit code 1[franz.srambical@hai008.haicore.berlin:~/jafar] $ bash experiments/tokenizer_grain_checkpointing.sh Running on 1 devices.Counting all components: ['decoder', 'encoder', 'vq']Parameter counts:{'decoder': 16858736, 'encoder': 16858752, 'vq': 32768, 'total': 33750256}Total memory size: 8.8 GB, Output size: 0.4 GB, Temp size: 8.3 GB, Argument size: 0.4 GB, Host temp size: 0.0 GB.FLOPs: 4.481e+13, Bytes: 3.947e+11 (367.6 GB), Intensity: 113.5 FLOPs/byteStarting training from step 0...Memstats: Before train step:        Using (GB) 0.41 / 77.61 (0.528282%) on cuda:0Memstats: After params initialized:        Using (GB) 0.44 / 77.61 (0.566937%) on cuda:0Step 0, loss: 0.25430572032928467Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 1, loss: 0.26582983136177063Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 2, loss: 0.27633073925971985Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 3, loss: 0.25486114621162415Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 4, loss: 0.23907887935638428Saved checkpoint at step 5Memstats: Before train step:        Using (GB) 0.46 / 77.61 (0.592707%) on cuda:0Step 5, loss: 0.22128331661224365Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 6, loss: 0.21060265600681305Memstats: Before train step:        Using (GB) 0.46 / 77.61 (0.592707%) on cuda:0Step 7, loss: 0.20288589596748352Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 8, loss: 0.20026752352714539Memstats: Before train step:        Using (GB) 0.46 / 77.61 (0.592707%) on cuda:0Step 9, loss: 0.1893157958984375Saved checkpoint at step 10Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 10, loss: 0.18349963426589966Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 11, loss: 0.18081218004226685Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 12, loss: 0.1745872050523758Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 13, loss: 0.16914372146129608Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 14, loss: 0.1608329713344574Saved checkpoint at step 15Memstats: Before train step:        Using (GB) 0.46 / 77.61 (0.592707%) on cuda:0Step 15, loss: 0.15873265266418457Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 16, loss: 0.15601487457752228Memstats: Before train step:        Using (GB) 0.46 / 77.61 (0.592707%) on cuda:0Step 17, loss: 0.15058937668800354Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 18, loss: 0.1418967843055725Memstats: Before train step:        Using (GB) 0.46 / 77.61 (0.592707%) on cuda:0Step 19, loss: 0.13888627290725708Saved checkpoint at step 20Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0^Csrun: interrupt (one more within 1 sec to abort)srun: StepId=23180.5 task 0: running^Csrun: sending Ctrl-C to StepId=23180.5srun: forcing job terminationsrun: Job step aborted: Waiting up to 32 seconds for job step to finish.[2025-08-23T15:03:10.390] error: *** STEP 23180.5 ON hai008 CANCELLED AT 2025-08-23T15:03:10 DUE to SIGNAL Killed ***[franz.srambical@hai008.haicore.berli[franz.srambical@hai008.haicore.berlin:~/jafar] $ bash experiments/tokenizer_grain_checkpointing.sh Running on 1 devices.Counting all components: ['decoder', 'encoder', 'vq']Parameter counts:{'decoder': 16858736, 'encoder': 16858752, 'vq': 32768, 'total': 33750256}Total memory size: 8.8 GB, Output size: 0.4 GB, Temp size: 8.3 GB, Argument size: 0.4 GB, Host temp size: 0.0 GB.FLOPs: 4.481e+13, Bytes: 3.947e+11 (367.6 GB), Intensity: 113.5 FLOPs/byteStarting training from step 0...Memstats: Before train step:        Using (GB) 0.41 / 77.61 (0.528282%) on cuda:0Memstats: After params initialized:        Using (GB) 0.44 / 77.61 (0.566937%) on cuda:0Step 0, loss: 0.25430572032928467Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 1, loss: 0.26582443714141846Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 2, loss: 0.27632513642311096Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 3, loss: 0.254859983921051Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0Step 4, loss: 0.2390667200088501Saved checkpoint at step 5Memstats: Before train step:        Using (GB) 0.46 / 77.61 (0.592707%) on cuda:0Step 5, loss: 0.22132423520088196Memstats: Before train step:        Using (GB) 0.45 / 77.61 (0.579822%) on cuda:0^Csrun: interrupt (one more within 1 sec to abort)srun: StepId=23180.6 task 0: running^Csrun: sending Ctrl-C to StepId=23180.6srun: forcing job terminationsrun: Job step aborted: Waiting up to 32 seconds for job step to finish.[2025-08-23T15:08:37.965] error: *** STEP 23180.6 ON hai008 CANCELLED AT 2025-08-23T15:08:37 DUE to SIGNAL Killed ***[franz.srambical@hai008.haicore.berlin:~/jafar] $ git stashSaved working directory and index state WIP on print-mem-stats-after-param-init: 0ddc202 feat: print memory stats after parameter init[franz.srambical@hai008.haicore.berlin:~/jafar] $ squeue  JOBID       USER PARTITION NODES  CPUS ST         SUBMIT_TIME          START_TIME     TIME TIME_LIMIT NODELIST(REASON)  23186 franz.sram interacti     1     2  R 2025-08-23T16:57:59 2025-08-23T16:57:59  1:20:14 1-00:00:00 hai001  23185   xiao.liu interacti     1    64  R 2025-08-23T16:11:47 2025-08-23T16:11:47  2:06:26   23:59:00 hai005  23184   xiao.liu interacti     1    64  R 2025-08-23T16:11:43 2025-08-23T16:11:43  2:06:30   23:59:00 hai004  23183 franz.sram interacti     1     2  R 2025-08-23T15:03:18 2025-08-23T15:03:18  3:14:55 1-00:00:00 hai002  23182 franz.sram interacti     1     2  R 2025-08-23T15:03:15 2025-08-23T15:03:15  3:14:58 1-00:00:00 hai001  23180 franz.sram interacti     1     2  R 2025-08-23T10:10:08 2025-08-23T10:10:08  8:08:05 1-00:00:00 hai008  23181 alfred.ngu  standard     1    16  R 2025-08-23T14:19:28 2025-08-23T14:19:28  3:58:45 1-00:00:00 hai003  23178 alfred.ngu  standard     1    16  R 2025-08-23T01:26:22 2025-08-23T01:26:39 16:51:34 1-00:00:00 hai007  23177 alfred.ngu  standard     1    16  R 2025-08-23T01:26:18 2025-08-23T01:26:19 16:51:54 1-00:00:00 hai006[franz.srambical@hai008.haicore.berlin:~/jafar] $ bash experiments/tokenizer_grain_checkpointing.sh Traceback (most recent call last):  File "/fast/home/franz.srambical/jafar/train_tokenizer.py", line 25, in <module>    from utils.train_utils import (ImportError: cannot import name 'print_mem_stats' from 'utils.train_utils' (/fast/home/franz.srambical/jafar/utils/train_utils.py)srun: error: hai008: task 0: Exited with exit code 1[franz.srambical@hai008.haicore.berlin:~/jafar] $ bash experiments/tokenizer_grain_checkpointing.sh Running on 1 devices.Memstats: Before params initialized:        Using (GB) 0.0 / 77.61 (0.000000%) on cuda:0Counting all components: ['decoder', 'encoder', 'vq']Parameter counts:{'decoder': 16858736, 'encoder': 16858752, 'vq': 32768, 'total': 33750256}Total memory size: 8.8 GB, Output size: 0.4 GB, Temp size: 8.3 GB, Argument size: 0.4 GB, Host temp size: 0.0 GB.FLOPs: 4.481e+13, Bytes: 3.947e+11 (367.6 GB), Intensity: 113.5 FLOPs/byteStarting training from step 0...Memstats: After params initialized:        Using (GB) 0.44 / 77.61 (0.566937%) on cuda:0Step 0, loss: 0.25430572032928467^Csrun: interrupt (one more within 1 sec to abort)srun: StepId=23180.8 task 0: running^Csrun: sending Ctrl-C to StepId=23180.8srun: forcing job terminationsrun: Job step aborted: Waiting up to 32 seconds for job step to finish.[2025-08-23T19:04:20.725] error: *** STEP 23180.8 ON hai008 CANCELLED AT 2025-08-23T19:04:20 DUE to SIGNAL Killed ***[franz.srambical@hai008.haicore.berlin:~/jafar] $ bash experiments/tokenizer_grain_checkpointing.sh Running on 1 devices.Memstats: Before params initialized:        Using (GB) 0.0 / 77.61 (0.000000%) on cuda:0Memstats: Right after model initialized:        Using (GB) 0.15 / 77.61 (0.193274%) on cuda:0Counting all components: ['decoder', 'encoder', 'vq']Parameter counts:{'decoder': 16858736, 'encoder': 16858752, 'vq': 32768, 'total': 33750256}Total memory size: 8.8 GB, Output size: 0.4 GB, Temp size: 8.3 GB, Argument size: 0.4 GB, Host temp size: 0.0 GB.FLOPs: 4.481e+13, Bytes: 3.947e+11 (367.6 GB), Intensity: 113.5 FLOPs/byteStarting training from step 0...Memstats: After params initialized:        Using (GB) 0.44 / 77.61 (0.566937%) on cuda:0Step 0, loss: 0.25430572032928467^Csrun: interrupt (one more within 1 sec to abort)srun: StepId=23180.9 task 0: running^Csrun: sending Ctrl-C to StepId=23180.9srun: forcing job terminationsrun: Job step aborted: Waiting up to 32 seconds for job step to finish.[2025-08-23T19:05:17.743] error: *** STEP 23180.9 ON hai008 CANCELLED AT 2025-08-23T19:05:17 DUE to SIGNAL Killed ***[franz.srambical@hai008.haicore.berlin:~/jafar] $ salloc: Job 23180 has exceeded its time limit and its allocation has been revoked.                                                                                                                                    srun: Job step aborted: Waiting up to 32 seconds for job step to finish.[2025-08-24T10:10:10.009] error: *** STEP 23180.interactive ON hai008 CANCELLED AT 2025-08-24T10:10:10 DUE TO TIME LIMIT ***srun: error: hai008: task 0: Killed