[terminal_command] TERMINAL ()
2025-06-25 00:38:23.929 [info] > git log --format=%H%n%aN%n%aE%n%at%n%ct%n%P%n%D%n%B -z --shortstat --diff-merges=first-parent -n50 --skip=0 --topo-order --decorate=full --stdin [67ms]2025-06-25 00:38:23.931 [info] > git status -z -uall [12ms]2025-06-25 00:38:23.950 [info] > git for-each-ref --sort -committerdate --format %(refname) %(objectname) %(*objectname) [23ms]2025-06-25 00:38:25.145 [info] > git ls-tree -l HEAD -- /lustre/groups/haicu/workspace/franz.srambical/jafar/train_tokenizer.py [11ms]2025-06-25 00:38:25.153 [info] > git ls-tree -l HEAD -- /lustre/groups/haicu/workspace/franz.srambical/jafar/train_tokenizer.py [11ms]2025-06-25 00:38:25.349 [info] > git show --textconv HEAD:train_tokenizer.py [17ms]2025-06-25 00:38:25.360 [info] > git show --textconv HEAD:train_tokenizer.py [12ms]' vscode.git.Git && cat -n vscode.git.Git | sed -n '175,191p'```
actor_val:  74%|███████▍  | 74/100 [01:04<00:21,  1.19it/s][2025-12-24 14:52:25] train_sft.py:377 - SFT rollout sample: tokens_len=4192, response_length=4100, [2025-12-24 14:52:25] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:25] train_sft.py:384 -  -i '536,536c\        attention_mask = jnp.ones((targ_B1QKet_seq_len, target_seq_len), dtB1QKe=jnp.bool_)' utils/nn.py && cat -n utils/nn.py | sed -n '526,546p'```
actor_val:  75%|███████▌  | 75/100 [01:05<00:20,  1.19it/s][2025-12-24 14:52:26] train_sft.py:377 - SFT rollout sample: tokens_len=3980, response_length=3888, [2025-12-24 14:52:26] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:26] train_sft.py:384 -  -n logs/logs_training/train_tokenizer_minecraft_overfit_sample_3290295.log | sed -n '101,113p'```
actor_val:  76%|███████▌  | 76/100 [01:06<00:19,  1.20it/s][2025-12-24 14:52:27] train_sft.py:377 - SFT rollout sample: tokens_len=4257, response_length=3859, [2025-12-24 14:52:27] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:27] train_sft.py:384 -  -n scripts_cremers/train_tokenizer.sh```
actor_val:  77%|███████▋  | 77/100 [01:06<00:19,  1.19it/s][2025-12-24 14:52:28] train_sft.py:377 - SFT rollout sample: tokens_len=2527, response_length=2435, [2025-12-24 14:52:28] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:28] train_sft.py:384 -  -n anysphere.remote-ssh.Remote - SSH```
actor_val:  78%|███████▊  | 78/100 [01:07<00:19,  1.10it/s][2025-12-24 14:52:29] train_sft.py:377 - SFT rollout sample: tokens_len=4111, response_length=3980, [2025-12-24 14:52:29] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:29] train_sft.py:384 -  -n slurm/jobs/mihir/horeka/yolo-runs/sampling.sh```
actor_val:  79%|███████▉  | 79/100 [01:08<00:18,  1.12it/s][2025-12-24 14:52:30] train_sft.py:377 - SFT rollout sample: tokens_len=2916, response_length=2491, [2025-12-24 14:52:30] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:30] train_sft.py:384 -  -n utils/nn.py```
actor_val:  80%|████████  | 80/100 [01:09<00:17,  1.15it/s][2025-12-24 14:52:30] train_sft.py:377 - SFT rollout sample: tokens_len=2897, response_length=752, [2025-12-24 14:52:30] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:30] train_sft.py:384 -  -n generate_arrayrecord_dataset.py | sed -n '159,179p'```
actor_val:  81%|████████  | 81/100 [01:10<00:16,  1.18it/s][2025-12-24 14:52:31] train_sft.py:377 - SFT rollout sample: tokens_len=3563, response_length=1418, [2025-12-24 14:52:31] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:31] train_sft.py:384 -  -n genie.py | sed -n '152,172p'```
actor_val:  82%|████████▏ | 82/100 [01:11<00:15,  1.19it/s][2025-12-24 14:52:32] train_sft.py:377 - SFT rollout sample: tokens_len=2895, response_length=2452, [2025-12-24 14:52:32] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:32] train_sft.py:384 -  -n utils/nn.py```
actor_val:  83%|████████▎ | 83/100 [01:12<00:13,  1.21it/s][2025-12-24 14:52:33] train_sft.py:377 - SFT rollout sample: tokens_len=3179, response_length=1034, [2025-12-24 14:52:33] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:33] train_sft.py:384 -  -i '325,326c\            z_FNM = z_FNM.at[:, pos_index[1], :].set(z_FM)' /fast/home/franz.srambical/jafar/utils/nn.py && cat -n /fast/home/franz.srambical/jafar/utils/nn.py | sed -n '315,335p'```
actor_val:  84%|████████▍ | 84/100 [01:12<00:13,  1.22it/s][2025-12-24 14:52:34] train_sft.py:377 - SFT rollout sample: tokens_len=3916, response_length=3419, [2025-12-24 14:52:34] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:34] train_sft.py:384 -  -n genie.py```
actor_val:  85%|████████▌ | 85/100 [01:13<00:12,  1.23it/s][2025-12-24 14:52:34] train_sft.py:377 - SFT rollout sample: tokens_len=3864, response_length=3419, [2025-12-24 14:52:34] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:34] train_sft.py:384 -  -n genie.py```
actor_val:  86%|████████▌ | 86/100 [01:14<00:11,  1.21it/s][2025-12-24 14:52:35] train_sft.py:377 - SFT rollout sample: tokens_len=3896, response_length=3361, [2025-12-24 14:52:35] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:35] train_sft.py:384 -  -n /fast/home/franz.srambical/jafar/utils/nn.py```
actor_val:  87%|████████▋ | 87/100 [01:15<00:11,  1.10it/s][2025-12-24 14:52:36] train_sft.py:377 - SFT rollout sample: tokens_len=3761, response_length=1617, [2025-12-24 14:52:36] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:36] train_sft.py:384 -  -n genie.py | sed -n '119,139p'```
actor_val:  88%|████████▊ | 88/100 [01:16<00:10,  1.13it/s][2025-12-24 14:52:37] train_sft.py:377 - SFT rollout sample: tokens_len=4177, response_length=3804, [2025-12-24 14:52:37] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:37] train_sft.py:384 -  -n train_dynamics.py```
actor_val:  89%|████████▉ | 89/100 [01:17<00:09,  1.14it/s][2025-12-24 14:52:38] train_sft.py:377 - SFT rollout sample: tokens_len=3982, response_length=1837, [2025-12-24 14:52:38] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:38] train_sft.py:384 -  -n .venv/lib/python3.10/site-packages/flax/nnx/nn/attention.py | sed -n '508,528p'```
actor_val:  90%|█████████ | 90/100 [01:18<00:08,  1.16it/s][2025-12-24 14:52:39] train_sft.py:377 - SFT rollout sample: tokens_len=4041, response_length=3949, [2025-12-24 14:52:39] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:39] train_sft.py:384 -  -n batch_size_benchmark_tokenizer_cudnn_flash_attention_larger_ffn.json```
actor_val:  91%|█████████ | 91/100 [01:19<00:07,  1.16it/s][2025-12-24 14:52:40] train_sft.py:377 - SFT rollout sample: tokens_len=3705, response_length=1560, [2025-12-24 14:52:40] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:40] train_sft.py:384 -  -n scripts_horeka/modelsize_scaling/tokenizer/train_tokenizer_80M.sbatch```
actor_val:  92%|█████████▏| 92/100 [01:19<00:06,  1.17it/s][2025-12-24 14:52:41] train_sft.py:377 - SFT rollout sample: tokens_len=4216, response_length=3793, [2025-12-24 14:52:41] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:41] train_sft.py:384 -  -n scripts_horeka/train_lam_overfit_sample.sbatch```
actor_val:  93%|█████████▎| 93/100 [01:20<00:06,  1.16it/s][2025-12-24 14:52:41] train_sft.py:377 - SFT rollout sample: tokens_len=4025, response_length=1880, [2025-12-24 14:52:41] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:41] train_sft.py:384 -  -n slurm/jobs/mihir/horeka/modelsize_scaling/lam/tester.sh```
actor_val:  94%|█████████▍| 94/100 [01:21<00:05,  1.19it/s][2025-12-24 14:52:42] train_sft.py:377 - SFT rollout sample: tokens_len=3284, response_length=2934, [2025-12-24 14:52:42] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:42] train_sft.py:384 -  -n genie.py```
actor_val:  95%|█████████▌| 95/100 [01:22<00:04,  1.20it/s][2025-12-24 14:52:43] train_sft.py:377 - SFT rollout sample: tokens_len=3073, response_length=928, [2025-12-24 14:52:43] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:43] train_sft.py:384 -  -n /fast/home/franz.srambical/jafar/sample.py | sed -n '144,164p'```
actor_val:  96%|█████████▌| 96/100 [01:23<00:03,  1.21it/s][2025-12-24 14:52:44] train_sft.py:377 - SFT rollout sample: tokens_len=3018, response_length=2503, [2025-12-24 14:52:44] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:44] train_sft.py:384 -  -n /fast/home/franz.srambical/jafar/utils/nn.py```
actor_val:  97%|█████████▋| 97/100 [01:23<00:02,  1.22it/s][2025-12-24 14:52:45] train_sft.py:377 - SFT rollout sample: tokens_len=4191, response_length=2782, [2025-12-24 14:52:45] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:45] train_sft.py:384 -  -n slurm/jobs/mihir/horeka/yolo-runs/sampling.sh```
actor_val:  98%|█████████▊| 98/100 [01:24<00:01,  1.20it/s][2025-12-24 14:52:45] train_sft.py:377 - SFT rollout sample: tokens_len=2626, response_length=2534, [2025-12-24 14:52:45] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:45] train_sft.py:384 -  -i '33,232c\    image_height: int = 90    image_width: int = 160    data_dir: str = "data_tfrecords/coinrun"    checkpoint: str = ""    # Optimization    vq_beta: float = 0.25    batch_size: int = 48    min_lr: float = 3e-4    max_lr: float = 3e-4    warmup_steps: int = 10000    # Tokenizer    model_dim: int = 512    latent_dim: int = 32    num_latents: int = 1024    patch_size: int = 4    num_blocks: int = 8    num_heads: int = 8    dropout: float = 0.0    codebook_dropout: float = 0.01    # Logging    log: bool = False    entity: str = ""    project: str = ""    log_interval: int = 5    log_image_interval: int = 250    ckpt_dir: str = ""    log_checkpoint_interval: int = 10000    log_gradients: bool = Falseargs = tyro.cli(Args)def tokenizer_loss_fn(params, state, inputs):    # --- Compute loss ---    outputs = state.apply_fn(        params,        inputs,        training=True,        rngs={"params": inputs["rng"], "dropout": inputs["dropout_rng"]},    )    mse = jnp.square(inputs["videos"] - outputs["recon"]).mean()    q_loss = jnp.square(jax.lax.stop_gradient(outputs["emb"]) - outputs["z"]).mean()    commitment_loss = jnp.square(        outputs["emb"] - jax.lax.stop_gradient(outputs["z"])    ).mean()    loss = mse + q_loss + args.vq_beta * commitment_loss    # --- Compute validation metrics ---    gt = inputs["videos"].clip(0, 1).reshape(-1, *inputs["videos"].shape[2:])    recon = outputs["recon"].clip(0, 1).reshape(-1, *outputs["recon"].shape[2:])    psnr = pix.psnr(gt, recon).mean()    ssim = pix.ssim(gt, recon).mean()    _, index_counts = jnp.unique_counts(        jnp.ravel(outputs["indices"]), size=args.num_latents, fill_value=0    )    codebook_usage = (index_counts != 0).mean()    metrics = dict(        loss=loss,        mse=mse,        q_loss=q_loss,        commitment_loss=commitment_loss,        psnr=psnr,        ssim=ssim,        codebook_usage=codebook_usage,    )    return loss, (outputs["recon"], metrics)@jax.jitdef train_step(state, inputs):    grad_fn = jax.value_and_grad(tokenizer_loss_fn, has_aux=True, allow_int=True)    (loss, (recon, metrics)), grads = grad_fn(state.params, state, inputs)    state = state.apply_gradients(grads=grads)    if args.log_gradients:        metrics["encoder_gradients_std/"] = jax.tree.map(            lambda x: x.std(), grads["params"]["encoder"]        )        metrics["vq_gradients_std/"] = jax.tree.map(            lambda x: x.std(), grads["params"]["vq"]        )        metrics["decoder_gradients_std/"] = jax.tree.map(            lambda x: x.std(), grads["params"]["decoder"]        )    return state, loss, recon, metricsif __name__ == "__main__":    jax.distributed.initialize()    num_devices = jax.device_count()    if num_devices == 0:        raise ValueError("No JAX devices found.")    print(f"Running on {num_devices} devices.")    if args.batch_size % num_devices != 0:        raise ValueError(            f"Global batch size {args.batch_size} must be divisible by "            f"number of devices {num_devices}."        )    per_device_batch_size_for_init = args.batch_size // num_devices    rng = jax.random.PRNGKey(args.seed)    if args.log and jax.process_index() == 0:        wandb.init(entity=args.entity, project=args.project, group="debug", config=args)    # --- Initialize model ---    tokenizer = TokenizerVQVAE(        in_dim=args.image_channels,        model_dim=args.model_dim,        latent_dim=args.latent_dim,        num_latents=args.num_latents,        patch_size=args.patch_size,        num_blocks=args.num_blocks,        num_heads=args.num_heads,        dropout=args.dropout,        codebook_dropout=args.codebook_dropout,    )    rng, _rng = jax.random.split(rng)    image_shape = (args.image_height, args.image_width, args.image_channels)    inputs = dict(        videos=jnp.zeros(            (per_device_batch_size_for_init, args.seq_len, *image_shape),            dtype=jnp.float32,        ),    )    init_params = tokenizer.init(_rng, inputs)    # --- Initialize optimizer ---    lr_schedule = optax.warmup_cosine_decay_schedule(        args.min_lr, args.max_lr, args.warmup_steps, args.num_steps    )    tx = optax.adamw(learning_rate=lr_schedule, b1=0.9, b2=0.9, weight_decay=1e-4)    train_state = TrainState.create(apply_fn=tokenizer.apply, params=init_params, tx=tx)    # FIXME: switch to create_hybrid_device_mesh for runs spanning multiple nodes    device_mesh_arr = create_device_mesh((num_devices,))    mesh = Mesh(devices=device_mesh_arr, axis_names=("data",))    replicated_sharding = NamedSharding(mesh, PartitionSpec())    train_state = jax.device_put(train_state, replicated_sharding)    # --- Load checkpoint ---    step = 0    if args.checkpoint:        restore_target = {"model": train_state}        restore_args = orbax_utils.restore_args_from_target(restore_target)        train_state.params["params"].update(            PyTreeCheckpointer()            .restore(args.checkpoint, item=restore_target, restore_args=restore_args)[                "model"            ]            .params["params"]        )        # Assume checkpoint is of the form tokenizer_<timestamp>_<step>        step += int(args.checkpoint.split("_")[-1])    # --- TRAIN LOOP ---    tfrecord_files = [        os.path.join(args.data_dir, x)        for x in os.listdir(args.data_dir)        if x.endswith(".tfrecord")    ]    dataloader = get_dataloader(        # NOTE: We deliberately pass the global batch size        # The dataloader shards the dataset across all processes        tfrecord_files,        args.seq_len,        args.batch_size,        *image_shape,    )    print(f"Starting training from step {step}...")    while step < args.num_steps:        for videos in dataloader:            # --- Train step ---            rng, _rng, _rng_dropout = jax.random.split(rng, 3)            videos_sharding = NamedSharding(                mesh, PartitionSpec("data", None, None, None, None)            )            videos = jax.make_array_from_process_local_data(videos_sharding, videos)            inputs = dict(videos=videos, rng=_rng, dropout_rng=_rng_dropout)            start_time = time.time()            train_state, loss, recon, metrics = train_step(train_state, inputs)            elapsed_time = (time.time() - start_time) * 1000            print(f"Step {step}, loss: {loss}, step time: {elapsed_time}ms")            step += 1            # --- Logging ---            if args.log:                if step % args.log_interval == 0 and jax.process_index() == 0:                    wandb.log(                        {                            "loss": loss,                            "step": step,                            "step_time_ms": elapsed_time,                            **metrics,                        }                    )                if step % args.log_image_interval == 0:                    gt_seq = inputs["videos"][0]                    recon_seq = recon[0].clip(0, 1)                    comparison_seq = jnp.concatenate((gt_seq, recon_seq), axis=1)                    comparison_seq = einops.rearrange(                        comparison_seq * 255, "t h w c -> h (t w) c"                    )                    # NOTE: Process-dependent control flow deliberately happens                    # after indexing operation since it must not contain code                    # sections that lead to cross-accelerator communication.                    if jax.process_index() == 0:                        log_images = dict(                            image=wandb.Image(np.asarray(gt_seq[0])),                            recon=wandb.Image(np.asarray(recon_seq[0])),                            true_vs_recon=wandb.Image(                                np.asarray(comparison_seq.astype(np.uint8))                            ),                        )                        wandb.log(log_images)            if step % args.log_checkpoint_interval == 0:                ckpt = {"model": train_state}                orbax_checkpointer = orbax.checkpoint.PyTreeCheckpointer()                save_args = orbax_utils.save_args_from_target(ckpt)                orbax_checkpointer.save(                    os.path.join(os.getcwd(), args.ckpt_dir, f"tokenizer_{
actor_val:  99%|█████████▉| 99/100 [01:25<00:00,  1.21it/s][2025-12-24 14:52:46] train_sft.py:377 - SFT rollout sample: tokens_len=3593, response_length=3501, [2025-12-24 14:52:46] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:46] train_sft.py:384 -  -n /hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/train_dynamics.py```
actor_val: 100%|██████████| 100/100 [01:26<00:00,  1.16it/s][2025-12-24 14:52:47] train_sft.py:663 - step 0: {'val/loss': 1.5654600524902345}[2025-12-24 14:52:47] train_sft.py:377 - SFT rollout sample: tokens_len=2539, response_length=394, [2025-12-24 14:52:47] train_sft.py:382 - === SFT TARGET TEXT ===[2025-12-24 14:52:47] train_sft.py:384 -  -n /hkfs/home/project/hk-project-p0023960/tum_cte0515/Projects/jafar/utils/dataloader.py | sed -n '37,57p'```
[2025-12-24 14:52:47] timer.py:24 - Timer actor_train startactor_train:  75%|███████▌  | 3/4 [00:01<00:00,  2.34it/s][2025-12-24 14:52:50] train_sft.py:601 - step 0: {'train/loss': 1.8810048699378967, 'train/grad_norm': 1.7276290655136108, 'train/lr_0': 2.0000000000000002e-07}actor_train: 100%|██████████| 4/4 [00:02<00:00,  1.79it/s][2025-12-24 14:52:50] timer.py:32 - Timer actor_train end (elapsed: 2.2s)[2025-12-24 14:52:50] timer.py:24 - Timer actor_train startactor_train:  75%|███████▌  | 3/4 [00:00<00:00,  3.37it/s][2025-12-24 14:52:51] train_sft.py:601 - step 1: {'train/loss': 1.3827951550483704, 'train/grad_norm': 1.3713090419769287, 'train/lr_0': 4.0000000000000003e-07}actor_train: 100%|██████████| 4/4 [00:01<00:00,  3.21it/s][2025-12-24 14:52:51] timer.py:32 - Timer actor_train end (elapsed: 1.2s)[2025-12-24 14:52:52] timer.py:24 - Timer actor_train startactor_train:  75%|███████▌  | 3/4 [00:00<00:00,  3.14it/s][2025-12-24 14:52:53] train_sft.py:601 - step 2: {'train/loss': 1.5883387327194214, 'train/grad_norm': 1.8439953327178955, 'train/lr_0': 6.000000000000001e-07}actor_train: 100%|██████████| 4/4 [00:01<00:00,  3.12it/s][2025-12-24 14:52:53] timer.py:32 - Timer actor_train end (elapsed: 1.3s)[2025-12-24 14:52:53] timer.py:24 - Timer actor_train startactor_train:  75%|███████▌  | 3/4 [00:00<00:00,  3.31it/s][2025-12-24 14:52:55] train_sft.py:601 - step 3: {'train/loss': 1.5394169092178345, 'train/grad_norm': 1.4356211423873901, 'train/lr_0': 8.000000000000001e-07}actor_train: 100%|██████████| 4/4 [00:01<00:00,  3.25it/s][2025-12-24 14:52:55] timer.py:32 - Timer actor_train end (elapsed: 1.2s)[2025-12-24 14:52:55] timer.py:24 - Timer actor_train startactor_train:  75%|███████▌  | 3/4 [00:00<00:00,  3.20it/s][2025-12-24 14:52:56] train_sft.py:601 - step 4: {'train/loss': 1.3747694492340088, 'train/grad_norm': 1.6878149509429932, 'train/lr_0': 1e-06}actor_train: 100%|██████████| 4/4 [00:01<00:00,  3.14it/s][2025-12-24 14:52:56] timer.py:32 - Timer actor_train end (elapsed: 1.3s)[2025-12-24 14:52:56] timer.py:24 - Timer actor_train startactor_train:  75%|███████▌  | 3/4 [00:00<00:00,  3.37it/s][2025-12-24 14:52:58] train_sft.py:601 - step 5: {'train/loss': 1.4506165981292725, 'train/grad_norm': 1.4593477249145508, 'train/lr_0': 1.2000000000000002e-06}actor_train: 100%|██████████| 4/4 [00:01<00:00,  3.29it/s][2025-12-24 14:52:58] timer.py:32 - Timer actor_train end (elapsed: 1.2s)[2025-12-24 14:52:58] timer.py:24 - Timer actor_train startactor_train:  75%|███████▌  | 3/4 [00:00<00:00,  3.30it/s][2025-12-24 14:52:59] train_sft.py:601 - step 6: {'train/loss': 1.504898726940155, 'train/grad_norm': 1.5677669048309326, 'train/lr_0': 1.4e-06}actor_train: 100%|██████████| 4/4 [00:01<00:00,  3.26it/s][2025-12-24 14:52:59] timer.py:32 - Timer actor_train end (elapsed: 1.2s)[2025-12-24 14:52:59] timer.py:24 - Timer actor_train startactor_train:  75%|███████▌  | 3/4 [00:01<00:00,  2.70it/s][2025-12-24 14:53:01] train_sft.py:601 - step 7: {'train/loss': 1.5336732864379883, 'train/grad_norm': 1.402663230895996, 'train/lr_0': 1.6000000000000001e-06}actor_train: 100%|██████████| 4/4 [00:01<00:00,  2.59it/s][2025-12-24 14:53:01] timer.py:32 - Timer actor_train end (elapsed: 1.5s)[2025-12-24 14:53:01] timer.py:24 - Timer actor_train startactor_train:  75%|███████▌  | 3/4 [00:00<00:00,  3.42it/s][2025-12-24 14:53:02] train_sft.py:601 - step 8: {'train/loss': 1.444072961807251, 'train/grad_norm': 1.431401014328003, 'train/lr_0': 1.8000000000000001e-06}actor_train: 100%|██████████| 4/4 [00:01<00:00,  3.38it/s][2025-12-24 14:53:02] timer.py:32 - Timer actor_train end (elapsed: 1.2s)[2025-12-24 14:53:03] timer.py:24 - Timer actor_train startactor_train:  75%|███████▌  | 3/4 [00:00<00:00,  3.42it/s][2025-12-24 14:53:04] train_sft.py:601 - step 9: {'train/loss': 1.7816160321235657, 'train/grad_norm': 1.7752093076705933, 'train/lr_0': 2e-06}actor_train: 100%|██████████| 4/4 [00:01<00:00,  3.37it/s][2025-12-24 14:53:04] timer.py:32 - Timer actor_train end (elapsed: 1.2s)[2025-12-24 14:53:04] timer.py:24 - Timer actor_train start^Csrun: interrupt (one more within 1 sec to abort)srun: StepId=38556.9 task 0: running^Csrun: sending Ctrl-C to StepId=38556.9srun: forcing job terminationsrun: Job step aborted: Waiting up to 32 seconds for job step to finish.actor_train:  75%|███████▌  | 3/4 [00:00<00:00,  3.40it/s][2025-12-24T14:53:05.654] error: *** STEP 38556.9 ON hai006 CANCELLED AT 2025-12-24T14:53:05 DUE to SIGNAL Killed ***^Csrun: sending Ctrl-C to StepId=38556.9^Csrun: job abort in progresssrun: sending Ctrl-C to StepId=38556.9^C^Csrun: sending Ctrl-C to StepId=38556.9^Csrun: sending Ctrl-C to StepId=38556.9srun: job abort in progress[mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ source .venv/bin/acbash: .venv/bin/ac: No such file or directory[mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ source .venv/bin/activate(miles) [mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ python test.py/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.  import pynvml  # type: ignore[import]Traceback (most recent call last):  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 479, in cached_files    hf_hub_download(  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn    validate_repo_id(arg_value)  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 160, in validate_repo_id    raise HFValidationError(huggingface_hub.errors.HFValidationError: Repo id must use alphanumeric chars, '-', '_' or '.'. The name cannot start or end with '-' or '.' and the maximum length is 96: 'Qwen/Qwen3-0.6B§'.During handling of the above exception, another exception occurred:Traceback (most recent call last):  File "/fast/home/mihir.mahajan/Projects/miles/test.py", line 3, in <module>    tok = AutoTokenizer.from_pretrained(          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py", line 1073, in from_pretrained    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py", line 905, in get_tokenizer_config    resolved_config_file = cached_file(                           ^^^^^^^^^^^^  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 322, in cached_file    file = cached_files(path_or_repo_id=path_or_repo_id, filenames=[filename], **kwargs)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 532, in cached_files    _get_cache_file_to_return(path_or_repo_id, filename, cache_dir, revision, repo_type)  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/transformers/utils/hub.py", line 143, in _get_cache_file_to_return    resolved_file = try_to_load_from_cache(                    ^^^^^^^^^^^^^^^^^^^^^^^  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn    validate_repo_id(arg_value)  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 160, in validate_repo_id    raise HFValidationError(huggingface_hub.errors.HFValidationError: Repo id must use alphanumeric chars, '-', '_' or '.'. The name cannot start or end with '-' or '.' and the maximum length is 96: 'Qwen/Qwen3-0.6B§'.(miles) [mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ python test.py/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.  import pynvml  # type: ignore[import]=== NO THINKING ===
systemYou are helpful.

userWhat is 2 + 2?

assistant<think></think>=== DEFAULT (thinking) ===
systemYou are helpful.

userWhat is 2 + 2?

assistant(miles) [mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ python test.py/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.  import pynvml  # type: ignore[import]=== NO THINKING ===
systemYou are helpful.

userWhat is 2 + 2?

assistant<think></think>The answer is 4.

assistant<think></think>=== DEFAULT (thinking) ===
systemYou are helpful.

userWhat is 2 + 2?

assistant<think></think>The answer is 4.

assistant(miles) [mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ python test.py/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.  import pynvml  # type: ignore[import]=== NO THINKING ===
systemYou are helpful.

userWhat is 2 + 2?

assistant<think></think>The answer is 4.
=== DEFAULT (thinking) ===
systemYou are helpful.

userWhat is 2 + 2?

assistant<think></think>The answer is 4.
(miles) [mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ python test.py  File "/fast/home/mihir.mahajan/Projects/miles/test.py", line 22    ))     ^SyntaxError: unmatched ')'(miles) [mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ (miles) [mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ python test.py/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.  import pynvml  # type: ignore[import]=== NO THINKING ===
systemYou are helpful.

userWhat is 2 + 2?

assistant<think></think>The answer is 4.
=== DEFAULT (thinking) ===
systemYou are helpful.

userWhat is 2 + 2?

assistant<think></think>The answer is 4.
(miles) [mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ sh slurm/jobs/mihir/berlin/crowd-pilot/run_sqwen3-0.6b-sft-torch-pythonic.sh#!/usr/bin/env bash#SBATCH --nodes=1#SBATCH --ntasks-per-node=1#SBATCH --time=24:00:00#SBATCH --cpus-per-task=5#SBATCH --gres=gpu:2#SBATCH --mem=600GB#SBATCH --signal=USR1@60#SBATCH --output=/fast/project/HFMI_SynergyUnit/mihir/logs/train/qwen3-0.6b-sft-torch/%x_%j.log#SBATCH --error=/fast/project/HFMI_SynergyUnit/mihir/logs/train/qwen3-0.6b-sft-torch/%x_%j.log#SBATCH --job-name=run_qwen3-0.6b-sft-torch#SBATCH --qos=lowtrap 'echo "Job killed by signal"' USR1cat $0module load CUDA/12.8source .venv/bin/activatesrun sh scripts/run-sft-torchrun.shHAS_NVLINK: 0 (detected 0 NVLink references)/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.  import pynvml  # type: ignore[import]W1224 15:12:54.782000 3217568 torch/distributed/run.py:774] W1224 15:12:54.782000 3217568 torch/distributed/run.py:774] *****************************************W1224 15:12:54.782000 3217568 torch/distributed/run.py:774] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. W1224 15:12:54.782000 3217568 torch/distributed/run.py:774] *****************************************/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.  import pynvml  # type: ignore[import]/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.  import pynvml  # type: ignore[import][2025-12-24 15:13:15] arguments.py:1538 - n_samples_per_prompt is set to 1, grpo_std_normalization will be set to False.[2025-12-24 15:13:15] arguments.py:1583 - args.eval_max_context_len is not set. Use args.rollout_max_context_len 8192 as default value.[2025-12-24 15:13:20] arguments.py:1538 - n_samples_per_prompt is set to 1, grpo_std_normalization will be set to False.[2025-12-24 15:13:20] arguments.py:1583 - args.eval_max_context_len is not set. Use args.rollout_max_context_len 8192 as default value.[Gloo] Rank 0 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1[2025-12-24 15:13:20] train_sft.py:120 - [Rank 0] Distributed initialized: world_size=2, local_rank=0[Gloo] Rank 1 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1[2025-12-24 15:13:20] train_sft.py:146 - [Rank 0] Device mesh: dp_size=2, cp_size=1, dp_rank=0, cp_rank=0[2025-12-24 15:13:20] train_sft.py:120 - [Rank 1] Distributed initialized: world_size=2, local_rank=1[2025-12-24 15:13:20] train_sft.py:146 - [Rank 1] Device mesh: dp_size=2, cp_size=1, dp_rank=1, cp_rank=0wandb: Currently logged in as: mihir-mahajan2002 (instant-uv) to https://api.wandb.ai. Use `wandb login --relogin` to force reloginwandb: WARNING The `shared` mode feature is experimental and may change. Please contact support@wandb.com for guidance and to report any issues.wandb: setting up run ktw480bxwandb: Tracking run with wandb version 0.23.1wandb: Run data is saved locally in /fast/home/mihir.mahajan/Projects/miles/wandb/run-20251224_151320-ktw480bxwandb: Run `wandb offline` to turn off syncing.wandb: Syncing run qwen3-0.6b-sft-torchrun_rcovlno3-RANK_0wandb: ⭐️ View project at https://wandb.ai/instant-uv/crowd-pilot-mileswandb: 🚀 View run at https://wandb.ai/instant-uv/crowd-pilot-miles/runs/ktw480bxwandb: Detected [openai] in use.wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/[2025-12-24 15:15:09] train_sft.py:219 - [Rank 0] Data source initialized: dataset_size=10394, num_rollout=10000Flash Attention 3 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("meta-llama/Llama-3.2-1B", attn_implementation="flash_attention_3", dtype=torch.float16)`Flash Attention 3 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3Model is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("meta-llama/Llama-3.2-1B", attn_implementation="flash_attention_3", dtype=torch.float16)`[2025-12-24 15:15:09] train_sft.py:219 - [Rank 1] Data source initialized: dataset_size=10394, num_rollout=10000Flash Attention 3 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3ForCausalLM is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("meta-llama/Llama-3.2-1B", attn_implementation="flash_attention_3", dtype=torch.float16)`Flash Attention 3 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen3Model is torch.float32. You should run training or inference using Automatic Mixed-Precision via the `with torch.autocast(device_type='torch_device'):` decorator, or load the model with the `dtype` argument. Example: `model = AutoModel.from_pretrained("meta-llama/Llama-3.2-1B", attn_implementation="flash_attention_3", dtype=torch.float16)`[2025-12-24 15:15:10] train_sft.py:300 - [Rank 0] Applied LoRA: LoRAConfig(lora_rank=8, lora_alpha=16, lora_dropout=0.0, target_modules=['q_proj', 'v_proj'], bias='none')[2025-12-24 15:15:10] actor.py:1103 - FSDP MixedPrecision Policy: param_dtype=torch.bfloat16, reduce_dtype=torch.float32[2025-12-24 15:15:10] train_sft.py:300 - [Rank 1] Applied LoRA: LoRAConfig(lora_rank=8, lora_alpha=16, lora_dropout=0.0, target_modules=['q_proj', 'v_proj'], bias='none')[2025-12-24 15:15:10] actor.py:1103 - FSDP MixedPrecision Policy: param_dtype=torch.bfloat16, reduce_dtype=torch.float32hai006:3217574:3217574 [0] NCCL INFO Bootstrap: Using bond0:10.86.2.60<0>hai006:3217574:3217574 [0] NCCL INFO cudaDriverVersion 12080hai006:3217574:3217574 [0] NCCL INFO NCCL version 2.27.3+cuda12.9hai006:3217574:3217574 [0] NCCL INFO Comm config Blocking set to 1hai006:3217574:3218827 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. hai006:3217574:3218827 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB bond0:10.86.2.60<0>hai006:3217574:3218827 [0] NCCL INFO Initialized NET plugin IBhai006:3217574:3218827 [0] NCCL INFO Assigned NET plugin IB to commhai006:3217574:3218827 [0] NCCL INFO Using network IBhai006:3217574:3218827 [0] NCCL INFO DMA-BUF is available on GPU device 0hai006:3217574:3218827 [0] NCCL INFO ncclCommInitRankConfig comm 0x55edbfcbd4e0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 9b000 commId 0xbdfa8db6c0da816f - Init STARThai006:3217576:3217576 [1] NCCL INFO cudaDriverVersion 12080hai006:3217576:3217576 [1] NCCL INFO Bootstrap: Using bond0:10.86.2.60<0>hai006:3217576:3217576 [1] NCCL INFO NCCL version 2.27.3+cuda12.9hai006:3217576:3217576 [1] NCCL INFO Comm config Blocking set to 1hai006:3217576:3218902 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. hai006:3217576:3218902 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB bond0:10.86.2.60<0>hai006:3217576:3218902 [1] NCCL INFO Initialized NET plugin IBhai006:3217576:3218902 [1] NCCL INFO Assigned NET plugin IB to commhai006:3217576:3218902 [1] NCCL INFO Using network IBhai006:3217576:3218902 [1] NCCL INFO DMA-BUF is available on GPU device 1hai006:3217576:3218902 [1] NCCL INFO ncclCommInitRankConfig comm 0x559f1d7ba1e0 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId bb000 commId 0xbdfa8db6c0da816f - Init STARThai006:3217576:3218902 [1] NCCL INFO RAS client listening socket at ::1<28028>hai006:3217574:3218827 [0] NCCL INFO RAS client listening socket at ::1<28028>hai006:3217576:3218902 [1] NCCL INFO Bootstrap timings total 0.008132 (create 0.000029, send 0.006837, recv 0.000800, ring 0.000041, delay 0.000001)hai006:3217574:3218827 [0] NCCL INFO Bootstrap timings total 0.413873 (create 0.000031, send 0.000130, recv 0.413116, ring 0.000019, delay 0.000001)hai006:3217576:3218902 [1] NCCL INFO MNNVL busId 0xbb000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0hai006:3217574:3218827 [0] NCCL INFO MNNVL busId 0x9b000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0hai006:3217574:3218827 [0] NCCL INFO Setting affinity for GPU 0 to 35,147hai006:3217576:3218902 [1] NCCL INFO Setting affinity for GPU 1 to 35,147hai006:3217574:3218827 [0] NCCL INFO comm 0x55edbfcbd4e0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0hai006:3217576:3218902 [1] NCCL INFO comm 0x559f1d7ba1e0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0hai006:3217576:3218902 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1hai006:3217574:3218827 [0] NCCL INFO Channel 00/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 01/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 02/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 03/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 04/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 05/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 06/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 07/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 08/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 09/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 10/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 11/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 12/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 13/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 14/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 15/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 16/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 17/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 18/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 19/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 20/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 21/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 22/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Channel 23/24 : 0 1hai006:3217574:3218827 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -1/-1/-1->0->1 [10] -1/-1/-1->0->1 [11] -1/-1/-1->0->1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] -1/-1/-1->0->1 [19] -1/-1/-1->0->1 [20] -1/-1/-1->0->1 [21] -1/-1/-1->0->1 [22] -1/-1/-1->0->1 [23] -1/-1/-1->0->1hai006:3217574:3218827 [0] NCCL INFO P2P Chunksize set to 524288hai006:3217576:3218902 [1] NCCL INFO P2P Chunksize set to 524288hai006:3217574:3218827 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. hai006:3217574:3218827 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0hai006:3217574:3218914 [0] NCCL INFO [Proxy Service] Device 0 CPU core 147hai006:3217576:3218902 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. hai006:3217574:3218915 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 147hai006:3217576:3218916 [1] NCCL INFO [Proxy Service] Device 1 CPU core 147hai006:3217576:3218917 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 147hai006:3217576:3218902 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512hai006:3217576:3218902 [1] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peerhai006:3217574:3218827 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512hai006:3217574:3218827 [0] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peerhai006:3217574:3218827 [0] NCCL INFO CC Off, workFifoBytes 1048576hai006:3217574:3218827 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.hai006:3217574:3218827 [0] NCCL INFO ncclCommInitRankConfig comm 0x55edbfcbd4e0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 9b000 commId 0xbdfa8db6c0da816f - Init COMPLETEhai006:3217574:3218827 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.84 (kernels 0.27, alloc 0.10, bootstrap 0.41, allgathers 0.00, topo 0.02, graphs 0.00, connections 0.01, rest 0.02)hai006:3217576:3218902 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.hai006:3217576:3218902 [1] NCCL INFO ncclCommInitRankConfig comm 0x559f1d7ba1e0 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId bb000 commId 0xbdfa8db6c0da816f - Init COMPLETEhai006:3217576:3218902 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.46 (kernels 0.28, alloc 0.12, bootstrap 0.01, allgathers 0.00, topo 0.02, graphs 0.00, connections 0.01, rest 0.02)hai006:3217574:3218919 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 10/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 11/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 12/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 13/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 14/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 15/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 16/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 17/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 18/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 19/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 20/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 21/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 22/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Channel 23/0 : 1[1] -> 0[0] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217574:3218919 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/CUMEMhai006:3217576:3218918 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1hai006:3217574:3218919 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1[2025-12-24 15:15:12] train_sft.py:323 - [Rank 1] Model initialized with FSDP[2025-12-24 15:15:12] train_sft.py:332 - [Rank 1] LoRA: 1,146,880 trainable params out of 597,196,800 total (0.19%)[2025-12-24 15:15:12] lr_scheduler.py:80 - > learning rate decay style: WSD[2025-12-24 15:15:12] checkpoint.py:121 - [FSDP] No tracker file at /fast/project/HFMI_SynergyUnit/tab_model/huggingface/Qwen3-0.6B/latest_checkpointed_iteration.txt; skipping load./fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.   warnings.warn(  # warn only once[2025-12-24 15:15:12] train_sft.py:323 - [Rank 0] Model initialized with FSDP[2025-12-24 15:15:12] train_sft.py:332 - [Rank 0] LoRA: 1,146,880 trainable params out of 597,196,800 total (0.19%)[2025-12-24 15:15:12] lr_scheduler.py:80 - > learning rate decay style: WSD[2025-12-24 15:15:12] checkpoint.py:121 - [FSDP] No tracker file at /fast/project/HFMI_SynergyUnit/tab_model/huggingface/Qwen3-0.6B/latest_checkpointed_iteration.txt; skipping load./fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.   warnings.warn(  # warn only once[2025-12-24 15:15:12] train_sft.py:102 - [Rank 0] SFTTrainer initialized successfully[2025-12-24 15:15:12] train_sft.py:102 - [Rank 1] SFTTrainer initialized successfully[2025-12-24 15:15:12] train_sft.py:714 - [Rank 0] Starting training: rollout_id 0 -> 10000[2025-12-24 15:15:12] train_sft.py:714 - [Rank 1] Starting training: rollout_id 0 -> 10000==================================================================================================== -n slurm/jobs/mihir/horeka/yolo-runs/sampling.sh | sed -n '18,30p'```
 -n slurm/jobs/mihir/horeka/yolo-runs/sampling.sh```
 -n slurm/jobs/mihir/horeka/yolo-runs/sampling.sh | ==================================================================================================== -n slurm/jobs/mihir/horeka/yolo-runs/sampling.sh | sed -n '18,30p'```
 -n slurm/jobs/mihir/horeka/yolo-runs/sampling.sh```
 -n slurm/jobs/mihir/horeka/yolo-runs/sampling.sh | actor_val:   0%|          | 0/100 [00:00<?, ?it/s]Traceback (most recent call last):  File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 769, in <module>    main()  File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 765, in main    trainer.train()  File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 724, in train    self.calculate_val_loss(rollout_id=0)  File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 644, in calculate_val_loss    samples = self.generate_sft_rollout(rollout_id, self.val_data_source)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 371, in generate_sft_rollout    full_text = tok.decode(token_ids)                ^^^NameError: name 'tok' is not defined[rank1]: Traceback (most recent call last):[rank1]:   File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 769, in <module>[rank1]:     main()[rank1]:   File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 765, in main[rank1]:     trainer.train()[rank1]:   File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 724, in train[rank1]:     self.calculate_val_loss(rollout_id=0)[rank1]:   File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 644, in calculate_val_loss[rank1]:     samples = self.generate_sft_rollout(rollout_id, self.val_data_source)[rank1]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^[rank1]:   File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 371, in generate_sft_rollout[rank1]:     full_text = tok.decode(token_ids)[rank1]:                 ^^^[rank1]: NameError: name 'tok' is not defined[rank0]: Traceback (most recent call last):[rank0]:   File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 769, in <module>[rank0]:     main()[rank0]:   File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 765, in main[rank0]:     trainer.train()[rank0]:   File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 724, in train[rank0]:     self.calculate_val_loss(rollout_id=0)[rank0]:   File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 644, in calculate_val_loss[rank0]:     samples = self.generate_sft_rollout(rollout_id, self.val_data_source)[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^[rank0]:   File "/fast/home/mihir.mahajan/Projects/miles/train_sft.py", line 371, in generate_sft_rollout[rank0]:     full_text = tok.decode(token_ids)[rank0]:                 ^^^[rank0]: NameError: name 'tok' is not definedwandb: wandb: 🚀 View run qwen3-0.6b-sft-torchrun_rcovlno3-RANK_0 at: https://wandb.ai/instant-uv/crowd-pilot-miles/runs/ktw480bxwandb: Find logs at: ../../../../fast/home/mihir.mahajan/Projects/miles/wandb/run-20251224_151320-ktw480bx/logs[rank1]:[W1224 15:15:14.943348424 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())hai006:3217576:3218928 [1] NCCL INFO misc/socket.cc:64 -> 3hai006:3217576:3218928 [1] NCCL INFO misc/socket.cc:81 -> 3hai006:3217576:3218928 [1] NCCL INFO misc/socket.cc:863 -> 3hai006:3217576:3218916 [1] NCCL INFO misc/socket.cc:915 -> 3hai006:3217576:3218928 [1] NCCL INFO misc/socket.cc:64 -> 3hai006:3217576:3218928 [1] NCCL INFO misc/socket.cc:81 -> 3hai006:3217576:3218928 [1] NCCL INFO misc/socket.cc:863 -> 3hai006:3217576:3218928 [1] NCCL INFO comm 0x559f1d7ba1e0 rank 1 nranks 2 cudaDev 1 busId bb000 - Abort COMPLETE[rank0]:[W1224 15:15:15.788134362 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())hai006:3217574:3218931 [0] NCCL INFO misc/socket.cc:64 -> 3hai006:3217574:3218931 [0] NCCL INFO misc/socket.cc:81 -> 3hai006:3217574:3218931 [0] NCCL INFO misc/socket.cc:863 -> 3hai006:3217574:3218914 [0] NCCL INFO misc/socket.cc:915 -> 3hai006:3217574:3218931 [0] NCCL INFO misc/socket.cc:64 -> 3hai006:3217574:3218931 [0] NCCL INFO misc/socket.cc:81 -> 3hai006:3217574:3218931 [0] NCCL INFO misc/socket.cc:863 -> 3hai006:3217574:3218931 [0] NCCL INFO comm 0x55edbfcbd4e0 rank 0 nranks 2 cudaDev 0 busId 9b000 - Abort COMPLETEW1224 15:15:15.860000 3217568 torch/distributed/elastic/multiprocessing/api.py:900] Sending process 3217574 closing signal SIGTERME1224 15:15:16.427000 3217568 torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 1 (pid: 3217576) of binary: /fast/home/mihir.mahajan/Projects/miles/.venv/bin/python3Traceback (most recent call last):  File "/fast/home/mihir.mahajan/Projects/miles/.venv/bin/torchrun", line 10, in <module>    sys.exit(main())             ^^^^^^  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper    return f(*args, **kwargs)           ^^^^^^^^^^^^^^^^^^  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/distributed/run.py", line 901, in main    run(args)  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/distributed/run.py", line 892, in run    elastic_launch(  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 143, in __call__    return launch_agent(self._config, self._entrypoint, list(args))           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 277, in launch_agent    raise ChildFailedError(torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================train_sft.py FAILED------------------------------------------------------------Failures:  <NO_OTHER_FAILURES>------------------------------------------------------------Root Cause (first observed failure):[0]:  time      : 2025-12-24_15:15:15  host      : hai006.haicore.berlin  rank      : 1 (local_rank: 1)  exitcode  : 1 (pid: 3217576)  error_file: <N/A>  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html============================================================srun: error: hai006: task 0: Exited with exit code 1(miles) [mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ python test.py/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.  import pynvml  # type: ignore[import]=== NO THINKING ===
systemYou are helpful.

userWhat is 2 + 2?

assistant<think>Hm. I think 2 + 2 must be 4</think>The answer is 4.
=== DEFAULT (thinking) ===
systemYou are helpful.

userWhat is 2 + 2?

assistant<think>Hm. I think 2 + 2 must be 4</think>The answer is 4.
(miles) [mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ python test.py/fast/home/mihir.mahajan/Projects/miles/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.  import pynvml  # type: ignore[import]=== NO THINKING ===
systemYou are helpful.

userWhat is 2 + 2?

assistant<think></think>The answer is 4.
=== DEFAULT (thinking) ===
systemYou are helpful.

userWhat is 2 + 2?

assistant<think></think>The answer is 4.
(miles) [mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ git diff(miles) [mihir.mahajan@hai006.haicore.berlin:~/Projects/miles] $ salloc: Job 38556 has exceeded its time limit and its allocation has been revoked.                                                                                                                                                   srun: Job step aborted: Waiting up to 32 seconds for job step to finish.[2025-12-24T15:44:51.011] error: *** STEP 38556.interactive ON hai006 CANCELLED AT 2025-12-24T15:44:51 DUE TO TIME LIMIT ***srun: error: hai006: task 0: Killed