[content] calculations_grant.md (markdown)
mihir
mihirma15
Online

mihir — 7/23/25, 5:25 PM
hahah fk
Alfred — 7/23/25, 5:25 PM
Egal haha
mihir — 7/23/25, 5:26 PM
vlt sushi oderso
wenn du bock hast
Alfred — 7/23/25, 5:26 PM
Jo safe
Alfred — 7/23/25, 6:17 PM
sag nicht franz
Image
mihir — 7/23/25, 6:18 PM
funktioniert der reshape hahah
mihir — 7/25/25, 6:59 PM
willst mal schnell drüberschauen wenn du zeit hast
Attachment file type: acrobat
Resume_with_about.pdf
99.53 KB
ich hab das so verlernt
ich schwöre ich war so während amazon so gut in lebenslauf und so reden und so
jz einfach hahn geworden
zurückentwickelt wieder
erinner mich nicht mehr was ich damals präsentiert hab einfach hahah
Alfred — 7/25/25, 8:34 PM
Hahhaha
Alfred — 7/25/25, 8:34 PM
Ja safe kann ich machen
mihir — 7/25/25, 8:39 PM
korrekt dikker
mihir — 8/1/25, 3:59 PM
bro haha wir sind keine dc freunde
Alfred — 8/1/25, 3:59 PM
Hhahahahahahahahahha
mihir — 8/4/25, 4:24 PM
kannst du mir die gemini convo shcicken iwie
screenshot oder hier reinpaseten
also von vid2world
Alfred — 8/4/25, 4:26 PM
https://aistudio.google.com/prompts/16n5osT0V-je4ZFBxUnxObrJ60Dv0id5m
Sign in - Google Accounts
Geht das?
https://aistudio.google.com/prompts/16n5osT0V-je4ZFBxUnxObrJ60Dv0id5m
Sign in - Google Accounts
mihir — 8/4/25, 4:26 PM
The requested prompt does not exist or you do not have access. If you believe the request is correct, make sure you have first allowed AI Studio access to your Google Drive, and then ask the owner to share the prompt with you
bei beiden
Alfred — 8/4/25, 4:26 PM
Of course. Here is a simple overview of the Vid2World architecture and how it enables "temporal teacher forcing" for its diffusion model.

### Simple Overview of the Architecture

The main goal of Vid2World is to convert a standard, pre-trained video diffusion model (which generates a whole video at once) into an **interactive world model**. An interactive world model needs to do two things differently:
Expand
message.txt
5 KB
mihir — 8/4/25, 4:27 PM
dnake man
Alfred — 8/4/25, 4:28 PM
👍
Stefan hat noch bis 17 Uhr ein call
mihir — 8/4/25, 4:32 PM
oh man
woher weißt
Alfred — 8/4/25, 4:34 PM
Image
Alfred — 8/4/25, 6:48 PM
Sollen wir so in einer halben stunde los?
mihir — 8/4/25, 6:57 PM
jo
mihir — Yesterday at 2:44 PM
hätten das mit franz besprechen sollen bevor in stefan gruppe schreiben
meiner coach
Alfred — Yesterday at 2:47 PM
Achso haha
Awa
Kurzer einblick in unser chaos haha
mihir — Yesterday at 2:47 PM
ahahah
mihir — Yesterday at 4:24 PM
sollen wir zu franz
oder kack drauf
Alfred — Yesterday at 4:46 PM
`ValueError: User-provided restore item and on-disk value metadata tree structures do not match: {'model': {'decoder': {'blocks': {'0': {'spatial_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None), 'temporal_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None)}, '1': {'spatial_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None), 'temporal_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None)}, '2': {'spatial_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None), 'temporal_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None)}, '3': {'spatial_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None), 'temporal_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None)}}, 'pos_enc': Diff(lhs=None, rhs={'pe': {'value': ValueMetadataEntry(value_type='jax.Array', skip_deserialize=False, write_shape=(625, 512))}})}, 'encoder': {'blocks': {'0': {'spatial_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None), 'temporal_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None)}, '1': {'spatial_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None), 'temporal_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None)}, '2': {'spatial_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None), 'temporal_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None)}, '3': {'spatial_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None), 'temporal_pos_enc': Diff(lhs={'pe': {'value': ShapeDtypeStruct(shape=(5000, 512), dtype=float32, sharding=NamedSharding(mesh=Mesh('data': 1, axis_types=(Auto,)), spec=PartitionSpec(), memory_kind=device))}}, rhs=None)}}, 'pos_enc': Diff(lhs=None, rhs={'pe': {'value': ValueMetadataEntry(value_type='jax.Array', skip_deserialize=False, write_shape=(625, 512))}})}}}`
message.txt
4 KB
mihir — Yesterday at 4:59 PM
--dyna_dim=128 \
    --dyna_num_blocks=2 \
    --dyna_num_heads=4
#!/usr/bin/env bash

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=01:00:00
#SBATCH --partition=accelerated
#SBATCH --cpus-per-task=5
#SBATCH --gres=gpu:1
#SBATCH --output=/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/logs/logs_mihir/yoloruns/%x_%j.log
#SBATCH --error=/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/logs/logs_mihir/yoloruns/%x_%j.log
#SBATCH --job-name=train_dynamics_overfit_sample_causal

# Log the sbatch script
cat $0

module unload mpi/openmpi/5.0
module unload devel/cuda/12.4
# source .venv/bin/activate

array_records_dir=/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/data_new/open_ai_minecraft_arrayrecords_chunked

job_name=$SLURM_JOB_NAME
slurm_job_id=$SLURM_JOB_ID

CHECKPOINT_DIR=$ws_dir/checkpoints/causal/overfit-seed69-1-no-noise/$job_name/$slurm_job_id
mkdir -p $CHECKPOINT_DIR

tokenizer_ckpt_dir=/hkfs/work/workspace/scratch/tum_ind3695-jafa_ws_shared/checkpoints/big-runs/tokenizer/train_tokenizer_1e-4/3388151


env | grep SLURM

srun python train_dynamics.py \
    --save_ckpt \
    --seed=69 \
    --num_steps=5000 \
    --warmup_steps=0 \
    --wsd_decay_steps=0 \
    --ckpt_dir $CHECKPOINT_DIR \
    --batch_size=8 \
    --init_lr=1e-4 \
    --max_lr=1e-4 \
    --log_image_interval=100 \
    --log \
    --log_checkpoint_interval=1000 \
    --name=dynamics-causal-overfit-no-noise-$slurm_job_id \
    --tags dynamics causal overfit \
    --entity instant-uv \
    --project jafar \
    --tokenizer_checkpoint=$tokenizer_ckpt_dir \
    --data_dir $array_records_dir \
    --no-use-flash-attention \
    --dyna_dim=128 \
    --dyna_num_blocks=2 \
    --dyna_num_heads=4
 
mihir — Yesterday at 7:13 PM
cursor explanation vom error
Ctrl+K to generate a command
what is this? go through the code step by step and try to explain it

You are encountering the following error repeatedly during your training run with JAX/Flax:
Apply to sample.py
Let's break down what this means, step by step, and how it relates to your code.
Expand
message.txt
4 KB
mihir — Yesterday at 8:35 PM
jnp.ascontiguousarray
mihir — 12:06 AM
AKIAQZFG433QQZOA73O5
SDk0HDPb3lpyh3UYGr4m5jzThH/LolOzAugW+q0J
mihir — 3:21 PM
LR scheduler plots
Image
modelsize scaling
Image
mihir — 3:30 PM
batch size scaling
Image
Image
mihir — 4:11 PM
589824
Alfred — 4:31 PM
80         159,2 - 123,3s  45 steps|
180        281,7 - 157,8s  45 steps|
255        336,8 - 161,0s  45 steps|
Image
Alfred — 4:54 PM
https://docs.google.com/spreadsheets/d/1DDX-W0TPxwNnUxmiTJOX7qyhf6zKYCwE7s81JObZNWA/edit?usp=sharing
Google Docs
Grant proposal
Image
mihir — 5:42 PM
https://wandb.ai/instant-uv/jafar/workspace/panel/uu0fjklvm?nw=nwusermihirmahajan2002
W&B
instant-uv
Weights & Biases, developer tools for machine learning
Image
Alfred — 5:59 PM
# Grant Proposal

## Graphics


1. Troughput (number of gpu to number of seq/tokens in 15 min)
Expand
message.txt
5 KB
﻿
Alfred
alfred3772
 
# Grant Proposal

## Graphics


1. Troughput (number of gpu to number of seq/tokens in 15 min)
   - 1 node to 16 nodes/or anything bigger than that

2. Batch size scaling plots

3. LR scheduler plots

4. model size scaling plots

5. compare with genie 1,2,3, minerl, oasis, veo??
   - extrapolate to our model

6. Sub project on 3 archtectures? (Causal, Maskgit, Diffusion)
   - worl model pretraining

     -  modelsize scaling (avg: 12nodes, 8 days, 5 model sizes)
     -  want to do avg 16 nodes 10 days, 7 model sizes


     -  lr scheduler scaling
     -  action scaling
     -  batch size scaling

     -  context length scaling/memory
     -  higher resolution
     -  higher fps
     -  other gameplay videos 
     -  other general purpose videos
   - agent training in the wm

     -  modelsize scaling (avg: 12nodes, 8 days, 5 model sizes)
     -  want to do avg 16 nodes 10 days, 7 model sizes
7. 

8. 

9. 

10. 



## Justification Table 

Training jobs: 
- Tokenizer
- LAM
- Dynamics model causal 
- Dynamics model diffusion
- Dynamics model maskgit


- Agent training

Datasets
- OAI VPT
- Ours 


 Sub-project Type of run  Dataset  # runs  # steps/run  Wall time/step [seconds]  num nodes  time         GPU hours  Cpu hours [core-h] 

| Sub-project             | Type of run | Dataset | # runs | # steps/run | Wall time/step [seconds] | num nodes | time         | GPU hours | Cpu hours [core-h] |
| ----------------------- | ----------- | ------- | ------ | ----------- | ------------------------ | --------- | ------------ | --------- | ------------------ |
|                         |             |         |        |             |                          |           |              |           |                    |
|                         |             |         |        |             |                          |           |              |           |                    |
| Batch size scaling rung | Dynamics    | OAI     | 1      | 200,000     | 3,4s **TODO**            | 127       | 8*24H = 192h | 97536h    | /                  |
| Model size scaling rung | Dynamics    | OAI     | 1      | 200,000     | 3,4s **TODO**            | 127       | 8*24H = 192h | 97536h    | /                  |
|                         |             |         |        |             |                          |           |              |           |                    |
|                         |             |         |        |             |                          |           |              |           |                    |
|                         |             |         |        |             |                          |           |              |           |                    |
|                         |             |         |        |             |                          |           |              |           |                    |
|                         |             |         |        |             |                          |           |              |           |                    |
|                         |             |         |        |             |                          |           |              |           |                    |
|                         |             |         |        |             |                          |           |              |           |                    |



Model size  Num steps  Wall time/step [seconds]  num nodes  time         GPU hours  Cpu hours [core-h]

| model size | num steps | wall time/step [seconds] | num nodes | time         | GPU hours  | Cpu hours [core-h] |
| ---------- | --------- | ----------------------- | --------- | ------------ | ---------- | ------------------ |
80         159,2 - 123,3s  45 steps|
180        281,7 - 157,8s  45 steps|
255        336,8 - 161,0s  45 steps|


| Parameters | num_layers | num_heads | d_model | k/q size | training hardware | training time | FLOPs      |
| ---------- | ---------- | --------- | ------- | -------- | ----------------- | ------------- | ---------- |
| 41M        | 18         | 8         | 512     | 64       | TPUv2             | 3 days        | 2.05 ×1020 |

| 41M | 18  | 8   | 512  | 64  | 64 | TPUv2 | 3 days | 2.05 ×1020 |
| 96M | 16  | 16 768 64 64 TPUv2 6 days 3.58 ×1020
| 192M|  20 |  18 1024 64 64 TPUv2 9 days 6.4 ×1020
| 404M|  21 |  12 1536 128 64 TPUv2 18 days 1.2 ×1021
| 811M|  20 |  20 2048 128 128 TPUv3 7 days 2.2 ×1021
| 1.6B|  28 |  22 2560 128 128 TPUv3 12 days 4.04 ×1021
| 2.7B|  36 |  22 3072 128 256 TPUv3 16 days 6.91 ×1021
message.txt
5 KB