p-doom/crowd-code-dataset-0.2 huggingface dataset:
[tab] scripts/batch_sizes.md (markdown)
## Batchsizes for jafar with Minecraft dataset

### Tokenizer
| GPU VRAM | Batch Size | Learning Rate |
|----------|------------|---------------|
| -        | 1          | 4.3e-5        | 
| 12 GB    | 6          | 1e-4          | 
| 24 GB    | 12         | 1.5e-4        | ?? 
| 40 GB    | 12         |               |
| 48 GB    | 24         |               | ?? why does this work on cremers
| 80 GB    | 48         | 3e-4          |

### LAM
| GPU VRAM | Batch Size | Learning Rate |
|----------|------------|---------------|
| -        | 1          | 5e-6/5e-7     | 
| 12 GB    | 6          |               | 
| 24 GB    | 12         |               |
| 48 GB    | 24         |               |
| -        | 36         |  3e-5/3e-6    |
| 80 GB    | 48         |               |
[tab] scripts/train_tokenizer_overfit_batch.sbatch (shellscript)
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=00:30:00
#SBATCH --cpus-per-task=5
#SBATCH --gres=gpu:1,VRAM:24G
#SBATCH --mem=50G
#SBATCH --output=logs/logs_training/%x_%j.log
#SBATCH --error=logs/logs_training/%x_%j.log
#SBATCH --job-name=train_tokenizer_minecraft_overfit_batch

# Log the sbatch script
cat $0

tf_records_dir="/storage/user/mahajanm/Projects/world-modeling/knoms_tfrecords_500/"
ws_dir='/storage/user/mahajanm/Projects/world-modeling'
job_name=$SLURM_JOB_NAME
slurm_job_id=$SLURM_JOB_ID

CHECKPOINT_DIR=$ws_dir/checkpoints/$job_name_$slurm_job_id
mkdir -p $CHECKPOINT_DIR

env | grep SLURM

SLURM_STEP_NODELIST=$SLURM_NODELIST python train_tokenizer.py \
    --ckpt_dir $CHECKPOINT_DIR \
    --batch_size=1 \
    --min_lr=1e-4 \
    --max_lr=1e-4 \
    --log_image_interval=3 \
    --log \
    --entity instant-uv \
    --project jafar \
    --data_dir $tf_records_dir