[tab] scripts_horeka/learning_rates.md (markdown)

# Scaling requirements (sqrt lr scaling)

## Tokenizer
| Batch Size | Learning Rate (sqrt scaling) | # Nodes | # GPUs |
|------------|-----------------------------|---------|--------|
| 1          | 4.33e-05                  |    1    |   1    |
| 6          | 1.06e-04                  |    1    |   1    |
| 12         | 1.50e-04                  |    1    |   1    |
| 24         | 2.12e-04                  |    1    |   2    |
| 48         | 3.00e-04                  |    1    |   4    |
| 96         | 4.24e-04                  |    2    |   8    |

 Expanded for nodes: 1, 2, 4, 8, 16, 32 (assuming per-GPU batch size = 12)
| Batch Size | Learning Rate (sqrt scaling) | # Nodes | # GPUs |
|------------|-----------------------------|---------|--------|
| 48         | 3.00e-04                  |    1    |   4    |
| 96         | 4.24e-04                  |    2    |   8    |
| 192        | 6.00e-04                  |    4    |   16   |
| 384        | 8.49e-04                  |    8    |   32   |
| 768        | 1.20e-03                  |   16    |   64   |
| 1536       | 1.70e-03                  |   32    |  128   |

## LAM
| Batch Size | Learning Rate (sqrt scaling) | # Nodes | # GPUs |
|------------|-----------------------------|---------|--------|
| 1          | 5.00e-06                  |    1    |   1    |
| 6          | 1.22e-05                  |    1    |   1    |
| 12         | 1.73e-05                  |    1    |   1    |
| 24         | 2.45e-05                  |    1    |   2    |
| 36         | 3.00e-05                  |    1    |   3    |
| 48         | 3.46e-05                  |    1    |   4    |
| 96         | 4.90e-05                  |    2    |   8    |

 Expanded for nodes: 1, 2, 4, 8, 16, 32 (assuming per-GPU batch size = 12)
| Batch Size | Learning Rate (sqrt scaling) | # Nodes | # GPUs |
|------------|-----------------------------|---------|--------|
| 48         | 3.46e-05                  |    1    |   4    |
| 96         | 4.90e-05                  |    2    |   8    |
| 192        | 6.93e-05                  |    4    |   16   |
| 384        | 9.80e-05                  |    8    |   32   |
| 768        | 1.39e-04                  |   16    |   64   |
| 1536       | 1.96e-04                  |   32    |  128   |

## Dynamics
| Batch Size | Learning Rate (sqrt scaling) | # Nodes | # GPUs |
|------------|-----------------------------|---------|--------|
| 1          | 5.00e-06                  |    1    |   1    |
| 6          | 1.22e-05                  |    1    |   1    |
| 12         | 1.73e-05                  |    1    |   1    |
| 24         | 2.45e-05                  |    1    |   2    |
| 36         | 3.00e-05                  |    1    |   3    |
| 48         | 3.46e-05                  |    1    |   4    |
| 96         | 4.90e-05                  |    2    |   8    |

 Expanded for nodes: 1, 2, 4, 8, 16, 32 (assuming per-GPU batch size = 12)
| Batch Size | Learning Rate (sqrt scaling) | # Nodes | # GPUs |
|------------|-----------------------------|---------|--------|
| 48         | 3.46e-05                  |    1    |   4    |
| 96         | 4.90e-05                  |    2    |   8    |
| 192        | 6.93e-05                  |    4    |   16   |
| 384        | 9.80e-05                  |    8    |   32   |
| 768        | 1.39e-04                  |   16    |   64   |
| 1536       | 1.96e-04                  |   32    |  128   |