| # Jam-CGPT | |
| Jam-CGPT is a GPT2-like model that follows [jam](https://huggingface.co/apcl/jam)'s pretraining procedure to pretrain models ranging from 38 million to 350 million parameters and finetuning with comments generated by GPT-3.5 and data size ranging from 170k to 2.15m. | |
| ## Jam-CGPT Training Details | |
| - We follow [jam](https://huggingface.co/apcl/jam)'s pretraining procedure and use the same data to pretrain our 38m, 110m and 350m parameters models. | |
| - We finetune our Jam-CGPT with the summaries generated by GPT-3.5 and 4 different dataset size [Jam-CGPT dataset](https://huggingface.co/datasets/apcl/Jam-CGPT). | |
| - We finetune our models for 3 epochs. | |
| - Our [GitHub repo](https://github.com/apcl-research/Jam-CGPT) contains the code for reproduction using the same [data](https://huggingface.co/datasets/apcl/Jam-CGPT). | |
| ## Jam-CGPT 38 million parameters model | |
| | Hyperparameter | Description | Value | | |
| | ----------- | ----------- |------------| | |
| |e | embedding dimensions | 512 | | |
| |L | number of layers | 4 | | |
| |h | attention heads | 4 | | |
| |c | block size / context length | 256 | | |
| |b | batch size | 64 | | |
| |a | accumulation steps | 2 | | |
| |d | dropout | 0.20 | | |
| |r | learning rate | 3e-5 | | |
| |y | iterations | 1e-5 | | |
| |iter | number of iterations after pretraing | 757,000 | | |
| ## Jam-CGPT 110 million parameters model | |
| | Hyperparameter | Description | Value | | |
| | ----------- | ----------- |------------| | |
| |e | embedding dimensions | 768 | | |
| |L | number of layers | 10| | |
| |h | attention heads | 8 | | |
| |c | block size / context length | 256 | | |
| |b | batch size | 32 | | |
| |a | accumulation steps | 4 | | |
| |d | dropout | 0.20 | | |
| |r | learning rate | 3e-5 | | |
| |y | iterations | 1e-5 | | |
| |iter | number of iterations after pretraing | 762,000 | | |
| ## Jam-CGPT 350 million parameters model | |
| | Hyperparameter | Description | Value | | |
| | ----------- | ----------- |------------| | |
| |e | embedding dimensions | 1024 | | |
| |L | number of layers | 24 | | |
| |h | attention heads | 16 | | |
| |c | block size / context length | 256 | | |
| |b | batch size | 4 | | |
| |a | accumulation steps | 32 | | |
| |d | dropout | 0.20 | | |
| |r | learning rate | 3e-5 | | |
| |y | weight decay | 1e-5 | | |
| |iter | iterations | 272,000 | | |
| - Note that you can adjust the batch size and accumulation steps based on your GPU memory. But, the batch size * accumulation steps should be 128. | |
| - If you finetune your models with multiple GPUs, you can turn down accumulation steps. For example, if you finetune with 2 GPUs, you will need to half the accumulation steps. | |
| - We pretrained 38m and 110m models for 3 epochs. | |