Fine-Tuning GPT-Neox-20B using Hugging Face Transformers
#16
by
Dulanjaya
- opened
Hi, I am new to GPT NeoX 20B,
Can you please explain whether I can fine-tune the slim version (40GB) of this model on 2xA6000 GPUs using the transformers library?
Thank you!
As far as I know, you need atleast 42GB free memory to load the model checkpoint with low_cpu_mem_usage=True argument. For finetuning you might need even more.