prompter: Which affordable GPU would you recommend to train a language model?

assistant: It is diffucult to say which GPU fits best for your usecase since larger language models typically require more computational power and VRAM than smaller ones. A recommendation would be the 24GB version of the NVIDIA Tesla M40 since it has a lot of VRAM and can be bought used on sites like https://ebay.com/ for below 150$, however it is worth mentioning that this GPU was launched all the way back in November 10th, 2015 so you shouldnt expect groundbreaking performance. If your budged is a little higher, I could also recommend the NVIDIA Tesla K80 24GB.

prompter: I want to finetune GPT-J-6B, would those be good enough for that?

prompter: How long would it take to fine tune a gpt-2 model?

assistant: The time it would take to fine tune a GPT-2 model will vary significantly depending what you are tuning it for. Keep in mind that GPT-2 XL is a very large language model with over 1.5 billion parameters, however GPT-2 is also available in smaller sizes that will be faster to fine tune but will likely not perform as well.

assistant: This depends on the size of the model, the size of your dataset, and the GPU you're using. However, most GPT-2 finetunes don't take longer than a day to complete.

assistant: It heavily depends on the size and architecture of the model you are looking to train! What type of model are you interested in training? If you'd like to work backwards to a model from a budget, let me know and I will do my best to find the best graphics card for training machine learning models with your money, and then recommend a model and accompanying size.

prompter: I would like to train a model like GPT-J-6B for conversations like ChatGPT

assistant: The most affordable GPU for training a language model greatly depends on your specific requirements, the largest factor being the amount of GPU memory needed. According to Tim Dettmers at https://timdettmers.com/2023/01/16/which-gpu-for-deep-learning/ , the most cost-efficient (relative to performance) GPU for deep learning is the RTX 4070 Ti (12 GB). Cheaper options include used RTX 3060 GPUs which also have 12GB of RAM, but lower inference and training performance. As a rough guide, you will need at least 12GB for image generation and at least 24GB to work with transformers which are common for language models. For language models, that essentially forces you to get an RTX 3090, 3090Ti, or 4090 due to the 24GB of GPU memory they have. 

Large Language Models with more than 13B parameters often require much more powerful cards in terms of GPU memory capacity, or multiple cards, to train, thus not being affordable. However, there are lots of improvements happening the deep learning scene to optimize models such as 8-bit inference and training, and the dynamics of the market change frequently, thus I would recommend active research to get the latest information possible.

prompter: That's out of my budget, I'm afraid. Is it possible to train a language model without buying a GPU?

assistant: You can rent GPU time in the cloud from companies such as AWS, Google Cloud, vast.ai, Paperspace, and others. Alternatively, if your model is small enough, you might be able to train it on your current GPU or using services with free tiers such as Google Colab or Kaggle.

prompter: How long will it take to train my own chat gbt with Google Colab’s free services?

assistant: Since Colabs free services are limited to time and computing power, you could only train a small model which may not be as capable as a bigger model. Since smaller models dont need as much time to train as bigger models, it shouldnt take too long, depending on your dataset of course!

assistant: Google Colab is not sufficient to train a model on the scale of ChatGPT. However, you can use it to prompt-tune (a weaker form of fine-tuning) a model on the scale of GPT-J with 6 billion parameters, thanks to their TPU (Tensor Processing Unit) offering. Alternatively, you can fully finetune a model, but that would require more VRAM and thus can only be done with smaller models, such as some versions of GPT-2.

prompter: But I would like to host my stuff locally so I am the one in charge to make my stuff work relieably. I found a cheap NVIDIA Tesla M40 24GB on ebay for ~150€. Is that a good deal and can it be used for my purpose?

assistant: It depends on how big of a model you are trying to train. You can train small models on CPU but they have very limited usability. Good language models require processing of large amounts of textual data in the order of 100s of GB. Processing this much data without GPUs is not very practical. So to answer your question, it depends on the size of the model you are trying to train. To train a model that could be used practically, no it is not possible to do so without GPUs.