In a published research paper, researchers described generative pretraining as the ability to train language models with unlabeled data and achieve accurate prediction. The first GPT model, GPT-1, was developed in 2018. GPT-4 was introduced in March 2023 as a successor to GPT-3.
GPT-3 was trained with over 175 billion parameters or weights. Engineers trained it on over 45 terabytes of data from sources like web texts, Common Crawl, books, and Wikipedia. Prior to training, the average quality of the datasets was improved as the model matured from version 1 to version 3.
GPT-3 trained in a semi-supervised mode. First, machine learning engineers fed the deep learning model with the unlabeled training data. GPT-3 would understand the sentences, break them down, and reconstruct them into new sentences. In unsupervised training, GPT-3 attempted to produce accurate and realistic results by itself. Then, machine learning engineers would fine-tune the results in supervised training, a process known as reinforcement learning with human feedback (RLHF).
You can use the GPT models without any further training, or you can customize them with a few examples for a particular task.