Here is the past conversation with you amswer the last request : "Do I fine tune GPT-2 on the entirety of wikipedia and then finetune it again to be able to follow instructions and be a conventional AI or do I just do the latter?
Fine-tuning GPT-2 on the entirety of Wikipedia will give it a broad base of knowledge but may not specifically make it better at following instructions or being a conventional AI. Instead, you should focus on gathering a dataset that contains examples of following instructions and conventional AI interactions. Then, fine-tune GPT-2 on this specific dataset.

If possible, begin with a GPT-2 model that has already been pre-trained on a large dataset like Wikipedia or use a more advanced version like GPT-3. This will save you time and computational resources.

In summary, prioritizing fine-tuning on a dataset of instructions and conventional AI-like interactions is more efficient than training on the entirety of Wikipedia followed by training on the specific dataset.
If I just fine tune it on just following instructions and be a conversational AI would it be able to have a good knowledge base
Fine-tuning GPT-2 on a dataset focusing on following instructions and being a conversational AI would provide it a more targeted skill set. However, for it to have a good knowledge base, it needs underlying general knowledge, which it can obtain from being pre-trained on a large dataset like Wikipedia or another extensive and diverse dataset.

If you start with a model like GPT-2 or GPT-3 that has already been pre-trained on a large corpus like Wikipedia, this would have a good knowledge base in addition to the targeted conversational and instructions-following ability you’d like to achieve.

To ensure your model retains a good knowledge base while being skilled at following instructions and being a conversational AI, it’s advised to use a pre-trained model as a starting point and then fine-tune it with your desired dataset.
Is GPT-2 already trained on wikipedia ?
Yes, GPT-2 has been pre-trained on a large dataset that includes portions of Wikipedia along with other text content from the internet, such as books, articles, and web pages. This pre-training helps GPT-2 have a good knowledge base about various topics. When you fine-tune GPT-2 on a specific task like following instructions or being conversational, it retains the knowledge acquired during the pre-training phase while also learning new skills to perform better on the specific task.
If I fine tune it on the entirety of wikipedia it wouldn’t forget its previous training
If you fine-tune GPT-2 on the entirety of Wikipedia, it is likely to retain most of its previous knowledge because the training is based on the same dataset. However, fine-tuning still causes some learning updates on the weights of the model, which might result in changes to the previously learned features. As a result, it will not entirely forget its previous training, but there might be slight shifts in knowledge representation.

Generally, it is unnecessary to fine-tune GPT-2 on the entirety of Wikipedia, as the benefits may be marginal. It makes more sense to fine-tune GPT-2 on a specific domain or dataset that helps your model follow instructions or function better as a conversational AI. This approach will enable your model to retain its general knowledge while becoming more efficient at a precise task.
Because this was the response to try and test its factual accuracy here is it: “King Henry VIII died in exile; his son Charles IV was succeeded by a younger son, Queen Mary, who also died at that time. Charles died in 1589; the son of Edward IV of England was appointed Prince of Normandy.”"