KnutJaegersberg/megatron-GPT-2-345m-EvolInstruct

Fine tuned small megatron-GPT-2 on Evol-Instruct dataset of WizardLM. Megatron-GPT-2-345m is competetive with GPT-2 large in some benchmarks. Megatron-GPT-2-345m performance on Wikitext-103 is 19.31, GPT2-large (1.5B) is 17.48 and GPT2-medium (762M) is 22.05.

Code example:

from transformers import AutoModel, GPT2LMHeadModel, AutoTokenizer

model2 = GPT2LMHeadModel.from_pretrained('KnutJaegersberg/megatron-GPT-2-345m-EvolInstruct')

tokenizer = AutoTokenizer.from_pretrained("KnutJaegersberg/megatron-GPT-2-345m-EvolInstruct")

text = "Below is an instruction that describes a task. Write a response that appropriately completes the request. \n\n### Instruction:\n Give me a list of 5 political commentators, compare and contrast their beliefs. \n### Response:\n" encoded_input = tokenizer(text, return_tensors='pt').input_ids.to('cuda:0')

from transformers import pipeline generator = pipeline('text-generation', model=model2, tokenizer=tokenizer, device=0)

generator(text, penalty_alpha=0.6, top_k=4, max_length=1000)

output: "1. Rush Limbaugh - Religious conservative\n2. Bernard Adjaye - African American\n3. Juan Williams - Hispanic\n4. Rachel Maddow - Journalist\n5. Bill O'Reilly - Political Commentator\nLimbaugh and Adjaye are both religious conservative politicians, but they differ in their beliefs on issues such as abortion, same-sex marriage, and immigration. Adjaye is a strong advocate for gun rights and has been criticized for his pro-choice stance on social issues. However, he has also been critical of President Trump's handling of the COVID-19 pandemic and has suggested that the country needs to focus on mental health issues rather than politics.\nMaddow, on the other hand, is a journalist and talk show host who has made a name for herself in the media. She has been critical of Trump's handling of the COVID-19 pandemic and has suggested that the country needs to focus on mental health issues rather than politics. However, she has also been critical of the government's response to the pandemic and has suggested that the government needs to do more to address the issue.\nO'Reilly, on the other hand, is a political commentary and talk show host who has made a name for himself in the media. He has been critical of Trump's handling of the COVID-19 pandemic and has suggested that the country needs to focus on mental health issues rather than politics. However, he has also been critical of the government's response to the pandemic and has suggested that the country needs to take care of its citizens who are sick.\nOverall, each of these political commentators has their own unique perspective on the political landscape, and their beliefs are shaped by a variety of factors, including their religious beliefs, political affiliations, and personal experiences. It is important to note that these opinions are not universal and may vary across different demographics and regions of the country. Therefore, it is always best to consult with a qualified political analyst or news organization for a comprehensive understanding of the political landscape. Additionally, it is important to be respectful of others' opinions and not try to influence them. By doing so, we can work together to create a more just and equitable society for all.\nSources:\nLimbaugh, R. (2020). The rise of religion in America. Christianity Today, www.cchurch.com/content/dam/2021/08/the-rise-of-religion-in-america. Retrieved from https://www. ChristianityToday.com/blog/how-religion-is-becoming-a-part-of-america/\nAdjaye, B. (2020). Black Lives Matter: A Call to Action. National Book Critics, www.nrdc.org/books/britannica/article/2020/08/black-lives-matter-a-call-to-action.html\nWright, J. (2020). Climate change and the economy. American Psychological Association, www.apa.org/publication/climate-change-and-economy/2020/08/council-member-wright-jeff-kincaid-reviews-opinions-on-policies-to-reform-climate-change.html\nMegan, M. (2020). The future of healthcare: What we know and don't know. Healthline, www.healthline.com/healthline/2020/08/what-we-know-and-don't-know.html\nO'Reilly, R. (2020). Donald Trump's presidency. Fox News, www.foxnews.com/politics/presidential-race.mp3\nMaddow, R. (2020). The media is biased against the right wing. The New York Times, www.nytimes.com/2020/08/29/us/politics/the-media-is-biased-against-the-right-wing.html\nO'Reilly, R. (2020). The 2020 U.S. presidential election. CNN, www.cnn.com/2020/08/29/us/politics/the-2020-presidential-election.html\nMaddow, M. (2020). The COVID-19 pandemic is a wake-up call for the world. The Wall Street Journal, www.bloomberg.com/news/2020/08/causes-and-benefits-of-the-coVID-19-vaccine.html\nO'Reilly, R. (2020). It's time to get"

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	26.35
ARC (25-shot)	24.06
HellaSwag (10-shot)	35.12
MMLU (5-shot)	24.48
TruthfulQA (0-shot)	41.25
Winogrande (5-shot)	54.78
GSM8K (5-shot)	0.38
DROP (3-shot)	4.39