Summarize this and list important information as bullet points: Open in app
Sign up

Sign In



Performance Evaluation of Text Generating NLP Models — GPT-Neo, GPT-2 and XLNet
Shashank Sahoo
Analytics Vidhya
Shashank Sahoo

·
Follow

Published in
Analytics Vidhya

8 min read
·
Aug 31, 2021

Listen


Share

An empirical approach to benchmark performances of three gigantic state-of-the-art language models when applied to same downstream task of text generation.


Photo by nadi borodina on Unsplash
Introduction
Natural Language models have evolved a lot over the years with rapid advancements in research and technology. Data Scientists have achieved excellent results applying these models to downstream tasks and applications like generating SQL code, writing legal documents, sending automatic mail replies, chat-bots, and answering maths questions. It is believed that GPT-3 (latest offering in Generative Pre-Trained Transformers) will usher in a new paradigm in AGI (Artificial General Intelligence) adoption by opening the field to non-technical folks and creating a brand-new domain of few-shot learning products at scale. Language modelling typically leverages statistical and probabilistic techniques to determine the probability of a given sequence of words in a sentence. To make word(s) predictions, language models analyse preceding text data in conjunction with context, grammar and vocabulary.

Common Knowledge
Researchers and developers are working tirelessly on building robust and efficient language models postulating that larger models, trained on a higher number of parameters with internet text corpus, produce better outcomes. Not surprisingly, pre-trained language models have got bigger and bigger in parameter size to the tune of 175B (GPT-3 large) with the motivation that the bigger the model, the better it performs with fewer training examples from domain-specific data. Apparently, it is also more efficient to train a large centralised multi-task model rather than a new model for every task individually. In this article, we strive to compare three such massive language models to debunk and demystify the theory.

Does bigger always mean better in language models context?

Whats trending in AI community ?
The combination of transformers architecture and transfer learning is leading the gold-standard in Natural Language Processing industry. There are numerous pre-trained language models with self-attention transformers block which are similar at first glance but behave differently on real world tasks due to their different inherent training algorithms. Models like BERT, GPT-2, GPT-3, Turing NLG (Microsoft), Megatron (NVIDIA) and T5(Google) have made headlines by stealing spotlight due to their state-of-the-art performances on unsupervised language applications.


Evolution of Language Models. Source : arXiv Research Paper
In our endeavour to choose three model candidates for comparative evaluation we considered various aspects such as open-source code availability, fine-tuning readiness, adaptability, tunable parameters and hardware computation time. Although there are plenty of pre-trained language models with their size derivatives available in the AI community, but we decided to restrict ourselves to only the fundamental ones having uncommon performances. For evaluation purposes, we eventually settled with GPT-Neo (GPT-3 replica), GPT-2 (predecessor) and XLNet( BERT-like) as these models came with pretty good generalised few-shot learning abilities.

Experiment Set-up and Dataset
We will be demonstrating and evaluating the responses of three NLP Models selected above by fine tuning each of them on a simple objective of predicting next word(s) sequence on given input sequence of words. Furthermore, a comprehensive comparison is performed to evaluate all plausible predictions. The goal is set to be achieved primarily through empirical methodologies and progressive iterations.

The experiment is designed to work with a kaggle domain dataset of COVID-19 vaccine tweets which encompasses entries from the vaccine manufacturers namely Pfizer/BioNTech, Sinopharm, Sinovac, Moderna, Oxford/AstraZeneca, Covaxin and Sputnik V.

The data is collected using tweepy Python package to access Twitter API. For each of the vaccine, relevant search term (most frequently used in Twitter to refer to the respective vaccine) is used to merge tweets from all sources across all vaccine manufacturers. We will fine-tune these language models on this dataset to complete vaccine related prompts or inputs by predicting and generating next best sequence of words fitting the context.

For fine-tuning with transfer learning purposes, we choose to use Google Collaboratory free instance with GPU/TPU runtimes. Also, since its shared online training workspace we will only use small-sized comparable versions of all language models under consideration. For this article, we will pick 124M sized GPT-2, 125M sized GPT-Neo and 110M sized XLNet for fine-tuning due to hardware constraints.

Model Candidate 1 : GPT-Neo
GPT stands for “Generative Pre-trained Transformer”. Last year in July, Open AI released GPT-3–an auto-regressive language model trained on public datasets with 500 billion tokens and 175 billion parameters– at least ten times bigger than previous non-sparse language models. To put things into perspective, its predecessor GPT-2 was trained on just 1.5 billion parameters. Unfortunately, GPT-3 (as of April 2021) has not yet been open-sourced. API access to GPT-3 is only available through private beta approvals on mail. However, there is a promising alternative called GPT-Neo, an open-source Transformer model with only 2.7 B parameters that resembles GPT-3 both in terms of design and performance. In this article, we will be implementing GPT-Neo supplanting GPT-3 as an alternative.

Happy Transformer is a package built on top of Hugging Face’s transformer library that makes it easy to utilise state-of-the-art NLP models for inference as well as training them on large variety of tasks such as Text Generation, Text Classification, Question Answering, Word Prediction, etc. GPT-Neo-125M is a transformer model designed using EleutherAI’s replication of the GPT-3 architecture.


Language Models Size Comparison. Source : Google Images
Model Candidate 2: GPT-2
It is the second iteration of the original series of language models released by OpenAI. GPT currently has 3 versions of the model {v1, v2 and v3}. Out of these only GPT-1 and GPT-2 are fully open-sourced, and hence we will pick the latest GPT-2 for our experiment. On the technical side, the architecture of GPT-2 is made up of the decoder part of the Transformer architecture. The encoder-decoder cross attention part of the block is removed because there is no encoder, and the self-attention part is replaced with the masked self-attention.


An illustration of GPT-2. Source : GPT Documentation
Model Candidate 3: XLNet (BERT)
XLNet is a BERT-like model of a different kind. But it is a very promising and potential one. XLNet incorporates a generalised auto-regressive (AR) pre-training method. It is an AR language model that uses the context word to predict the next word. But here the context word is constrained to two directions, either forward or backward and not both at once. Language model consists of two phases, the pre-train phase, and fine-tune phase. XLNet focus on pre-train phase. In the pre-train phase, it proposed a new objective called Permutation Language Modeling using permutations for generation.


An illustration of XLNet. Source : A SOTA XLNet
Unlike the AR language model, BERT is categorized as autoencoder (AE) language model. The AE language model aims to reconstruct the original data from corrupted input. The corrupted input means the [MASK] used to replace the original token into in the pre-train phase. And the objective is to predict into to get the original sentence. The AE language model can see the context in both forward and backward direction equivalently.

Result & Observation
The experiment comprises of fine-tuning the pre-trained generalised language models of GPT-Neo, GPT-2 and XLNet on vaccine datasets through transfer learning technique. To execute this, we have parsed few-prompts of vaccine related phrases and short texts which acts as input precursor for these models to optimise weights and biases of final few neural network layers. This enables the model to respond on vaccine specific text prompts by predicting the next best meaningful sequence in an intelligible manner.

For the sake of simplicity and rapid comparison, we have parsed same text prompts to all three now fine-tuned language models during evaluation phase to record their responses for four iterations. Below set of table images depict the comparative view of all such responses in a comprehensive manner.


Four iterations of first text prompt. Image by author

Four iterations of second text prompt. Image by author

Four iterations of third text prompt. Image by author

Four iterations of fourth text prompt. Image by author

Four iterations of fifth text prompt. Image by author

Four iterations of sixth text prompt. Image by author
In general, GPT-Neo detects and responds contextually better than other model candidates for all six prompts. It is also quite verbose even with most optimised GENSettings. While, GPT-2 generates much shorter but contextual prompts and takes more time to optimise during fine-tuning phase. Finally, XL-Net stands at the bottom in ranking as it deviates from context often and also picks irrelevant word due to its masked nature training paradigm.

We also established a relative BLEU (Bilingual Evaluation Understudy Score)score to benchmark the results of each iteration for each prompts and tabulated them as below:


BLEU Scores. Image by author
BLEU is a precision focused metric that calculates n-gram overlap of the reference and generated texts. This n-gram overlap means the evaluation scheme is word-position independent apart from n-grams’ term associations. One thing to note in BLEU — there is a brevity penalty i.e. a penalty applied when the generated text is too small compared to the target text. All source code for this can be forked from here.

Conclusion
The results thus obtained are interesting as it helps to decipher the transfer learning pedagogy followed by pre-trained language models to rapidly adapt to any downstream task. Also, we successfully tested the performances and validated across BLEU scores and surely it needs improvement by fine-tuning across more epochs. But the gist is clear as these models tend to perform better as they evolve with time in terms of size and multi tenant capabilities. These models were good if not great and could produce meaningful results when fine-tuned with right configurations through multiple trials. This makes them the perfect baseline starters if not the state-of-the-art for many NLP applications.

References
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, and Sharan Narang. Exploring the limits of transfer learning with a unified text-to-text transformer. 2020. arXiv:1910.10683
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and others. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
GPT-Neo
GPT-2
48



Language Model
Gpt 2
Bert
Xlnet
NLP
48




Shashank Sahoo
Analytics Vidhya
Follow

Written by Shashank Sahoo
39 Followers
·
Writer for 
Analytics Vidhya

Lead Data Scientist with focus on Generative AI and Large Language Models(LLMs), Transformers, GPT, Advanced NLP & MLOps. On mission to "Democratize AI for All"

More from Shashank Sahoo and Analytics Vidhya

Shashank Sahoo
Shashank Sahoo

in

Analytics Vidhya

Perfecting recommendation pipeline with advanced statistical and algebraic computations
This post is an extension to my last blog that described the most lucid pipeline of general recommendation systems through a promising…
10 min read
·
Nov 28, 2019
39



Sumedh Datar
Sumedh Datar

in

Analytics Vidhya

Application of Foreground and Background separation with Deep Learning
Foreground and background separation had always been a huge problem before the onset of object detection based neural networks. Techniques…
3 min read
·
Apr 19, 2021
10.1K



Kia Eisinga
Kia Eisinga

in

Analytics Vidhya

How to create a Python library
Ever wanted to create a Python library, albeit for your team at work or for some open source project online? In this blog you will learn…
7 min read
·
Jan 26, 2020
1.8K


23


Shashank Sahoo
Shashank Sahoo

in

Analytics Vidhya

Designing Web-Service Endpoints using public data set and BigQuery
Many of you have experienced the need to overcome latency in getting time-bound response while dealing with massive data sets. Data as we…
4 min read
·
Nov 13, 2019
23


See all from Shashank Sahoo
See all from Analytics Vidhya
Recommended from Medium

Skanda Vivek
Skanda Vivek

in

Towards Data Science

Fine-Tune Transformer Models For Question Answering On Custom Data
A tutorial on fine-tuning the Hugging Face RoBERTa QA Model on custom data and obtaining significant performance boosts

·
5 min read
·
Dec 15, 2022
371


3

Two stochastic parrots sitting on a chain of large language models: LangChain
Leonie Monigatti
Leonie Monigatti

in

Towards Data Science

Getting Started with LangChain: A Beginner’s Guide to Building LLM-Powered Applications
A LangChain tutorial to build anything with large language models in Python

·
12 min read
·
Apr 25
1.7K


12

Lists



Staff Picks
310 stories
·
77 saves

LucianoSphere
LucianoSphere

in

Towards AI

Build ChatGPT-like Chatbots With Customized Knowledge for Your Websites, Using Simple Programming
Like ChatGPT but in a form that you can plug into your website and expand with any kind of tailored information by combining basic…

·
11 min read
·
Dec 26, 2022
850


14


Cameron R. Wolfe
Cameron R. Wolfe

in

Towards Data Science

Language Models: GPT and GPT-2
How smaller language models inspired modern breakthroughs

·
13 min read
·
Nov 24, 2022
22



Skanda Vivek
Skanda Vivek

in

Towards Data Science

Extractive vs Generative Q&A — Which is better for your business?
The arrival of ChatGPT hints at a new era of search engines, this tutorial dives into the 2 basic types of AI based question answering

·
6 min read
·
Feb 6
23



𝚃𝚑𝚎 𝙻𝚊𝚝𝚎𝚜𝚝 𝙽𝚘𝚠
𝚃𝚑𝚎 𝙻𝚊𝚝𝚎𝚜𝚝 𝙽𝚘𝚠

in

MLearning.ai

Building Your Own Mini ChatGPT
Take a dive into NLP by creating your own small-scale language model with code examples and technical details

·
7 min read
·
Jan 19
358


9

See more recommendations