Paraphrase any question with T5 (Text-To-Text Transfer Transformer) — Pretrained model and training script provided | by Ramsri Goutham | Towards Data Science

The input to our program will be any general question that you can think of -
Which course should I take to get started in data Science?                 
The output will be paraphrased versions of the same question. Paraphrasing a question means, you create a new question that expresses the same meaning using a different choice of words.
Paraphrased Questions generated from our T5 Model ::0: What should I learn to become a data scientist?1: How do I get started with data science?2: How would you start a data science career?3: How can I start learning data science?4: How do you get started in data science?5: What's the best course for data science?6: Which course should I start with for data science?7: What courses should I follow to get started in data science?8: What degree should be taken by a data scientist?9: Which course should I follow to become a Data Scientist?
As you can see we generated about 10 questions that are paraphrases to the original question — “Which course should I take to get started in data science?”
Today we will see how we can train a T5 model from Huggingface’s transformers library to generate these paraphrased questions. We will also see how we can use the pre-trained model provided to generate these paraphrased questions.
Imagine a middle school teacher preparing a quiz for the class. Instead of giving a fixed question to every student he/she can generate multiple variants of a given question and distribute them across students. The school can also augment their question bank with several variants of a given question using this technique.
Let’s get started —
I used the Quora Question Pairs dataset to collect all the questions marked as duplicates and prepared training and validation sets. Questions that are duplicates serve our purpose of getting paraphrase pairs.
We will discuss in detail how you can -
Use my pre-trained model to generate paraphrased questions for any given question.Use my training code and dataset to replicate the results on your own GPU machine.
Use my pre-trained model to generate paraphrased questions for any given question.
Use my training code and dataset to replicate the results on your own GPU machine.
T5 is a new transformer model from Google that is trained in an end-to-end manner with text as input and modified text as output. You can read more about it here.
It achieves state-of-the-art results on multiple NLP tasks like summarization, question answering, machine translation, etc using a text-to-text transformer trained on a large text corpus.
I trained T5 with the original sentence as input and paraphrased (duplicate sentence from Quora Question pairs) sentence as output.
Use the Google Colab link in the “Using Pre-trained model” section to test this out. All the code for using training the model with given data is available at -
github.com
The Google Colab notebook t5-pretrained-question-paraphraser contains the code presented below.
First, install the necessary libraries -
!pip install transformers==2.8.0
Run inference with any question as input and see the paraphrased results.
The output from the above code is -
device  cpuOriginal Question ::Which course should I take to get started in data science?Paraphrased Questions :: 0: What should I learn to become a data scientist?1: How do I get started with data science?2: How would you start a data science career?3: How can I start learning data science?4: How do you get started in data science?5: What's the best course for data science?6: Which course should I start with for data science?7: What courses should I follow to get started in data science?8: What degree should be taken by a data scientist?9: Which course should I follow to become a Data Scientist?
Again all the training code and dataset used for training are available in the Github repo mentioned earlier. We will go through the steps that I used to train the model.
First I downloaded the Quora Question pairs tsv file (quora_duplicate_questions.tsv) as mentioned in this link.
Extracted only the rows that have is_duplicate =1 since they are the paraphrased question sentences. Then I had split the data into train and validation sets and stored them in separate CSV files.
In the end, each of the CSV files has two columns “question1” and “question2”. “question2” is a paraphrased version of “question1”. Since T5 expects a text as input, I gave “question1” as the input source and asked it to generate “question2” as target output.
The code used to generate the train and validation CSV files is shown below. The CSV files are available under the paraphrase_data folder in the Github repo.
filename = "quora_duplicate_questions.tsv"import pandas as pdquestion_pairs = pd.read_csv(filename, sep='\t')question_pairs.drop(['qid1', 'qid2'], axis = 1,inplace = True)question_pairs_correct_paraphrased = question_pairs[question_pairs['is_duplicate']==1]question_pairs_correct_paraphrased.drop(['id', 'is_duplicate'], axis = 1,inplace = True)from sklearn.model_selection import train_test_splittrain, test = train_test_split(question_pairs_correct_paraphrased, test_size=0.1)train.to_csv('Quora_Paraphrasing_train.csv', index = False)test.to_csv('Quora_Paraphrasing_val.csv', index = False)
Thanks to Suraj Patil for the amazing Colab notebook on training T5 for any text-to-text task. I borrowed most of the training code from the Colab notebook, changing only the dataset class and training parameters. I adapted the dataset class to our Quora Question Pair dataset.
The training code is available as train.py in the Github Repo.
All you need to do is clone the repo on any GPU machine, install requirements.txt, and run train.py to train the T5 model.
Training this model for 2 epochs (default) took about 20 hrs on p2.xlarge (AWS ec2).
The dataset class looks like below —
The key is how we give our input and output to the T5 model trainer. For any given question pair from the dataset, I gave input (source) and output (target) to the T5 model as shown below -
Input format to T5 for training
paraphrase: What are the ingredients required to make a perfect cake? 

Output format to T5 for training
How do you bake a delicious cake? 

That’s it! You have a state-of-the-art question paraphraser in your hand.
Perhaps this is the first work of it's kind out there to generate paraphrased questions from any given question!
Happy coding!
I launched a very interesting Udemy course titled “Question generation using NLP” expanding on some of the techniques discussed in this blog post. If you would like to take a look at it, here is the link.