## 3 Methodology
### Generating Text
#### 3.1.2 GPT-2 Text Generation  
GPT-2 model proposed by [PERSON] et al. [22] is a revolution in processing text. The model is trained on 40 GB of Internet text for generating new text, the creative side of AI. It is a Transformer [23] based architecture with 1.5 billion parameters exploiting text from eight million webpages. GPT-2 has demonstrated an exceptional human-like quality in generating conditional synthetic text of long sequences. In this section, we discuss the steps followed to retrain GPT-2 model using our own custom MOOCs lecture dataset. As discussed earlier, GPT-2 performs better than basic LSTM text generators for text generation on long text sequences. The dataset [17] is the collection of MOOCs video lecture transcripts annotated into the eight general-level categories (i.e., Art and Humanities, Physical Sciences and Engineering, Computer Science, Data Science, Business, Information Technology, Health, Social Science). Originally, the data is highly imbalanced and needs to be balanced. For this task, we have used the GPT-2 (117M) model. The steps involved in generation and balancing the transcript dataset are shown in Figure 5 and explained in the subsequent sections. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Original** & \multicolumn{3}{c}{**Generated**} \\ \hline \# & **Text** & **Class Label** & **Text** & **Class Label** \\ \hline
**1** & he is very well experienced & Experience & well-experienced & Experience \\
**2** & give result on time & Assessment & sir still hasn’t given result of 1 st & Assessment \\
**3** & sir is very kind & Behavior & always pay respect to students & Behavior \\
**4** & sir is very slow in carry & Knowledge & he is very good teacher for & Knowledge \\  & out course & & case study especially & Knowledge \\
**5** & he improves teaching style & Teaching Skills & he is very fast! & Teaching Skills \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Original + Generated Sample of Student’s Feedback Dataset [15]. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Original** & \multicolumn{3}{c}{**Generated**} \\ \hline \# & **Text** & **Class Label** & **Text** & **Class Label** \\ \hline
**1** & Thinks that @melbahughes had & Surprise & My dad just got a guest cell & Surprise \\  & a great 50 th birthday party :) & & of my be & Surprise \\  & My heart and soul @lay\_Babe & & & \\
**2** & is leaving me and I can’t even & Sadness & I miss the fact that I was so & Sadness \\  & see here & & & \\
**3** & @[PERSON] [PERSON], & & & \\  & what’s your favourite colour? & & & \\
**4** & [PERSON] im so mad its burning my & & & \\  & fuckin stomach! & & & \\
**5** &...and guess what song just popped & & & \\  & in my head. & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Original + Generated Sample of Twitter Emotion Dataset [16]. **Preparing Custom Text Dataset**  
First of all, a text corpus is created for each of the class label belonging to general-level categories. This contains transcripts related to the specific class. For preparing the training data for each of the corpus, each single transcript is delimited with **\"<l endoftextl>\"**. This delimiter will allows the model to understand the formatting of the training data. **Encoding the Training Data**  
The next step is to encode the training corpus which we created in previous step. This encoding will be performed for all of the eight text corpus files belonging to the class labels. The primary purpose of encoding the text data is to convert it into numbers so that the model can understand it. Moreover, for GPT-2, the widely used encoding technique is byte-pair encoding [24] technique, a compression technique used for text data. **Training the GPT-2 Model**  
Here, we start training the GPT-2 model using the encoded corpus dataset created in previous step. GPT-2 requires configuration of three parameters to start learning and training on the custom dataset. The parameters are \(batch\_size\), \(learning\_rate\) and \(sample\_every\).