# Enhancement of Text Analysis Using Context-Aware Normalization of Social Media Informal Text
## 3 Proposed Sentiment Analysis Method
### Methods Used for Text Analysis
#### 3.2.2 Bidirectional Encoder Representations from Transformers (BERT)  
BERT [75] is a state-of-the-art pre-trained encoder stack. BERT is regarded for achieving high performance in many NLP tasks. The strength of this model is its highly pragmatic approach, and its training on huge datasets such as Wikipedia and BookCorpus. The datasets consist of more than 10,000 books from various genres. BERT is made using trans  
\begin{table}
\begin{tabular}{l l l} \hline \hline  & **Attribute** & **Value** \\ \hline \multirow{5}{*}{Settings} & Sequence Input & Sequence input with 1 dimensions \\  & Word Embedding Layer & Word embedding layer with 100 dimensions \\  & LSTM & LSTM with 180 hidden units \\  & Fully Connected & N fully connected layer, N is numClasses \\  & Softmax layer & softmax \\  & Classification Output & crossentropyvex \\  & numClasses (N) & Number of categories in training dataset \\ \hline \multirow{5}{*}{Options} & Training Solver & Adam Solver \\  & mini-batch size & 16 \\ \cline{1-1}  & Shuffle & Shuffle the data every epoch \\ \cline{1-1}  & Gradient Threshold & 1 \\ \cline{1-1}  & Initial Learning Rate & 0.01 \\ \cline{1-1}  & Validation data & yes \\ \cline{1-1}  & Verbose & “false” to suppress verbose output \\ \hline \hline \end{tabular}
\end{table}
Table 1: Settings and options of our LSTM model.  
formers, which enhances the performance of the model. The keystone of the transformers is the self-attention mechanism. BERT transformer uses bidirectional self-attention, where each token can attend to context from both directions [75]. It is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. BERT uses WordPiece embedding [76] with a vocabulary of 30,000 tokens. The WordPiece model is a data-driven approach, where the words to be processed are first broken into sub-word units (wordpieces) of the most frequent sequences of words in the vocabulary. The WordPiece algorithms ensure normalization of many OOV words by its segmentation mechanism to allow only the valid vocabulary sequences (characters combination) [77]. Besides its ability to normalize some of the OOV words, there are some OOV words/sequences which cannot be handled by only using BERT. We have used the BERT implemented in MATLAB 2021b. The basic architecture and implementations are available on Toward Data Science ([[https://towardsdatascience.com/bert-for-dummies-step-by-step-tutorial-fb90890](https://towardsdatascience.com/bert-for-dummies-step-by-step-tutorial-fb90890) ffe03]([https://towardsdatascience.com/bert-for-dummies-step-by-step-tutorial-fb90890](https://towardsdatascience.com/bert-for-dummies-step-by-step-tutorial-fb90890) ffe03), accessed on 30 March 2021) and Github ([[https://github.com/matlab-deep-learning/transformer-models](https://github.com/matlab-deep-learning/transformer-models)]([https://github.com/matlab-deep-learning/transformer-models](https://github.com/matlab-deep-learning/transformer-models)), accessed on 10 April 2021), respectively. We have used the BERT base uncased model which consists of 12 layers, 768 hidden, N fully connected layers. The model is trained on lower-case English text. The settings and options of our BERT model is provided in Table 2.  
In our model, we first train a text classifier on the cleaned training dataset. In the next step we passed the preprocessed, normalized test data from the classifier to classify the text into their respective categories.