## 2 Materials and Methods
### Model Architecture  
The model architecture is intended to be composed of an input word embedding layer, an encoder to extract contextualized information, and an ANN to perform the classification into the three classes. Figure 2 illustrates the general architecture that will be developed. At the input, the text data are tokenized to be digested by the model. This is done using either the Blankspace or Wordpiece tokenizer. The architecture described below is mostly inspired by models found in the literature [16; 29; 34], disregarding the use of CNN layers and using the Word2 Vec algorithm to initialize the embedding weights:  
* Word embedding layer (WEL): the input matrix has the dimension \(N\times D\), where \(N\) is the vocabulary size and \(D\) is the number of intended features. This is the interface that transforms the tokenized text into a primary vector representation. The input layer receives a tensor of dimension \(T\), which is the pre-defined maximum number of tokens, where each position, or timestep, is an integer that represents a unique token from the vocabulary. Internally, each integer is converted into a one-hot encoding representation and multiplied by the embedding matrix. The vocabulary size was determined heuristically using a quick search algorithm, where a baseline model was trained multiple times, testing values from 5000 to 30,000 tokens. The performance parameter was the ROC-AUC. The resultant vocabulary size was \(N=11\),560. The \(D\) features for each \(N\) dimension are learned using Word2 Vec, an unsupervised neural network algorithm used to extract contextual relationships between adjacent tokens by considering a window of four tokens to each side of the target token, five training epochs, and an initial learning rate of 0.03 to learn \(D\) features. Additionally, position embedding is included to identify the position of each token in the input sequence. This is required since the self-attention mechanism does not operate sequentially over tokens. These features are pre-trained and are fine-tuned during the globally supervised training.
* Recurrent encoder (RE): \(R\) stacked bidirectional LSTM layers are employed to extract sequential and contextualized information, with \(D\) recurrent units each. Additionally, each LSTM layer features a dropout value (in percentage) to avoid overfitting. Unidirectional LSTM layers are also considered for further optimization. Adding multiple layers possibly allows for a more complex model that could learn more intricate relations between the input and output data. For this reason, testing multiple stacked layers is valuable to consider during optimization.
* Multi-head self-attention layer (MSL): This layer emphasizes the most relevant tokens from the input. It is relevant to determine the best position to apply the self-attention mechanism--before, after, or before and after the recurrent encoder. The multi-head self-attention layer is parametrized with \(H\) attention heads and an internal projection of \(D_{k}\), given by \[D_{k}=D/H,\] (1) with an activation function to introduce non-linearities during the computation of the weights. The recurrent encoder and the multi-head self-attention layer constitute the encoder block.
* Classification layer (CL): Once the context vector is produced by the previous layers, an operation is performed to extract and reduce the information that will be passed to the output layer, followed by an ANN with a number of parameters as a function of \(D\) and an activation function.  
* Output layer (OL): The decoded representation enters a linear layer with three neurons, one for each class. This layer employs a SoftMax activation function to convert the output of the real values into a probability distribution, where the position of the higher value indicates whether the input text is negative, neutral, or positive.