Question: What are the max-pooling and average-pooling used for?
Evidence: In this section, we introduce the deep neural network architecture that we used to classify emotions in the tweets dataset. Emotional expressions are more complex and context-dependent even compared to other forms of expressions based mostly on the complexity and ambiguity of human emotions and emotional expressions and the huge impact of context on the understanding of the expressed emotion. These complexities are what led us to believe lexicon-based features like is normally used in conventional machine learning approaches are unable to capture the intricacy of emotional expressions.
Our architecture was designed to show that using a model that captures better information about the context and sequential nature of the text can outperform lexicon-based methods commonly used in the literature. As mentioned in the Introduction, Recurrent Neural Networks (RNNs) have been shown to perform well for the verity of tasks in NLP, especially classification tasks. And as our goal was to capture more information about the context and sequential nature of the text, we decided to use a model based on bidirectional RNN, specifically a bidirectional GRU network to analyze the tweets.
For building the emotion classifier, we have decided to use 7 binary classifiers-one for each emotion- each of which uses the same architecture for detecting a specific emotion. You can see the plot diagram of the model in Figure FIGREF6 . The first layer consists of an embedding lookup layer that will not change during training and will be used to convert each term to its corresponding embedding vector. In our experiments, we tried various word embedding models but saw little difference in their performance. Here we report the results for two which had the best performance among all, ConceptNet Numberbatch BIBREF35 and fastText BIBREF36 both had 300 dimensions.
As none of our tweets had more than 35 terms, we set the size of the embedding layer to 35 and added padding to shorter tweets. The output of this layer goes to a bidirectional GRU layer selected to capture the entirety of each tweet before passing its output forward. The goal is to create an intermediate representation for the tweets that capture the sequential nature of the data. For the next step, we use a concatenation of global max-pooling and average-pooling layers (with a window size of two). Then a max-pooling was used to extract the most important features form the GRU output and an average-pooling layer was used to considers all features to create a representation for the text as a whole. These partial representations are then were concatenated to create out final hidden representation. For classification, the output of the concatenation is passed to a dense classification layer with 70 nodes along with a dropout layer with a rate of 50% to prevent over-fitting. The final layer is a sigmoid layer that generates the final output of the classifier returning the class probability.
Then a max-pooling was used to extract the most important features form the GRU output and an average-pooling layer was used to considers all features to create a representation for the text as a whole.
Answer: The max-pooling was used to extract the most important features form the GRU output and the average-pooling layer was used to considers all features to create a representation for the text as a whole.