# EmmDocClassifier: Efficient Multimodal Document Image Classifier for Scarce Data
## 3 Methodology
### Textual Stream
With some studies and experiments, it is observed that taking the final four layers and combining them has given better word embedding [50]. As it is clear that there can be different ways to combine the results, and it is entirely data-dependent, we experimented with different strategies and obtained F1 scores for each of them on top frequent tokens from text data. As we can see from Figure 5, both concatenation and summation of the final four layers yield a similar result. We chose the summation approach over concatenation to reduce the embedding dimension. Using summation, results would hold all the information. The only shortcoming of this approach is using a pre-trained BERT-Base model. Although it is trained on a massive dataset and is generic enough to understand, it still precisely lacks knowledge of our corpus. Thus, we decided to fine-tune this BERT-Base model on our text corpus. We extract word embedding after the model is trained. Since the context is a prime focus of the BERT transformers, training it for our text helps the model learn context better. The fine-tuned BERT achieved an F1 score of 96.5 for the same set of tokens. We freeze all the layers of the BERT-Base model except the last four layers and add a fully connected layer to classify the text documents. Even if the text size is small, the broad range BERT model can smartly pick up with the context. Unlike any LSTM or GRU-based model, BERT requires no more than 4 epochs to understand the entire corpus. Once this fine-tuning is done, the weights are frozen, and the fully connected layer is removed. The final embedding is obtained from the vector formed by the summation of the final four layers. This embedding is used for the textual stream model. Figure 4: Here we can see the typical breakdown of the pair of sentences into three embeddings, Token embedding, Sentence embedding, and Transformer positional Embedding. The exciting thing about BERT is that it can differentiate between the same word ‘bank’ used in different contexts in two separate sentences. **HAN:** In the next step, we use the HAN model proposed by [PERSON] et al. [5]. This attention network learns the text both from sentence level and word level. By forming the hierarchy of documents, sentences, and words, it tries to learn the linked low-level features. It helps to differentiate highly overlapped classes of documents and also understand at the same time their global contexts. As seen in Figure 6, HAN has 2 hierarchies for word and sentence. For each of these components, it has an encoder and attention. For instance, consider a document \(D_{i}\), which has \(n\) number of sentences in it, and in turn, each sentence has \(m\) number of words. Contextualized word vectors are then obtained by passing these words through a fine-tuned BERT embedding. Further, Bidirectional GRU [51] is used to encode these vectors in both directions. Figure 5: This Figure depicts different strategies for obtaining better embedding for our problem. Each layer in the figure corresponds to the layer of the BERT-Base, which consists of 12 layers of transformer encoders. Figure 6: This figure depicts the architecture of Hierarchical Attention Network [5]. First, all the words are passed through Bi-directional GRU (F representing forward, B representing backward) to form the encoder. Then they are then passed to an attention network. These sets of words are then mapped to their corresponding sentence from the input, which goes through a similar process of Bi-directional GRU and Attention. The final vector V then represents the feature vector, which can be used to classify with softmax or combined with the feature vector of the visual stream to form a Two-stream model. For attention, the encoded-word \(H_{e}\) is fed through MLP with \(tanh\) activation. This is then normalized before taking the weighted sum that eventually forms the sentence \(s_{i}\). On similar grounds, the sentence level encoding and attention are computed.