# EmmDocClassifier: Efficient Multimodal Document Image Classifier for Scarce Data
## 3 Methodology
### Textual Stream  
In this stream, the textual information is extracted from the documents. Since both the datasets we use for our experiments have only images, the first step is to convert the images to an equivalent textual representation. For this purpose, we employ the Tesseract-OCR. In particular, we use the text extracted from the Tesseract-4 OCR engine. This engine is based on LSTM and simultaneously performs line recognition and character pattern recognition. This generated text contains various symbols and non-alphanumeric characters apart from the punctuation. Unnecessary words/characters that do not add any significance are removed as stopword removal and text preprocessing. Although this helps retain meaningful and weighted words, the text will still contain some meaningless words or malformed words generated from the OCR engine. Moreover, some words are missing in the text due to OCR errors. To address both of the issues, we employ fine-tuned BERT in the next step. Furthermore, It is important to note that in order to address Out-of-Vocabulary (OOV) or malformed words, BERT uses a WordPiece tokenization technique. With the help of this, we predict the missing words and generate the dynamic word embedding using the contextual sentence vectors by summing the corresponding word embedding and segment embedding. Therefore, the words from different contexts will have a different entry for embedding, which the model can further use. The details regarding BERT are explained below. BERT: BERT stands for Bidirectional Encoder Representations from Transformers. It encodes the text bi-directionally and requires minimal architectural changes for any downstream natural language processing tasks using a pre-trained transformer encoder. There are two variants of BERT, namely BERT-Base and BERT-Large. According to [22], the base version consists of 12 layers (transformer blocks), 12 attention heads, and 110 million parameters. The large version consists of 24 layers (transformer blocks), 16 attention heads, and 340 million parameters. In our experiments, we use the base version for fine-tuning and obtaining the word embedding. Before fine-tuning or fetching word embedding, BERT needs a special text preprocessing. The BERT input sequence consists of text tokens and two other unique tokens, i.e., [CLS] and [SEP]. The input to BERT consists of the concatenation of these tokens. The training of BERT functions is based on predicting an unknown token. To this end, it randomly replaces a token with another special token [MASK] and tries to predict the word from the context. This step helps BERT to understand the context better in the sentence. This particular token is used only in pretraining and not in other cases. Figure 3: The architecture of EfficientNet [21]. As we can see, it consists of only fewer layers and lesser parameters. The ’x’ marks on the top indicate the number of such layers present. Every input embedding of BERT is a combination of three embeddings, namely position embedding, segment embedding, and token embedding, as we can see in Figure 4. Positional embedding is necessary to obtain the word's position in the given sentence input. This embedding is essential since it helps BERT to understand the word order. BERT takes sentence pairs as inputs in understanding the context better. Hence it learns a unique embedding for both the sentences to distinguish between them. This is accomplished with segment embedding. As mentioned above, BERT depends on WordPiece token vocabulary. The token embedding is necessary for this. For example, if the word is 'playing', this embedding splits the token as 'play' and '##ing'. With these, BERT handles Out-of-Vocabulary (OOV) words. After preprocessing, it can be passed on to the pre-trained BERT-Base model, which has 12 layers of transformer encoders. The output from any of these layers can be taken out as the word embedding as they are highly contextualized. However, it is always better to consider top layers, since more contextual information is added.