# A Pipeline Approach to Context-Aware Handwritten Text Recognition
## 3 Proposed Method
### The Proposed Context-Aware HTR Pipeline
#### 3.2.3 Residual Network with Transformer (ResNet-101T)  
Resnet is proposed by [PERSON] et al. [21] as a residual learning framework for better neural network training with a deeper architecture. The Residual network has shortcut connections inserted to the network to make a counterpart of the Residual version. There are ResNets with different lengths of layers, which can be implied from their names; for example, ResNet-101 stands for a ResNet architecture containing 101 layers.  
In residual learning, the stacked layers are the building block for feature mapping. A building block can be defined as:  
\[y=F(x,\ \{W_{i}\}+x) \tag{2}\]  
where \(x\) is the input vector, y is the output vector, and the function, \(F\) stands for residual mapping. The operation for \(F\) can be performed by applying a shortcut connection or element-wise addition.  
On the other hand, Transformer is a neural network proposed by [PERSON] et al. [9], that applies an attention mechanism to connect the encoder and decoder. The encoder maps the input to a sequence of continuous representation. Given the mapped sequence, the decoder generates the output of one element at a time, where the model is autoregressive at each step as it uses the generated output at the previous step as additional input for output generation. Transformer with attention mechanism is introduced to tackle the problem of memory constraints in RNN variations by modeling the dependencies without considering the input and output sequence distances. Transformer has an advantage in that it has a relatively low inductive bias compared to RNN.  
The encoder stack contains a multi-head attention mechanism, which is followed by a feed-forward neural network layer, and each layer is followed by a normalization layer. As opposed to the encoder, the decoder contains more layers. The output of the encoder is inserted into the third layer of the decoder. Additionally, a linear layer is appended to the end of the decoder for output prediction. Figure 6 shows the decoder-encoder structure of the Transformer model.  
The attention in the Transformer model is known as scaled-dot attention. The attention function is defined as:  
\[Attention(Q,K,V)=softmax\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\bigg{)}V \tag{3}\]  
where \(Q\) is a matrix of a set of queries of the attention function, and \(K\) and \(V\) stand for keys and values. Dot-product attention is used as it is faster and has better space efficiency compared to additive attention. However, the attention is implemented as a multi-head mechanism. Thus, the outputs of the attention are concatenated as:  
\[Multihead\ attention(Q,K,V)=Concatenate(head_{1},\dots,head_{i})W^{O} \tag{4}\]  
where \(head_{i}\) stands for \(Attention\big{(}QW^{O}_{i},KW^{K}_{i},VW^{V}_{i}\big{)}\). The multi-head attention is applied in the encoder-decoder of the Transformer model to allow the decoder to look through every position of the input sequences, coming from the queries of previous layers and the memory keys and values. Additionally, the self-attention layers in the encoder and decoder allow the output of previous layers to be attended for its position.  
The Transformer model uses positional encoding, \(PE\), to make use of the sequence order by providing information of the token of the sequences,  
\[PE_{(pos,2i)}=\sin\Bigg{(}\frac{pos}{10000^{\frac{2i}{2}}}\Bigg{)},\ PE_{(pos,2i+1)}=\cos \Bigg{(}\frac{pos}{10000^{\frac{2i}{2}}}\Bigg{)} \tag{5}\]  
where \(pos\) and \(i\) are the position and dimension of the input, and d\({}_{\text{model}}\) is the hidden size of the Transformer model.