# Cascade Network with Deformable Composite Backbone for Formula Detection in Scanned Document Images
## 3 Method
### Deformable Convolution  
We incorporate deformable convolution filters [42] instead of the conventional convolutions that exist in the ResNeXt-101 architecture [46]. The convolutional neural networks extract the important spatial features that are essential to perform the required task. Based on the hierarchy, convolutional layers discover different features [47]. Convolutional layers present at the bottom search for crude features such as sharp edges or the gradients, whereas the layers at higher levels look for the abstract components such as complete object [48]. The conventional convolution operation has the same effective receptive field for all the neurons. The 2D convolution is comprised of two parts: (1) the first step samples the input feature map through a grid \(R\), and (2) aggregation of sample values is multiplied by the weight \(\mathbf{w}\). For conventional convolution, the output of feature map \(y\) for each position \(p_{o}\) is elaborated in [42] as follows:  
\[\mathbf{y}(\mathbf{p}_{0})=\sum_{\mathbf{p}_{n}\in\mathcal{R}}\mathbf{w}( \mathbf{p}_{n})\times\mathbf{x}(\mathbf{p}_{0}+\mathbf{p}_{n}) \tag{3}\]  
where x represents the input feature map, and \(p_{n}\) iterates over the locations in a grid \(R\) that can be defined as \(R=(-1,-1),(-1,0),(-1,1),(0,-1),(0,0),(0,1),(1,-1),(1,0),(1,1)\) for a \(3\times 3\) convolutional layer. The effective receptive field of such a filter is restricted to these nine positions. In the case of deformable convolution, an additional offset represented as \(\Delta(p_{n})\) is added, which deforms the filter's receptive field by augmenting the predefined offsets. Hence, Equation (3), as explained in [42], is transformed into  
\[\mathbf{y}(\mathbf{p}_{0})=\sum_{\mathbf{p}_{n}\in\mathcal{R}}\mathbf{w}( \mathbf{p}_{n})\times\mathbf{x}(\mathbf{p}_{0}+\mathbf{p}_{n}+\Delta\mathbf{p }_{n}) \tag{4}\]  
This modification makes the sampling process irregular with an offset value of \(p_{n}+\Delta(p_{n})\). As these offsets are differentiable and fractional, bilinear interpolation is used to implement them. Considering \(p=p_{0}+p_{n}+\Delta(p_{n})\), the bilinear interpolation is implemented as follows:  
\[\mathbf{x}(\mathbf{p})=\sum_{\mathbf{q}}\mathbf{G}(\mathbf{q},\mathbf{p}) \times\mathbf{x}(\mathbf{q}) \tag{5}\]  
where q iterates over all the possible places on the input feature map \(x\), and the symbol \(G\) represents the bilinear interpolation kernel. It is vital to mention that \(G\) is a two-dimensional \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Stage** & **Output** & **DCN** & **ResNeXt-101 (\(\Omega\)\(\times\) 4d)** \\ \hline conv1 & \(112\times 112\) & ✗ & \(7\times 7\), 64, stride 2 \\ \hline \multirow{3}{*}{conv2} & \multirow{3}{*}{\(56\times 56\)} & - & \(3\times 3\) max pooling, stride 2 \\ \cline{3-4}  & & & \(1\times 1\), 128 \\  & & & \(3\times 3\), 128, C = \(32\times 3\) \\  & & & \(1\times 1\), 128 \\ \hline conv3 & \(28\times 28\) & ✓ & \(\begin{array}{c}1\times 1\), 256 \\ \(3\times 3\), 256, C = \(32\times 4\) \\ \(1\times 1\), 512 \\ \end{array}\) \\ \hline conv4 & \(14\times 14\) & ✓ & \(\begin{array}{c}1\times 1\), 512 \\ \(3\times 3\), 512, C = \(32\times 23\) \\ \(1\times 1\), 1024 \\ \end{array}\) \\ \hline conv5 & \(7\times 7\) & ✓ & \(\begin{array}{c}1\times 1\), 1024 \\ \(3\times 3\), 1024, C = \(32\times 3\) \\ \(1\times 1\), 2048 \\ \end{array}\) \\ \hline \hline \end{tabular}
\begin{tabular}{c} \hline \hline \end{tabular}
\end{table} 
Table 1: Architectural details of the employed dual composite ResNeXt-101 backbone network. DCN represents the incorporation of deformable convolution. kernel that can be further divided into two one-dimensional kernels. It is mathematically explained as  
\[\mathbf{G}(\mathbf{q},\mathbf{p})=\mathbf{g}(\mathbf{q}_{\mathbf{v}},\mathbf{p}_{ \mathbf{v}})\times\mathbf{g}(\mathbf{q}_{\mathbf{y}},\mathbf{p}_{\mathbf{y}}) \tag{6}\]  
where \(g\) is explained as \(g(a,b)=max(0,1-|a-b|)\).