# Fi-Fo Detector: Figure and Formula Detection Using Deformable Networks
## 3 Fi-Fo Detector
### Deformable Convolutional Network (DCN)  
The proposed method is based on deformable convolution networks (DCN) [26; 27]. Convolutional neural networks learn the relevant feature representation depending on the task at hand [28]. These features are extracted in every layer using filters. Filters in lower convolutional layers usually capture textures and preliminary objects, which includes gradients, textures, materials, and colors, whereas filters in higher convolutional layers describe more abstract objects and their parts [29]. In traditional CNNs, a convolutional layer samples the input feature-maps at fixed locations, which the subsequent layers carry forward, resulting in a fixed and known geometric transformation. Using the fixed grid for the detection of objects occurring at different scales and different transformations is not ideal. Deformable convolutional networks address these constraints of traditional CNNs by introducing two additional modules to existing deep neural networks, namely (i) the deformable convolution and (ii) deformable Rol-pooling. Regular convolutional layers are augmented with a \(2D\)-offset convolutional layer to form the deformable convolution layer. Regular convolution operates on a uniform grid as its receptive field, whereas deformable convolution leverages the additional offset layers to augment the uniform grid conditioned on the input. The adaptive receptive field allows filters in convolution layers to adapt to different scales and transformation. Since objects like figures and formulas appear at vastly different scales, the deformable property significantly helps in coping up with these intense input variations. Mathematical formulation of deformable convolution is explained in [26] as:  
\[y(p_{0})=\sum_{p_{a}\in\mathcal{R}}w(p_{n})\times x(p_{0}+p_{n}+\Delta(p_{n})) \tag{1}\]  
where \(\mathcal{R}\) defines the offsets from the point under consideration (\(p_{0}\)) in a regular-grid pattern, \(x\) represents the input, \(y\) represents the output feature map while \(w\) represents the filter weights. Considering a \(3\times 3\) convolutional layer, the set \(\mathcal{R}=\{(-1,-1),(-1,0),(-1,1),(0,-1),(0,0),(0,1),(1,-1),(1,0),(1,1)\}\) defines this regular-grid comprising of the nine positions within the receptive field of the filter. In deformable convolution, sampling is done on irregular locations determined by the offset. The offset is defined as \(\Delta(p_{n})\), which augments the predefined offsets to arbitrarily deform the receptive field of the filter. Both the features as well as the offsets are learned by back-propagation of gradients. Since these offsets are fractional, they are implemented via bilinear interpolation. For simplicity, let us consider \(p=p_{0}+p_{n}+\Delta(p_{n})\). Hence, the operation can be represented as:  
\[x(p)=\sum_{q}G(q,p)\times x(q)\quad\text{where}\quad G(q,p)=g(q_{x},p_{x}) \times g(q_{y},p_{y}) \tag{2}\]  
where q enumerates all the possible spatial locations on the feature map \(x\), \(G\) is the bilinear interpolation kernel and \(g\) is defined to be \(g(a,b)=max(0,1-a-b)\). Region proposals are an integral part of object detection methods, achieved by using RoI-pooling, which converts an arbitrary sized input region into fixed-size feature representation. Regular RoI-pooling divides the RoI into \(k\times k\) spatial bins. Similar tothe deformable convolution, deformable RoI-pooling introduces additional offsets that are added to spatial bins. This can be mathematically written as Equation (3).  
\[y(i,j)=\sum_{p\in bin(i,j)}\frac{x(p_{0}+p+\Delta p_{i}j)}{n_{ij}} \tag{3}\]  
where \(bin(i,j)\) defines a bin over spatial locations for feature aggregation (\(\lfloor i\frac{w}{k}\rfloor<=p_{x}<\lceil(i+1)\frac{w}{k}\rceil\), \(\lfloor j\frac{h}{k}\rfloor<=p_{y}<\lceil(j+1)\frac{h}{k}\rceil\)), and \(n_{ij}\) represents the number of items in \(bin(i,j)\). We refer readers to [26; 27] for a comprehensive introduction to the deformable convolutional layers.