# Object Detection Based on Multiple Information Fusion Net
## 3 The Proposed Method
### Relationship Module  
In most previous object detection methods based on the convolutional neural network [7; 16], each object is identified independently, and the relationship of objects is neglected. To overcome this limitation, the proposed approach models the relationship of objects by groups. That is, the feature vector of an object is obtained by fusing the features of itself and other objects to enrich the information, as shown in Figure 2.  
In Figure 2, given an input set of \(N\) objects \(\left(\left(f_{t}^{n},\ f_{c}^{n}\right)\right)_{n=1}^{N}\), in which \(f_{t}^{n}\) is the original appearance feature of the \(n\)th object extracted by convolutional neural network, \(f_{c}^{n}\) denotes the location feature of the \(n\)th object composed by the 4-dimensional feature of object bounding box. The bounding box feature comprises width (\(w\)), height (\(h\)) and the center coordinates (\(x,y\)) of the box in our study. The relationship channel is a module that handles relationships among different objects, \(N_{r}\) is the number of channels (\(N_{r}=64\)). By object relation module, the \(f_{c}^{1}(n),\ f_{c}^{2}(n)\ldots f_{c}^{N_{r}}(n)\) which fuse the location information of all surrounding objects can be gained. For the purpose of obtaining the output \(f_{t}^{n^{\prime}}\) that is finally sent into the relation GRU, we concatenate the vectors on all channels \(f_{c}^{1}(n),f_{c}^{2}(n)\ldots f_{c}^{N_{r}}(n)\). Because the processing mechanisms of relation channel modules are the same, we take one relation channel module as an example to explain how relation channel works. Figure 3 shows the process of one relation channel module. Firstly, the dot product operation is applied to obtain the appearance weight \(w_{t}^{mn}\) between the \(m\)th and \(n\)th objects, as shown in Equation (2). \[w_{t}^{mn}\ =\ \frac{(W_{K}f_{t}^{m})\cdot(W_{Q}f_{t}^{n})}{\sqrt{d}}, \tag{2}\]  
where \(W_{K}\), \(W_{Q}\) are matrices which map the original appearance \(f_{t}^{m}\) and \(f_{t}^{n}\) into subspaces, \(\cdot\) denotes the operation of dot product to obtain the degree of matching between \(W_{K}f_{t}^{m}\) and \(W_{Q}f_{t}^{n}\). Figure 2: Object relation module. Secondly, the location weight \(w_{g}^{mn}\) is calculated by Equation (3). \[w_{g}^{mn}\ =\ \text{Relu}\Big{(}W_{g}\cdot\varepsilon_{g}(f_{c}^{m},f_{c}^{n} )\Big{)}, \tag{3}\]  
where \(f_{c}^{m}\) and \(f_{c}^{n}\) are geometry features that contain six relative position information (\(\log\!\left(\frac{|y_{m}-y_{m}|}{h_{n}}\right)\)), \(\log\!\left(\frac{|y_{m}-y_{m}|}{h_{n}}\right)\), \(\log\!\left(\frac{|y_{m}-y_{m}|}{m_{n}}\right)\), \(\log\!\left(\frac{|y_{m}-y_{m}|}{h_{n}}\right)\), \(\log\!\left(\frac{|y_{m}-y_{m}|}{h_{n}}\right)\), \(\sqrt{\frac{|y_{m}-y_{m}|}{m_{n}}}\), in which \(w_{n}\), \(h_{n}\), \(x_{n}\) and \(y_{n}\) are the width, height and center coordinates of the \(n\)th object, \(\varepsilon_{g}\) is a function based on sine and cosine to embed the geometry features into high-dimensional space [38]. Then we use \(W_{g}\) to convert the embedded vector to scalar weight. The Relu activation function is utilized to ensure that only objects with certain geometric relationship can participate in this relationship calculation. Next, the relationship weight is obtained by Equation (4). \[w^{mn}\ =\ \text{softmax}\Big{(}w_{g}^{mn}\cdot\exp\Big{(}w_{t}^{mn} \Big{)}\Big{)}, \tag{4}\]  
where the relationship weight \(w^{mn}\) represents the impact of the \(m\)th object to the \(n\)th object, softmax is employed for normalization. Finally, Equation (5) can be utilized to get a feature \(f_{c}^{N_{r}}(n)\) that has the influence of surrounding objects on it. \[f_{c}^{N_{r}}(n)\ =\ \sum_{m}w^{mn}\cdot\Big{(}W_{g}f_{t}^{m} \Big{)}, \tag{5}\]  
Figure 3: Relation channel module. where \(W_{v}\) is used to transform the original appearance feature \(f_{t}^{n}\) linearly. Equation (5) is the process of integrating the information of the object and other objects into the original appearance feature.