# Object Detection Based on Multiple Information Fusion Net
## 3 The Proposed Method
### Message Passing Module  
As we have discussed previously, context information is important for accurate object detection. For example, in Figure 4a, if road is considered as the global or scene-level information, the objects in this image are hardly to be detected as ships and planes since it is generally impossible for them to appear in the road scene. Similarly, in Figure 4b, when a dinner table appears, the probability of detecting chairs increases, because the dinner tables and chairs always appear in pairs. Thus, the Gated Recurrent Unit (GRU) [54] is utilized in this study. Similar to the long short-term memory (LSTM) model [55], GRU unit also has the function of adjusting the information flow in the unit, but it is lightweight and effective [35]. In the message passing module, information is continuously passed between the scene GRU and the relation GRU so that the useful information can be preserved. GRU only has two gates. It combines the input gate and the forget gate in the LSTM into one, and the combined gate is called update gate, which determines how much information from the previous time and the current time is to be passed on. The other gate in GRU is reset gate, which controls how much past information is forgotten. In our work, two parallel GRUs are applied to pass information to each other, one is the scene GRU and the other is the relation GRU. The scene GRU receives the whole image information \(f_{s}\) as the input. The input of relation GRU is the integrated object information \(f_{t}^{n^{\prime}}\), which includes the object's own information and the influence of surrounding objects on it. We represent the initial state \(h_{i}\) of the  
Figure 4: Some detection results of our model. (**a**) is a picture of road scene, (**b**) is a picture of dining room. network with the original appearance feature (without any scene information or relation information). Here, since the processing mechanisms of scene GRU and relation GRU are identical, we take the relation GRU as an example to show how GRU works. Firstly, the reset gate of the \(t\)th moment \(r_{t}\) is calculated as follows:  
\[r_{t}\ =\ \sigma\Big{(}W_{r}\Big{[}f_{t}^{nr^{\prime}},h_{i}\Big{]}\Big{)}, \tag{7}\]  
where \(\sigma\) is the logistic sigmoid function, \([,]\) denotes the concatenation of vectors, and \(W_{r}\) is a weight matrix learned through the convolutional neural network. The output of reset gate \(r_{t}\) determines whether the previous state is forgotten. When \(r_{t}\) is close to zero, the status information \(h_{i}\) of the previous moment will be forgotten, and the hidden status is reset to the current input. Similarly, the update gate of the \(t\)th moment \(z_{t}\) is computed by  
\[z_{t}\ =\ \sigma\Big{(}W_{z}\Big{[}f_{t}^{nr^{\prime}},h_{i}\Big{]}\Big{)}, \tag{8}\]  
where \(z_{t}\) is used to determine how much past information can continue to be passed on, \(W_{z}\) is a weight matrix. If the value of the update gate is larger, the state information introduced at the previous moment is more, and vice versa. In GRU, the new hidden state \(\widetilde{h_{t}}\) can be obtained through Equation (9):  
\[\widetilde{h_{t}}\ =\tanh(W\Big{[}r_{t}*h_{i},f_{t}^{nr^{\prime}}\Big{]}), \tag{9}\]  
where the new hidden state \(\widetilde{h_{t}}\) is determined by the value of the reset gate, \(W\) is a weight matrix, \(*\) denotes the element-wise multiplication. The actual output \(h_{i+1}\) is then computed by  
\[h_{i+1}\ =\ 1-z_{t}*h_{i}+z_{t}*\widetilde{h}_{t}, \tag{10}\]  
where some of the previous state \(h_{i}\) will be passed, and the new hidden state \(\widetilde{h_{t}}\) will be selectively updated. Through GRU, the scene module and the relation module can pass information to each other and constantly update new information. In this way, useful information will continue to be passed, and useless information will be ignored. Finally, richer information can be obtained through Equation (11).