# Dynamic Detection and Recognition of Objects Based on Sequential RGB Images
## 3 Framework of the System
### MVRFFNet  
MVRFFNet consists of two modules: the registering module and the feature mapping module. The former extracts multiview features of an object and combines them into a compact registered feature. The latter uses a WGAN-based framework [3] to bridge the gap between registered features and on-conveyor features.  
Denote the registering view set of \(N\) objects as \(\mathcal{V}=\left\{v_{ij,i},i=1,\cdots,N,j=1,\cdots,M\right\}\), the seen conveyor view set as \(\mathcal{U}=\left\{u_{i,k},i=1,2,\cdots,N_{1},j=1,2,\cdots,\infty\right\}\), and the unseen conveyor view set as \(\mathcal{S}=\left\{s_{i,j,k},i=1,2,\cdots,N_{2},j=1,2,\cdots,\infty\right\}\), where \(M\) is the number of views, and \(N\) satisfies \(N=N_{1}+N_{2}\). In the registering module, features of different views are extracted separately with a feature extractor \(R\) first; that is \(s_{i,j}=R(v_{i,j})\). \(s_{i,i}\) are fused to a registered feature \(r_{i}\) with a fusing network \(F\); that is \(r_{i}=F(s_{i,1},s_{i,2},\cdots,s_{i,M})\). \(F\) could be a simple max-pooling layer [2] or a recurrent neural network (RNN) [1] that evolves along with the variation of view. A graphic neural network (GNN) is also a qualified candidate fuser. \(E_{1}\) is trained separately, and \(F\) is trained together with the mapping module. The architecture is depicted in Figure 5.  
In the mapping module, the redundancy-free feature mapping framework [3] is adopted. The structure is depicted in Figure 6. The generator \(G\), with the concatenation of a registered feature and a noise vector as its input, generates a synthetic or fake on-conveyor feature. The noise represents the properties of difference between the registered and on-conveyor features, which includes environment light, background noise, the motion of the camera, and multiview modal to single-view modal. The mapping function \(M\) maps the on-conveyor features into a latent space, which is the redundancy-free feature space. \(D\) is the discriminator that is realized in the form of Wasserstein distance. Wasserstein distance is a symmetrical measurement for the difference between two random distributions. \(E_{2}\) is  
Figure 5: Architecture of the registering module of MVRFFNet.  
a feature extractor to obtain features from on-conveyor images, and \(C\) is a final classifier to predict the categories of latent features.  
Besides the usual fake loss and Wasserstein distance in WGAN, the mutual information (MI) loss based on Kullback-Leibler divergence and the center loss [3] are also considered when training \(G\) and \(M\).  
The training configuration of MVRFFNet is similar to that of [3], except that the attribute vector is replaced with a registering module and \(F\) is updated together with \(G\). \(E_{2}\) is trained separately, and it is frozen when training other parts. In each training batch, \(F\), \(G\), and \(M\) are updated once or several times simultaneously first; and then, \(M\) is frozen while \(F\) and \(G\) are updated at the same time. At the last step of a training epoch, \(F\), \(G\), and \(M\) are frozen, and \(C\) is trained separately. The details about loss functions and training strategies can be referred to in [3].