# Complex Background Reconstruction for Novelty Detection
## 3 Methodology
### Mutual Information Siamese Network (MNet)  
We proposed a network called the mutual information Siamese network (MNet) to measure the similarity between reconstructed images and original images. The similarity judges the quality of the reconstructed image and determines whether the images are novelties. In the previous methods of novelty detection based on GAN/AE, they detect whether the sample is a novelty by calculating the reconstruction error of the input image \(x\) and the reconstruction image \(\hat{x}\). The general selection for the reconstruction error is the mean square error (MSE). However, RNet pays more attention to the object rather than the background for images with complex backgrounds. In this case, the quality of reconstruction is limited. In this work, we employ ideas from metric learning based on deep features. [PERSON] et al. [24] introduced the Siamese neural network to solve signature verification as an image matching problem. The Siamese neural network is made up of twin networks, and the parameters between them are shared. These two networks' distinct inputs are joined by an energy function in the end. Note that the Siamese network effectively mitigates the issue of imbalanced samples, making it particularly suitable in the current one-shot environment, which is suitable for the present case. The energy function computes some metrics between the highest-level feature representation on each side. As Figure 3 shows, we feed \(x\) and \(\hat{x}\) into the encoder network \(E\) to map them to the feature vectors in the feature space, and we determine whether the two samples are similar by calculating the distance between the two feature vectors. In one-class novelty detection, only samples from the normal class can be provided for training. Therefore, we lack the label when training with the Siamese neural network because we do not have an accurate measure of the reconstructed images of input samples. Thus, we train the network to extract the features of the training target in the input space without supervision. **Mutual Information Maximization Loss:** Based on [31], we use a training encoder to maximize the mutual information between original samples and reconstructed samples, so that the features of reconstructed ones can better represent the features of input ones. Let \(X\) denote the set of original images, and \(x\in X\) is an original sample. \(Z\) is the combination of encoder vectors. \(z\in Z\) is the encoding vector corresponding to \(x\). \(p(z|x)\) denotes the distribution of coding vectors produced by \(x\), which estimates the mutual information by training the discriminator to distinguish between samples coming from the joint \(\mathbb{J}\), and the product of marginals \(\mathbb{M}\). \(I(X,Z)\) denotes the mutual information between \(x\) and \(z\) and indicates the correlation between them. We hope that the encoder \(E\) extracts \(X\) more discriminative features \(Z\) as much as possible, which lets \(I(X,Z)\) be as large as possible:  
Figure 3: Mutual information Siamese network. Its purpose is to measure the similarity between the reconstructed image \(\hat{x}\) and the original image \(x\). \(E\) is the mutual information feature extractor, which extracts the features of \(x\) and \(\hat{x}\). Then, we measure the distance of \(E_{w}(x)\) and \(E_{w}(\hat{x})\). \[p(z,x)=\operatorname*{arg\,max}_{p(z,x)}I(X,Z) \tag{6}\]  
Following the formulation provided in [PERSON] et al. [31], we take Jensen-Shannon divergence (JSD) as the mutual information estimation. Therefore, the loss is defined as  
\[\mathcal{L}_{info} =\max\hat{\mathcal{I}}^{\text{(JSD)}}(X;Z)\] \[=\max\mathbb{E}_{\mathcal{I}}[\log\sigma(T(x,z))]-\mathbb{E}_{ \mathcal{M}}[\log\sigma(T(x^{\prime},z))], \tag{7}\]  
where \(x\) is an input sample, \(z\) is the high-level representation related to \(x\), and \(x^{\prime}\) is another input sample unrelated to \(z\). We also perform an a priori constraint on the encoder to give the encoder an a priori specific expected statistical feature.