# Extended Autoencoder for Novelty Detection with Reconstruction along Projection Pathway
## 3 Preliminaries of RaPP
### Implementation of Approximate \(\vec{f}\)  
In this section, we discuss why the abstract decoder \(\vec{f}\) should be implemented to improve the RaPP method, and we show how the adversarial training can force \(f\) to be an approximate \(\vec{f}\). To use the RaPP method, the hidden reconstruction of the input must be equal to the hidden activation of the reconstructed input. For this computation, a necessary underlying assumption is that the autoencoder is able to perfectly copy the input \(x\). Practically, however, this assumption is not acceptable. The autoencoder has constraints on its network; thus, it is not capable of perfectly reconstructing the input. The aim of the autoencoder is not to be the identity function but to learn the compressed representation of the input. Because \(f(g(x))\) and \(\vec{f}(g(x))\) only approximately copy the input \(x\) in their own way, \(\vec{f}(g(x))=f(g(x))\) is no longer valid. It is still true that RaPP works because the autoencoder at least attempts to copy the input \(x\). However, we expect that using the abstract decoder \(\vec{f}\), rather than decoder \(f\), will improve RaPP performance. The implementation of \(\vec{f}\) can be difficult even though that neural networks are highly flexible frameworks. Thus, the number of criteria required may approach the number of layers needed to force all deep encoder-decoder pairs to be inverses of each other. As such, autoencoder learning is susceptible to failure given the extensive criteria. To avoid this failure, we focus only on the last layer and extend this to the whole encoder-decoder system to implement the approximate \(\vec{f}\).  
Since we only consider the last layer, the second condition of \(\vec{f}\) can be expressed as follows:  
\[g(x)=(g_{l}\circ\vec{f}_{l})(g(x))\]  
As the second condition of \(\vec{f}\) holds that \(\vec{f}_{i}=g_{i}^{-1}\), it is also true that \(\vec{f}=g^{-1}\). Thus, we can substitute \((g_{l}\circ\vec{f}_{l})\) with \((g\circ\vec{f})\) in this equation. In addition, the extended autoencoder can be based on a configuration in which a generative autoencoder is trained to match the encoding distribution with the prior distribution. In this paper, we set the encoding distribution \(g(x)\) to the standard Gaussian distribution \(z\ \sim N(0,1)\). Finally, we can define an approximate \(\vec{f}\) such that  
\[z=(g\circ\vec{f})(z)\]  
This equation states that the latent variable of the input (the left side of the equation) is equal to the latent variable of the reconstructed input (the right side of the equation). To implement the  
Figure 6: Architecture of the extended AAE, which is based on the AAE architecture, with the addition of discriminator \(D_{l}\). The dotted encoder is not updated during the training.  
We selected a semi-supervised learning approach for novelty detection [15]. Thus, we provided only normal samples during the training phase and used both normal and novelty samples during the testing phase. Half of the test sets were made up of normal samples, and the other half were novelty samples. After training the autoencoder, we used the RaPP method to calculate the novelty score by normalizing and aggregating the hidden activation values of an input and its autoencoder reconstruction. With this novelty score, we evaluated novelty detection performance using the area under the receiver operating characteristic curve (AUROC) [16]. To alleviate random errors during training, we obtained the AUROC by averaging AUROC scores from 30 trials for novelty datasets and 5 trials for benchmark datasets.