# Extended Autoencoder for Novelty Detection with Reconstruction along Projection Pathway
## 3 Preliminaries of RaPP  
RaPP, a method for detecting novelty samples by exploiting hidden activation values, compares the input data and the reconstructed data, similar to ordinary reconstruction-based methods; however, RaPP also extends these comparisons to hidden spaces. An intuitive application of this idea would be to compare the activation in hidden space and the corresponding reconstruction in that hidden space; however, this direct comparison has no effect due to the lack of an explicit correspondence between members of encoding-decoding pairs of hidden layers during training. However, RaPP can compute the hidden reconstructions indirectly, using the hidden activations of the reconstructed input. This indirect computation is depicted in Figure 3.  
Hidden activations of the reconstructed input are shown to be equivalent to the corresponding hidden reconstructions of the original input. For this computation of hidden reconstruction, RaPP assumes the existence of abstract decoders, which are inverse functions of corresponding encoders. An autoencoder \(A=f\circ g\) is a pretrained neural network, such that \(g\) and \(f\) represent the encoder and the decoder, and \(l\) is the number of hidden layers of \(g\) and \(f\) where \(g=g_{l}\circ\cdots\circ g_{2}\circ g_{1}\) and \(f=f_{1}\circ f_{2}\circ\cdots\circ f_{l}\). The autoencoder can completely reconstruct the input \(x\): i.e., \(x=A(x)\). Partial computation of \(g\) and \(f\) are as follows:  
\[g_{:i}=g_{i}\circ\cdots\circ g_{1}\]  
\[f_{l:i}=f_{i}\circ\cdots\circ f_{l}\]  
That is, \(g=g_{:l}\) and \(f=f_{l:1}\). The \(i\)-th hidden activation of the reconstructed input is defined by  
\[\hat{g}_{:i}(x)=g_{:i}(A(x))\]  
Let \(l\) be the number of hidden layers, and let us assume that there exists an abstract decoder \(f=\hat{f_{1}}\circ\cdots\circ\hat{f_{l}}\) such that  
\[\hat{f}(g(x))=f(g(x)) \tag{1}\]  
\[g_{:i}(x)=\big{(}g_{i}\circ f_{i}\big{)}(g_{:i}(x)) \tag{2}\]  
The first condition indicates that an abstract decoder \(\hat{f}\) can, like the decoder \(f\), perfectly reconstruct the input \(x\). The second condition states that \(f_{i}\) is the inverse function of \(g_{i}\). Then, the \(i\)-th hidden reconstruction is defined by  
\[\hat{g}_{:i}^{\prime}(x)=(\hat{f}_{i+1}\circ g)(x)\]  
Figure 3: The reconstruction along projection pathway (RaPP) method. The hidden activation of the reconstructed input \(\hat{h}_{i}\) is equivalent to the hidden reconstruction of the input \(\hat{h}_{i}^{\prime}\). Comparing the hidden activation of input \(h_{i}\) with the hidden activation of the reconstructed input \(\hat{h}_{i}\) can improve novelty detection.  
In the sections below, we first describe the architecture and the training procedure of the proposed model in detail. We also applied our extended autoencoder model to autoencoder variants, such as VAE and AAE. Second, we explain how the extended autoencoder can improve novelty detection with RaPP. Computation of the hidden reconstruction is tractable by assuming the existence of abstract decoders \(\tilde{f}\). On the premise that the autoencoder can copy the input \(x\) perfectly, there is no need to implement the abstract decoder. However, in practice, the abstract decoder should be implemented because the autoencoder assumption is no longer true.