# Serial Decoders-Based Auto-Encoders for Image Reconstruction
## 3 Theory
### Recall of Classical Auto-Encoders  
The architecture of classical auto-encoders is illustrated in Figure 2. Classical auto-encoders are composed of two units: encoder and decoder. The encoder reduces the high-dimensional input data to a low-dimensional representation, and the decoder reconstructs the high-dimensional data from the low-dimensional representation. The classical auto-encoder can be described by the following formulas:  
\[ \begin{split}\mathbf{z}=\mathrm{E}(\mathbf{x})\\ \mathbf{y}=\mathrm{D}(\mathbf{z})\\ \mathbf{x},\mathbf{y}\in\mathbb{R}^{\mathrm{H}};\mathbf{z}\in \mathbb{R}^{\mathrm{L}}\\ \mathrm{H}\gg\mathrm{L}\end{split}  \tag{1}\]  
where the terms are defined as follows:  
\(\mathbf{x}\) is the high-dimensional input data. Taking image data as an example, \(\mathbf{x}\) is the normalized version of original image for the convenience of numerical computation; each element of the original image is an integer in the range [0, 255]; each element of \(\mathbf{x}\) is a real number in the range [0, 1] or [\(-\)1, +1]; \(\mathbf{x}\) with elements in the range [0, 1] can be understood as probability variables; \(\mathbf{x}\) can also be regarded as a vector which is a reshaping version of an image matrix. \begin{table}
\begin{tabular}{c c} \hline \hline
**Notations and Abbreviations** & **Meanings** \\ \hline AE & auto-encoders \\ AAE/VAE/WAE & adversarial/variational/Wasserstein AE \\ CD & cascade decoders \\ GCD/RCD/ACD/RACD & general/residual/adversarial/residual-adversarial \\ CD/GCDAE/RCDAE/ACDAE/RACDAE/RACDAE & CD/GCD/RCD/ACD/RACD-based AE \\ CDAAE/CODAE/RCDAE/ACDAAE/RACDAE & CD/GCD/RCD/ACD/RACD-based AAE \\ CDVAE/CCDAVE/ACDVAE/CCDAVE/RACDAVE/RACDAVE & CD/GCD/RCD/ACD/RACD-based VAE \\ CDWAE/GCDWAE/RCDWAE/ACDWAE/RACDWAE & CD/GCD/RCD/ACD/RACD-based WAE \\ E/D/DC & encoder/decoder/discriminator \\ \(\mathbf{x}\)/\(\mathbf{y}\)/\(\mathbf{z}\) & original/reconstructed/latent sample \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Mathematical notations and abbreviations. Figure 2: The architecture of classical auto-encoders. \(\mathbf{z}\) is the low-dimensional representation in a latent space. \(\mathbf{y}\) is the high-dimensional data, such as a reconstruction image. E is the encoder. D is the decoder. H is the dimension of \(\mathbf{x}\) or \(\mathbf{y}\); for image data, H is equal to the product of image width and height. L is the dimension of \(\mathbf{z}\) and L is far less than H.  
The classical auto-encoders can be resolved by the following optimization problem:  
\[ \begin{split}(\theta,\mathbf{z},\mathbf{y})=\underset{ \theta,\mathbf{y},\mathbf{z}}{\operatorname{argmin}}\left\|\mathbf{y}- \mathbf{x}\right\|_{2}^{2}\\ \text{s.t.}\ \mathbf{z}=\text{E}(\mathbf{x}),\mathbf{y}=\text{D}( \mathbf{z}),\text{C}_{\mathbf{z}}=\left\|\mathbf{z}-\mathbf{z}_{\text{S}} \right\|_{2}^{2}<\delta_{\mathbf{z}},\text{C}_{\mathbf{y}}=\left\|\mathbf{y}- \text{D}_{\mathbf{y}}\mathbf{s}_{\mathbf{y}}\right\|_{2}^{2}+\lambda_{\mathbf{ y}}\left\|\mathbf{s}_{\mathbf{y}}\right\|_{1}<\delta_{\mathbf{y}}\end{split}  \tag{2}\]  
where the terms are defined as follows:  
\(\theta\) are the parameters of auto-encoders, including the parameters of the encoder and decoder. \(\text{C}_{\mathbf{z}}\) is the constraint on low-dimensional representation \(\mathbf{z}\); for example, \(\mathbf{z}\) satisfies a given probability distribution; it has been considered to match a known distribution by classical adversarial auto-encoders, variational auto-encoders, and Wasserstein auto-encoders. \(\mathbf{z}_{\text{g}}\) is a related variable which meets a given distribution. \(\delta_{\mathbf{z}}\) is a small constant. \(\text{C}_{\mathbf{y}}\) is the constraint on high-dimensional reconstruction data \(\mathbf{y}\); for instance, \(\mathbf{y}\) meets a prior of local smoothness or non-local similarity.