Generative imaging and image processing via generative encoder

&lt;p style='text-indent:20px;'&gt;This paper introduces a novel generative encoder (GE) framework for generative imaging and image processing tasks like image reconstruction, compression, denoising, inpainting, deblurring, and super-resolution. GE unifies the generative capacity of GANs and the stability of AEs in an optimization framework instead of stacking GANs and AEs into a single network or combining their loss functions as in existing literature. GE provides a novel approach to visualizing relationships between latent spaces and the data space. The GE framework is made up of a pre-training phase and a solving phase. In the former, a GAN with generator &lt;inline-formula&gt;&lt;tex-math id="M1"&gt;\begin{document}$ G $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt; capturing the data distribution of a given image set, and an AE network with encoder &lt;inline-formula&gt;&lt;tex-math id="M2"&gt;\begin{document}$ E $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt; that compresses images following the estimated distribution by &lt;inline-formula&gt;&lt;tex-math id="M3"&gt;\begin{document}$ G $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt; are trained separately, resulting in two latent representations of the data, denoted as the generative and encoding latent space respectively. In the solving phase, given noisy image &lt;inline-formula&gt;&lt;tex-math id="M4"&gt;\begin{document}$ x = \mathcal{P}(x^*) $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt;, where &lt;inline-formula&gt;&lt;tex-math id="M5"&gt;\begin{document}$ x^* $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt; is the target unknown image, &lt;inline-formula&gt;&lt;tex-math id="M6"&gt;\begin{document}$ \mathcal{P} $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt; is an operator adding an addictive, or multiplicative, or convolutional noise, or equivalently given such an image &lt;inline-formula&gt;&lt;tex-math id="M7"&gt;\begin{document}$ x $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt; in the compressed domain, i.e., given &lt;inline-formula&gt;&lt;tex-math id="M8"&gt;\begin{document}$ m = E(x) $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt;, the two latent spaces are unified via solving the optimization problem&lt;/p&gt;&lt;p style='text-indent:20px;'&gt;&lt;disp-formula&gt; &lt;label/&gt; &lt;tex-math id="FE1"&gt; \begin{document}$ z^* = \underset{z}{\mathrm{argmin}} \|E(G(z))-m\|_2^2+\lambda\|z\|_2^2 $\end{document} &lt;/tex-math&gt;&lt;/disp-formula&gt;&lt;/p&gt;&lt;p style='text-indent:20px;'&gt;and the image &lt;inline-formula&gt;&lt;tex-math id="M9"&gt;\begin{document}$ x^* $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt; is recovered in a generative way via &lt;inline-formula&gt;&lt;tex-math id="M10"&gt;\begin{document}$ \hat{x}: = G(z^*)\approx x^* $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt;, where &lt;inline-formula&gt;&lt;tex-math id="M11"&gt;\begin{document}$ \lambda&amp;gt;0 $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt; is a hyperparameter. The unification of the two spaces allows improved performance against corresponding GAN and AE networks while visualizing interesting properties in each latent space.&lt;/p&gt;