page_title: Autoencoder
section_title: Basic Architecture
hierarchical_section_title: Autoencoder / Basic Architecture
caption_attribution_description: English: Autoencoder schema
context_page_description: An autoencoder is a type of artificial neural network used to learn efficient data codings in an unsupervised manner. The aim of an autoencoder is to learn a representation for a set of data, typically for dimensionality reduction, by training the network to ignore signal “noise”. Along with the reduction side, a reconstructing side is learnt, where the autoencoder tries to generate from the reduced encoding a representation as close as possible to its original input, hence its name. Several variants exist to the basic model, with the aim of forcing the learned representations of the input to assume useful properties. Examples are the regularized autoencoders, proven effective in learning representations for subsequent classification tasks, and Variational autoencoders, with their recent applications as generative models. Autoencoders are effectively used for solving many applied problems, from face recognition to acquiring the semantic meaning of words.
context_section_description: The simplest form of an autoencoder is a feedforward, non-recurrent neural network similar to single layer perceptrons that participate in multilayer perceptrons (MLP) – having an input layer, an output layer and one or more hidden layers connecting them – where the output layer has the same number of nodes (neurons) as the input layer, and with the purpose of reconstructing its inputs (minimizing the difference between the input and the output) instead of predicting the target value  given inputs . Therefore, autoencoders are unsupervised learning models (do not require labeled inputs to enable learning).
  An autoencoder consists of two parts, the encoder and the decoder, which can be defined as transitions  and  such that:
  In the simplest case, given one hidden layer, the encoder stage of an autoencoder takes the input  and maps it to :
  This image  is usually referred to as code, latent variables, or latent representation. Here,  is an element-wise activation function such as a sigmoid function or a rectified linear unit.   is a weight matrix and  is a bias vector. Weights and biases are usually initialized randomly, and then updated iteratively during training through Backpropagation. After that, the decoder stage of the autoencoder maps  to the reconstruction  of the same shape as :
  where  for the decoder may be unrelated to the corresponding  for the encoder.
  Autoencoders are trained to minimise reconstruction errors (such as squared errors), often referred to as the "loss":
  where  is usually averaged over some input training set.
  As mentioned before, the training of an autoencoder is performed through Backpropagation of the error, just like a regular feedforward neural network.
  Should the feature space  have lower dimensionality than the input space , the feature vector  can be regarded as a compressed representation of the input . This is the case of undercomplete autoencoders. If the hidden layers are larger than (overcomplete autoencoders), or equal to, the input layer, or the hidden units are given enough capacity, an autoencoder can potentially learn the identity function and become useless. However, experimental results have shown that autoencoders might still learn useful features in these cases. In the ideal setting, one should be able to tailor the code dimension and the model capacity on the basis of the complexity of the data distribution to be modeled. One way to do so, is to exploit the model variants known as Regularized Autoencoders.
caption: Schema of a basic Autoencoder
exif: {}