# On the Quality of Deep Representations for Kepler Light Curves Using Variational Auto-Encoders
## 2 Materials and Methods
### Extending VRAE to Include Temporal Information
#### 2.2.2 VRAE Including Delta Times  
A VAE, as any other auto-encoder architecture, is composed of a tangled encoder and decoder models trained on an unsupervised scenario (unsupervised refers to the fact that no labels are used as inputs to the model). The encoder model \(q_{\phi}(\vec{z}|\vec{x})\), with parameters \(\phi\), codifies the input pattern \(\vec{x}\) to a multi-dimensional latent variable \(\vec{z}\), which has fewer dimensions than the original space (data compression). The decoder model \(p_{\theta}(\vec{x}|\vec{z})\), with parameters \(\theta\), reconstructs the input pattern from the codification \(\vec{z}\). The objective of the model is to maximize a (variational) lower bound \(\mathcal{L}(\theta,\phi;D)\) of the log-likelihood \(\ell(\theta,\phi;D)\)[29]. For example, for an input pattern \(\vec{x}\), we have:  
\[\ell\geq \mathcal{L}(\theta,\phi;\vec{x})=\mathbb{E}_{q_{\phi}(\vec{z}| \vec{x})}\left[\log p_{\theta}(\vec{x},\vec{z})-\log q_{\phi}(\vec{z}|\vec{x} )\right]\] \[\mathcal{L}(\theta,\phi;\vec{x})=\mathbb{E}_{q_{\phi}(\vec{z}| \vec{x})}\left[\log p_{\theta}(\vec{x}|\vec{z})\right]-D_{\text{\tiny{KL}}} \left(q_{\phi}(\vec{z}|\vec{x})\|p_{\theta}(\vec{z})\right)\,, \tag{1}\]  
where the first term of \(\mathcal{L}\) is related to the expected reconstruction likelihood and the second enforces the consistency between the posterior obtained by the encoder \(q_{\phi}(\vec{z}|\vec{x})\) and some prior \(p_{\theta}(\vec{z})\), based on the Kullback-Leibler (KL) divergence [38]. In the standard VAE, the distribution \(q_{\phi}(\vec{z}|\vec{x})\) is typically normal, \(\mathcal{N}(\mu(\vec{x}),\text{diag}(\sigma(\vec{x})))\), where \(\mu(\vec{x})\) and \(\sigma(\vec{x})\) are modeled with neural networks yield vectors. However, the common choices of \(p_{\theta}(\vec{z})\) lead to a KL divergence that can be analytically integrated. However, the first term of \(\mathcal{L}\) needs to be approximated with the so-called _re-parametrization trick_. \(\hat{\vec{z}}=\vec{\mu}+\vec{\sigma}\odot\vec{x}\), with an auxiliary noise variable \(\vec{\epsilon}\sim\mathcal{N}(\vec{0},\mathbf{I})\). The operator \(\odot\) symbolizes the Hadamard product, i.e., an element-wise product of vectors and matrices.  
The fully connected VAE for light curves proposed by [35] could be extended to adapt RNN models into the encoder \(q_{\phi}(\vec{z}|\vec{x})\) and decoder \(p_{\theta}(\vec{x}|\vec{z})\) as a VRAE model. However, to adapt the model to incorporate the time information of the light curves, we need the time intervals (\(\vec{\delta}\)) as an extra input channel. To do this, we followed the idea of the RAE\({}_{t}\) archi  
Figure 1: Examples of Kepler mission light curves. The first column corresponds to 4 years’ worth of detrended measurements with a sampling rate of half an hour, while the second column corresponds to the phase-folded transit. The time unit is BKJD, with \(\text{BKJD}=\text{BJD}-2454833\).  
tecture proposed by [PERSON] et al. [10] (shown in Figure 2), which adds the time information in both encoding and decoding sections of the auto-encoder. The proposed extension is named \(\mathbf{VRAE}_{t}\) model (VRAE _plus time information_). This model could be expressed by the encoder \(q_{\varphi}(\vec{x}|\vec{x},\vec{\delta})\) and the decoder \(p_{\theta}(\vec{x}|\vec{x},\vec{\delta})\), which have the time information \(\vec{\delta}\) of every observation on the time series \(\vec{x}\) as an extra input signal.  
The motivation behind the variational extension is that the generative (stochastic) model learns a latent variable with smoother transitions, meaning that their latent space must be continuous [29]. Furthermore, since the model learns the distribution of the encoded variable, it becomes robust to input variations, similarly to a denoising auto-encoder [31]. Indeed, the learned distribution must have more likely regions or confidence intervals where the input data should be.