# On the Quality of Deep Representations for Kepler Light Curves Using Variational Auto-Encoders
## 2 Materials and Methods
### Extending VRAE to Include Temporal Information
#### 2.2.4 Loss Function  
In this section, we describe the optimization objectives of the proposed variational auto-encoder models (VRAE\({}_{t}\) and S-VRAE\({}_{t}\)) for the time series domain. **Reconstruction loss**: We used a modified version of the mean squared error (MSE) function for the un-scaled variable time series, named _weighted_ or _re-scaled_ MSE, given by  
\[\text{SMSE}(\mathsf{X},\hat{\mathsf{X}})=\frac{1}{N}\sum_{i=1}^{N}w^{(i)}\cdot \frac{1}{T_{i}}\sum_{j=1}^{T_{i}}\left(x_{j}^{(i)}-\hat{x}_{j}^{(i)}\right)^{2}, \tag{4}\]  
where the additional weight \(w^{(i)}\) is associated to every input pattern (as a sample weight [42]) and is defined as the inverse of the variance in the time series: \(w^{(i)}=(\text{var}(\tilde{\mathsf{x}}^{(i)}))^{-1}\). The weight value comes from \(w_{i}=(s^{(i)})^{-2}\), with the idea of normalizing and removing the intrinsic dispersion of the time series \(\tilde{\mathsf{x}}^{(i)}\), and so every input pattern has the same impact in the objective (note that SMSE is similar to the chi-square error of a linear least-squares fit with (uncorrelated) uncertainties that follows from a maximum likelihood approach). In order to perform a proper reconstruction of the input patterns, the SMSE is applied as a loss function for the deterministic counterpart of VRAE\({}_{t}\), i.e., the RAE\({}_{t}\) model. **Variational loss**: The two factors to optimize on the variational lower bound (Equation (1)) could be expressed by: (i) a reconstruction factor through the SMSE described above; and (ii) the closed solution for the KL divergence with normal priors [29]. Following [PERSON] et al. [43], we combine these two factors using a regularization parameter \(\beta\) obtaining the loss function of the VRAE\({}_{t}\) model:  
\[\text{V}(\mathsf{X},\hat{\mathsf{X}})=\text{SMSE}(\mathsf{X},\hat{\mathsf{X} })+\beta\cdot D_{\text{sz}}\left(\mathsf{Z}\|\mathcal{N}(\vec{0},\mathbf{I}) \right). \tag{5}\]  
Figure 3: Diagram of the S-VRAE\({}_{t}\) (VRAE\({}_{t}\)_with re-scaling_) architecture for time series. Firstly, the raw time series input is re-scaled and passed to the RNN encoder block together with the delta times. In parallel, the _F-std_ input is also encoded. A concatenation is performed on both learned embedding values to obtain the Normal distribution parameters of the latent variable. After a sample is performed over this distribution, the value is used to reconstruct the _F-std_. In parallel, the sample is repeated and concatenated with the delta times in order to reconstruct the scaled time series on the time space. Finally, the _F-std_ is returned to the time series through the network estimation. Our earlier experiments on the validation set led us to set the \(\beta\) hyper-parameter to a small value of \(10^{-3}\). This means that the priority is set on the reconstruction task but with a variational component of regularization. **Variational loss with re-scaling**: As our proposed S-VRAE\({}_{t}\) adds the \(F\)-\(std\) to the reconstruction task, we need to specify an additional loss that guides this reconstruction. We used the mean squared logarithm error (MSLE) as a loss function, given by  
\[\text{MSLE}(\vec{S},\hat{\vec{S}})=\frac{1}{N}\sum_{i=1}^{N}\left(\log s^{(i)} -\log s^{(i)}\right)^{2}. \tag{6}\]  
Then, the learning objective of the S-VRAE\({}_{t}\) model is given by  
\[\text{SV}\left((\mathbb{X},\vec{S}),(\hat{\mathbb{X}},\hat{\vec{S}})\right)= \text{V}(\mathbb{X},\hat{\mathbb{X}})+\alpha\cdot\text{MSLE}(\vec{S},\hat{ \vec{S}})\,, \tag{7}\]  
where the \(\alpha\) hyper-parameter was set to \(10^{-1}/\text{var}(\log\vec{S})\). The same motivation used for \(\beta\) led to set the \(10^{-1}\) value, firstly giving relevance to the reconstruction loss SMSE inside V. However, the \(\text{var}(\log\vec{S})\) value is to re-scale the values of the MSLE loss (by removing the standard deviations) to the same proportions of SMSE loss. The loss function used helps the models, guiding them into the right reconstruction.