# On the Quality of Deep Representations for Kepler Light Curves Using Variational Auto-Encoders
## 2 Materials and Methods
### Extending VRAE to Include Temporal Information
#### 2.2.3 VRAE with Embedded Re-Scaling  
Currently, deep learning methods need a standardized version of the input representation \(\vec{x}^{(i)}\) that retains the original distribution but re-scaled to more tractable magnitudes. This is used for properly training neural network models, based on the generalization principle that magnitudes of weights and activation functions must be somehow bounded [39; 40]. Furthermore, [PERSON] & [PERSON] [41] recommend that a normalization step is added after each layer to obtain more stable training. On a time series, this transformation is usually based on its own magnitude behavior, e.g., the range, standard deviation, and norm. For example, \(\vec{x}^{(i)}=(\vec{x}^{(i)}-\min(\vec{x}^{(i)}))/(\max(\vec{x}^{(i)})-\min( \vec{x}^{(i)}))\) change the range magnitudes to \([0,1]\), or \(\vec{x}^{(i)}=(\vec{x}^{(i)}-\mean(\vec{x}^{(i)}))/\std(\vec{x}^{(i)})\) could lead to \(\vec{x}^{(i)}\sim\mathcal{N}(0,1)\) if \(\vec{x}^{(i)}\) is normally distributed. These transformations are necessary for the model to detect pattern behaviors instead of magnitude behaviors. The problem is that the information on the original scale of variability is lost in the process, which can be useful in some cases. Focused on light curves, we propose adding the Flux standard deviation \(s^{(i)}\) of the flux time series \(\vec{x}^{(i)}\) as an additional input to the \(\mathrm{VRAE}_{t}\) model, but still use the re-scaled version of the data for achieving bounded weights and activations. To the best of our knowledge, this is the first approach to do this as a end-to-end architecture. Figure 2: Diagram of the \(\mathrm{RAE}_{t}\) (RAE _plus time information_) architecture for irregularly sampled time series proposed by [10]. The sequence is processed by recurrent layers and produce a final fixed-length embedding with a single fully connected layer on last state. The decoder first repeats this embedding \(T_{i}\) times, and then appends the delta times (\(\delta_{t}\), in Figure \(\Delta t\)). To determine the points at which the function will be evaluated, the delta times are input to both the encoder and decoder. We consider the _F-std_\(s^{(i)}\) (Section 2.2.1 for problem notation) as another input pattern of the auto-encoder model \(\text{VRAE}_{t}\) that also needs to be reconstructed. With this, the objective of the \(\text{VRAE}_{t}\)_with re-scaling_ or **S-VRAE\({}_{t}\)** is to reconstruct the time series on the un-scaled values, based on the _F-std_. **S-VRAE\({}_{t}\) architecture**: The outline of the main components of the S-VRAE\({}_{t}\) model which clarify the differences with respect to \(\text{VRAE}_{t}\), as summarized here:  
1. Re-scale data: The first layer of the encoder \(q_{\phi}(\cdot)\) re-scales the data by dividing on the _F-std_. This step is performed in order to use the standardized version of the data, as the literature recommends. 2. Encode: The encoder \(q_{\phi}(\cdot)\) adds the _F-std_ as an input pattern to the coding task by \(q_{\phi}(\vec{z}^{(i)}|\vec{x}^{(i)},\vec{\delta}^{(i)},s^{(i)})=\mathcal{N} (\vec{\mu}^{(i)},\text{diag}(\vec{\sigma}^{(i)}))\) in order to extract the information on it. 3. Sample: The sampled latent variable is given by: \(\vec{\bar{z}}^{(i)}=\vec{\mu}^{(i)}+\vec{\sigma}^{(i)}\odot\vec{\epsilon}\), with \(\vec{\epsilon}\sim\mathcal{N}(\vec{0},\mathbf{I})\). 4. Reconstruct: The decoder \(p_{\theta}(\vec{x}^{(i)}|\vec{z}^{(i)},\vec{\delta}^{(i)})\) adds the _F-std_ to the reconstruction task in order to estimate the original _F-std_ by \(p_{\theta}(s^{(i)}|\vec{z}^{(i)})\). 5. Re-scale reconstruction: The last layer of the decoder re-scales the data, by returning the reconstructed _F-std_ (multiplied by it). This final step is performed in order to obtain a reconstructed time series on the un-scaled values' representation. Based on the high variability in the range of values of the _F-std_ and inspired by [41], we introduced a normalization logarithm layer (\(Norm_{L}\)) that is applied to \(s^{(i)}\).