Disentangling Time Series Representations via Contrastive Independence-of-Support on l-Variational Inference
Authors: Khalid Oublal
Said Ladjal
David Benhaiem
Emmanuel LE BORGNE
François Roueff
Venue: ICLR 2024 poster (2024)
Keywords: Learning disentangled representations
Generalization
Weak supervised learning
Appliance usage Electricity
Multi-modal learning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Disentanglement under Independence-of-Support via Contrastive Learning based $l$-Variational Inference for appliance usage electricity, addressing realistic correlations during training to capture real-world complexity.

Abstract: Learning disentangled representations for time series is a promising path to facilitate reliable generalization to in- and out-of distribution (OOD), offering benefits like feature derivation and improved interpretability and fairness, thereby enhancing downstream tasks. We focus on disentangled representation learning for home appliance electricity usage, enabling users to understand and optimize their consumption for a reduced carbon footprint. Our approach frames the problem as disentangling each attribute's role in total consumption. Unlike existing methods assuming attribute independence which leads to non-identiability, we acknowledge real-world time series attribute correlations, learned up to a smooth bijection using contrastive learning and a single autoencoder. To address this, we propose a Disentanglement under Independence-Of-Support via Contrastive Learning (DIOSC), facilitating representation generalization across diverse correlated scenarios. Our method utilizes innovative \textit{l}-variational inference layers with self-attention, effectively addressing temporal dependencies across bottom-up and top-down networks. We find that DIOSC can enhance the task of representation of time series electricity consumption. We introduce TDS (Time Disentangling Score) to gauge disentanglement quality. TDS reliably reflects disentanglement performance, making it a valuable metric for evaluating time series representations disentanglement. Code available at https://institut-polytechnique-de-paris.github.io/time-disentanglement-lib.

Published as a conference paper at ICLR 2024
DISENTANGLING TIME SERIES REPRESENTATIONS VIA CONTRASTIVE ANDEPENDENCE-OF-SSPPORR ON 1-VARIATIONAL INFERENCE
Khalid Oublal Saïd Ladjal', David Benhaiem*, Emmanuel Le-borgne*, François Roueff Institute Polytechnique de Paris, Telecom Paris LTCI/22A, 'OneTTech TotalEnergies, DS&AI PPrimary contact: khali i c yteclyteciiigo que.edu edu
ABSTRACT
Learning disentangled representations for time series is promising path facili- tate reliable generalization to in- and out-of distribution (OOD), offering benefits like feature derivation and improved interpretability and fairness, thereby enhancing downstream tasks. We focus on disentangled representation learning for home appli- ance electricity usage, enabling users understand and optimize their consumption for reduced carbon footprint. Our approach frames the problem as disentangling each attribute's role in total consumption. Unlike existing methods assuming at- tribute independence which leads to non-identiability. we acknowledge real-world time series attribute correlations, learned up to a smooth bijection using contrastive learning and single autoencoder. To address this, propose Disentanglement under Independence-Off-e Of-pppor via Contrastive Learning (DTOSC), facilitating representation generalization across diverse correlated scenarios. Our method utilizes innovative /-variational inference layers with self-attention, effectively addressing temporal dependencies across bottom-up and top-down networks. We find that DIOSC can enhance the task of representation of time series electricity consumption. We introduce TDS (Time Disentangling Score) gauge disentan- glement quality. TDS reliably reflects disentanglement performance, making it a valuable metric for evaluating time series representations disentanglement. Code available at https: //tttt//// tut- ppo ytechn que de pa github. io/time- diw.imeeddieettaeeeeee anglenenn -lib.
INTRODUCTION
Realistic Static Independences Among Factors?
Disentangled representation learning is cru- cial in various fields like computer vision, speech processing, and natural language pro- cessing (Bengio et al., 2014). There have been efforts learn disentangled time series represen- tation (Woo et al., 2022; Yao et al., 2022), with the aim improve generalization, robustness, and explainability. A core task in representation learning provable representation identifica- tion. We call a representation disentangled when identified attributes in the data are specifically coded in the structure of its latent units. How this can be achieved remains an open research question. In (Locatello et al., 2019), it is shown that disentangling requires some kind of supervised learning and inductive bias. Moreover, standard methods such as 3- VAE (Higgins et al., 2016), TCVAE (Chen et al., 2018c), and rely the too stringent assumption of statistical independence among ground truth attributes. In real-world time series, attributes are often correlated. In the application of this study, the attributes correspond the contributions of specific devices in an aggregated consumption signal. We illustrate the correlation of the attributes in Fig 1, where green boxes contain typical consumption for their respective appliances
smunsal possibbe
Dountly
MM
Usunrly
Profte
Wasching Macchine
Dishwasher
Applionce
Figure 1:
Time series real-world often showcases attributes exhibiting strong correlation.

---

Published as a conference paper at ICLR 2024
Pelozing Independency Fortoriation independent
Sampligant Augnentiiinn
Eecoding
Dccoding
Aapragan Powers
Z
X
Enoadirr
Decodar vakeetrrr Layer
L- Veterrrr
X
f
g4
Aec
Zi
Ym
Hellaving Independenry
Afract
Anchor Secuence, with upo Rariee
aspfoonceee Condation withj
With xpplaneem:: [-] Loowr Cormastion with m
Figure 2: Disentanglement under saddependence-o-SSupppee via Contrastive (DIOSC), Representations of positive pairs attract each other, while negative repels their corresponding representations. The latent attributes exhibit causal correlations (Shanmugam, 2018), DIOSC allows for scenarios where unlikely (but exist in data) combinations occur ((i) and (ii) leading to the existence (iii).. It's worth noting that forcing strict statistical independence does not prevent these cases.
and purple ones show consumption profiles that are correlated to those of the other appliances. Existing methods this context, such as B-VAE, TCVAE, often assume independent attributes. This paper provides a unified framework for disentangling time series by relaxing the assumption of statistical independence in the latent representation. To illustrate this, we will focus on a crucial application of time series disentanglementt household energy consumption disaggregation, also known as Non-Intrusive Load Monitoring (NILM). Given only the main consumption of a houschold _exT seen e-variate time series observed times L, ..... I. the NILM algorithm identifies the active consumption Ym E IRT of each operating appliance m. Such a task has received a growing interest and still raises unsolved problems. The fact that many households rely past bills to adjust future energy use underscores the importance of energy disaggregation aaciiionnlgono algorithms in reducing carbon footprints. Recent work (Bucci et al., 2021; Nalmpantis Vrakas, 2020) hold promising results, yet challenges in generalization to in- and out-of distribution. In this paper, we address NILM with disentanglement perspective, we assume that different appliance m need have latent 2m E R On downstream disaggregation task, we show that disentangled improves generalization to distribution shifts. We draw on connection between contrastive learning and identifiability in the form of Nonlinear ICA (Jutten et al., 2010; Hyvärinen, 2013).
Contributions and Main results. Our approach stands out by relying on weak contrastive learning using support factorization on the prior (rather than strict statistical independence) and Attentive I- Variational Inference. We evaluate our method qualitatively and quantitatively across various datasets with ground-truth labels, examining the generalization capabilities of the learned representations on correlated data. In summary:
[1] We define new regularization term and its theoretical justification, DICSC, whose goal is to address latent space misalignment issues and preserve disentanglement. This is achieved by promoting specific Pairwise (dis-)stie fhiss)similariiess over the latent sub-variables (c.f. $ 4).
[2] Our experiments across three datasets and diverse correlation scenarios demonstrate that DIOSC significantly enhances robustness to attribute correlation, yielding up to +61.4% for reconstruc- tion and +21.7% % improvement in disentanglement metrics (RMIG, DCI, and TDS) compared to state-of-the-art methods (c.f. $ 5.3).
[3] We propose /-variational pirratiioallbbsedd self-attention for extracting high semantic representations from time series, ensuring complex representations without temporal locality.
[4] To evaluate disentanglement we proposed TDS score, aling with the performance in the down- stream task. We implemented our framework in user-friendly library, making it the first time-series disentanglement framework.
2 RELATED WORK
Time Series Disentanglement in the Realm of Correlated Attributes. Traditional methods for time series disentanglement often emphasize enforcing statistical independence among representa- tion dimensions (Do & Tran, 2021; Klindt et al., 2020), even when dealing with highly correlated data. In recent computer vision disentanglement methods, there has been an exploration of using auxiliary information to improve identifiability, moving away from the assumption of statistical
2

---

Published as a conference paper at ICLR 2024
independence (Roth al,, 2023). However, both (Träuble et al., 2021; Roth et al., 2023) point out the limitations of this approach due to non-identifiabiiity. Another study by (Wang & Jordan, 2022a) proposes support factorization for disentanglement from causal perspective, incorporating a Hausdorff objective akin to (Roth et al., 2023). In our unique approach, we tackle time series disentanglement without explicit auxiliary variables or prior models. Instead, we achieve pairwise fac- torized support through contrastive learning, departing from the traditional independence assumption. Recent contributions (Wang & Jordan, 2022b; Roth et al., 2023) seek alleviate this assumption, yet remain disconnected from observational data and grapple with numerical stability. This method pioneers disentanglement in correlated time series by emphasizing independence--ff -upppor through contrastive learning during training. This contrasts with methods like (Ren et al., 2021), where representation discovery relies on contrastive learning of pre-trained generative models with assumed independence factorization during training. To the best of our knowledge, the present work provides the first identifiability and disentanglement result for time series in real correlated scenarios.
On The Non- onn-Intrusive Load Monitoring and Representation Learning. Recent work (Bucci et al., 2021; Nalmpantis & Vrakas, 2020) has produced promising results for separation source power. Nevertheless, they encounter challenges related to generalization and robustness when confronted with out-of-distribution scenarios. Several approaches have been suggested to address these challenges. Some methods tackle them through either transfer learning by enhancing the learned representations for each individual appliance. Exploring ways to enhance representation learning in this field has been the focus of recent studies (Woo et al., 2022; Vahdat & Kautz, 2021; Maalee et al., 2019). However, achieving an informative and disentangled representation remains an open and challenging question. Existing models, like ENN-VAE (Chung et al,, 2015) for sequential data and D3VAE (Li et al., 2023), assume statistically independent attributes. This assumption hampers their performance on real-world data and makes them less applicable to -f---iiitributionn scenarios. Developing models that effectively capture informative and disentangled representations in realistic and versatile manner continues to significant challenge.
3 PROBLEM STATEMENT AND PRELIMINARIES
Our approach belongs to the general framework of Variational Auto-Encoders VAEs, and thus relies on two main ingredients: 1) a variational family (ge), which approximates the conditional density of the latent variable given the observed variable based on an encoder fa: 2) a generative model (pe) based on a latent variable, and a decoder go. We consider C-variate time series observed at times / I,...,T, T, we denote by E RCx1 the C x T resulting matrix with rows denoted by I1. sC. Each row can be seen as a univariate time series. The goal is to recover the following decomposition of the active power Tc=1 = y + { where y is matrix with M columns ym E IR denotes the contribution of the m-th electric device, among the total of M devices identified, and E R contains the contribution of K unknown sources and/or additive noise. The NILM mapping, denoted as x ++ y, is typically learned from a training set X' = {x, M where each X; = (x; y;) represents pair of input-output samples used for training purposes. In VAE, both (unknown) parameters 0 and e are learnt from the training set X. A key idea for defining the goodness of fit part of the learning criterion is to rely the Evidence Lower Bound (ELBO), which provides a lower bound on (and a proxy of) the likeliiddd
log po (x) 2 (x)() log (l.(gg(2(X)) || p(z))
(3.1)
where we denoted the latent variable by z, defined as a de x (M+ K) matrix and p denotes its distribu- tion. The use of ELBO goes back to traditional variational Bayes inference. The encoder So provides an approximation of z = {zi...., , ZM+K] }} ~ Ps from x while y == go (z). A standard choice in a VAE is to rely on Gaussian distributions and, for instance, to set 4s ( x) (z; plx, Φ), ox d)). where ,x(x, Φ) and o² [x, Φ) are the outputs of the encoder fa. As discussed in Section 1. various crite- rion functions such as 3/TC/Factor/DIP VAE have been introduced, aiming to learn a disentangled la- tent variable and align it with the corresponding attributes. However, these methods typically assume statistical independence among attributes, leading to the assumption: p(z) = p(21) ... p(*M+() In the real world, this assumption does not hold, appliances are not used independently; rather, they are used simultaneously, and their profiles may exhibit correlation (though less likely). thereby challenging the validity of Independent Factorization.
3

---

Published as a conference paper at ICLR 2024
Definition 1. Independene-of-Suppor Factorization (IOS). For a latent variable z (21, RM+ K} sampled from p(z), if 2 21 x x... x 2M+K. K. where Z is the support of p(z), and 2, denotes the supports of marginal distributions of zy. then z exhibits Independence-of-Suppor. To address this, we propose a contrastive pairwise similarity to strengthen the constraint for better representation identification with observational data. As starting point, we assume that observed data samples y of appliance powers are generated from set of latent random vectors z through diffeomorphism' go Z - X. mapping from a latent space Z to an observation space y C X,
Px
y go (2).
(3.2)
The only assumption we place on Pa is that it is fully supported on Z. In particular, we do not require independence and allow for arbitrary dependencies between components of, motivated by the fact that the properties of certain time series profiles may be correlated with those of other profles.
4 DISENTANGLING UNDER IOS VIA CONTRASTIVE THEORY
Under the problem of non-identifiability, to solve the NILM problem effectively in real-world scenarios, we seek an ideal disentangled representation. We use single encoder/decoder for simultaneous and more efficient disentangling. To achieve this, we employ the following strategies: 1) Disentanglement and saaleeendenceoo-SSuppeee
via Contrastive Learning (DI which first promotes similarity between the latent representations Zm when the device m is present in both x; and its augmentation while inducing dissimilarity in negative cases; 2) we propose using an Attentive I- Variatiionl
enhance the model's 's ability to capture intricate patterns and achieve robust reconstruction. In the upcoming sections, will delve into the theoretical underpinnings of our proposed methods.
integrate self-attentie mechanisms (Vaswani et al., 2017)
4.1 DISENTANGLING UNDER INDEPENDENT-OR-SPPRN
VIA CONTRASTIVE (DIOSC)
Given appliance power sampled from the generative model outlined in Eq. (3.2), we now seek to understand under what conditions an inference model so X + 2 will provably identify the ground-truth latent representations. Ideally, we would like fo recover the true inverse fo :- ge but that is generally only possible up to certain irresolvable ambiguities. In our NILM setting, the objective is to separate the power representations such that each inferred latent captures one and only one ground-truth latent one individual latent block, we can learn fully disentangled representation. To this end, we define block affine identification: the true latent variables are block cookkiidettifiedd by a function fo X + Z if the inferred latent 2 (x) contains all and only information about i... if there exists some smooth invertible mapping I'; : dim Z) + dim(Z) s.t. 2 = I(2). This identification can be connected to disentnaglement under IOS Def. 1.
Theorem (Identifiability and Disentanglement), Suppose the observational data is generated
from Eq. (3.2) under the following assumptions:
(i) The support of Pz satisfies Def. 1, and interior of the support is a non-empty subset of Rdd, and for n + m, a pair (2n, &m) satisfies 2n,m = 2n x 2mm-
(ii) x satisfies positivity condition: for every m, we have p(zm (x) > 0 if and only p( > 0,
under this condition, if the representation = (21,...,7M), is disentangled, then: The
support of each dimension m remains unchanged whether conditioned on other dimensions
n + m or not.
A unique disentangled representation is then defined by the autoencoder (foz ga) that solves Eq. (4.4) achieves Permutation, Translation, and Scaling identification, i.e., Vz E 2,2 = Allz + Const., where à is the output of the encoder fo. z the true latent and II a permutation matrix and A is an invertible diagonal matrix.
To demonstrate the validity of Thm. 1, we rely on (Thm. 5.3, Ahuja et al. (2023)) and (Thm. 9, Wang & Jordan (2022b)). The full proof is provided in App. A. Thm. 1 demonstrates that ensuring independence between the supports of latent variables is key to achieving identification in observational data, allowing for permutation, shift, and scaling transformations. Representation encodes the same data properties, specifically, two representations and its augmented version z
A bijective function between differentiable manifolds that is smooth and the inverse is also smooth.
4

---

Published as a conference paper at ICLR 2024
satisfy the same sigma algebra (z)) = (z²)) if bijective function I' exists such that = T To achieve this, we use a contrastive objective to learn representations that enforce identifiability and disentanglement via support independence, leaverged in Thm. 1. Additionally, this perspective offers an alternative understanding of identifiability of Zimmermann et al. (2022), as it has been shown for the hypersphere, and convex bodies 2, the minimization of the adapted objective function Lor solve the unmixing problem of non-linear ICA.
efo(x) fo(x*)//
Lculfit, := (x.X P L
E
log
(4.1)
Tim1 Pdats
(ox(x)" + Σ elo6x))
Here N Z+ is a fixed number of negative samples, Pdata is the distribution of all observations and Ppositive is the distribution of positive pairs. In this equation the sum in the numerator extends to all N negative pairs generated and its size depends on the batch size. In multiclass settings like NILM, the contrastive loss (Eq. 4.1) struggles when multiple samples share the same appliance latent 2m due to labeled data. A suggested generalization (Khosla et al., 2021) tackles labeled cases. However, drawbacks include the lack of invariance and challenges with limited noisy labels, especially in obtaining both negative and positive labels for time series.
Disentanglement, Invariant, and Axis- llggmment Latents. When the m-th component of denoted by 2m. remains invariant regardless changes in x, 2m is meaningless and contains no information from x. We consider that the latent space aligned when the variations of latent variables 2m only have an influence on the mnth output of the decoder ge applied to z. Both terms form the basis of the disentanglement principle. Thus, z considered disentangled when there a one-to-one correspondence between each ground truth 3/m and the corresponding 4m in the representation. To uphold this despite the constraint of limited labels, hard to get both negative and positive in NILM, and without relying on static attribute independence, we leverage weak contrastive learning (Zimmermann et al., 2022; Zbontar et al., 2021), and adjusting it for disentanglement, Rather, to overeome the constraints of Eq. (4.1) and establish invariance and alignment in a single step, we extend contrastive Inveriinn
objective Zbontar et al. (2021). Our objective, integrates two core components: latent-Alignnet Alignment component seeks to minimize information overlap between 2m and its negatives i latent-Al compelling similarity between 2m and its augmented z/h accommodating potential changes to cover the variability factor of variation in ground-truth attributes, ensuring both invariance, alignment and enabling the discrimination task. To further enforce (i)-Thm. 1) empirically, we demonstrate how this constraint could encompass an extra assumption relaxing IOS Asm. 4.1.
Assumption 4.1 (Empirical Relaxing Independent Factorization to IOS). Consider an empirical support Z Z where Z (2. and mini-batch size. Forcing IOS implies that Z aligns with its Cartesian product Z Therefore, we minimize sliced/pairwise contrastive approximating Z and Z 1 X X z. K. where 2.m E Bgxd
Building upon Asm. 4.1, which incorporates the invariant and alignment properties, we impose constraint on given mini-batch Z of size b. Specifically, we ensure that the elements Z... are close to their corresponding augmented Z and far from any negative Z while simultaneously preserving the independence of the Support (IoS). This involves minimizing the distance between sets (mmm)/m) and Z.mm X Z. for all appliances m. Owing the discriminating nature of contrastive learning over data, this IoS constraint can be met by focusing contrastive learning Z... and its augmentation Z without involving the Cartesian product x between support latent. Essentially, our approach contrastive effectively addresses the same instance discrimination task as when considering the Cartesian product over all possible combinations. This observation aligns with insights from disentanglement causality perspective? (Wang & Jordan, 2022a). Further explanations are given in 5.3.
DPOSS = =1 "Σ P(zm, 2m) 12
+Σ
(1 D(2,, 2
(4.2)
Latent-Invariant
Latent- Aeient-Aligmentt
where D(.,)) :) is the cosine similarity distance, with 24 and V representing sets of positive and negative latent values. It is shown that both terms contribute equally to the improvement, i.e. 1) = 1.
Thhis study embraces causal of representation learning, contrasting with DIOSC's relaxation of the independence assumption to Independence-of-Supooo (IoS) via contrastive.
5

---

Published as a conference paper at ICLR 2024
Redidual Block Dcooder()) (1)
x-
200
Residual Block Decoder
N/6
Kg
Disnd Duaa
Warghtem
leven
06
Save 01
Diiied Suae City
Alg Ag
Seif Anannton
Self Ansention
We feen
M4
Vila Nersiling
P. Kattal
Mard Foow Vondblis
Aethen Aneer
kg
Figure 3: Performing Attentive /- Variiational Inference entails processing power input x E RCXT through / levels of residual blocks (Res!)), generating key and query feature maps. Parameters [K (((+)) To {
are initially set zero, and (2+1) _ X.
4.2 ATTENTIVE 1-VARIATIONAL AUTO-ENCODERS
To avoid time locality during dimension reduction, and keep long-range capability we refer to an in-depth Temporal Attention with /-Variational layers. NVAE (Vahdat & Kautz, 2020; Apostolopoulou et al., 2021) proposed an in-depth autoencoder for which the latent space z is level-structured and attended locally (Apostolopoulou et et., 2021), I), this shows an effective results for image reconstruction. We employ Temporal Multihead Self-attention (Vaswani et al., 2017) for constructing beliefs of variational layers, allowing effective handling of long context sequences. /- /-Variational Inference is illustrated in Fig. 3, where the construction of Temporal context 7 at level relies on preview contexts i.e denoted TT(+t) query map Q and key map k((t), This approach enables the model attend information from different representation subspaces at various scales. The use of Multihead self-attention aids in capturing diverse relationships and patterns. The detailed mechanism is given in App. D. For the remainder, we assume that D DIOSC uses attentive variational inference We adopt the Gaussian residual parametrization between the prior and the posterior. The prior given by (2 = NuT( (T), 0). The posterior is then given by g(z(1) x, z(<() N Nu(Tz, 0) + ( implemented as convolutions layers. Hence, the term LKI. in Eq. 3.1 adding the residual and then the LKI. is given by:
Φi Φ), aT 0) (T di 6)) where u(.), (-), ōμ(-), and (-) are transformations
Σ 0.5 (22()) g[l)2
L
+ (0) log (60)0)=11)
(x: (x; CxL(xx o,
(4.3)
4.3 SETTING OVERALL OBJECTIVE FUNCTION
Our final objective function combines the regularization (Eq. 3.1) and the VAE loss (Eq. 3.1), which consisting of a reconstruction term Cree. LKL term. We present balancing parameters, denoted as and 8, with X weight the disentanglement regularization and | balancing emphasis between the reconstruction and KI. divergence terms.
C(D, 0.
E6 x [XDiOoS
E
( (xx
(4.4)
b
XEX.EE
4.4 HOW TO EVALCATE DISENTANGLEMRN FOR TIME SERIES?
Evaluating disentanglement in series representation is more challenging than established computer vision metrics. Exist- ing time series methods rely on qualitative observations and predictive performance, while metrics like Mutual Information Gap (MIG) (Li et al., 2023) have limitations with continuous labels. To address this, we adapted RMIG (Carbonneau et al., 2022) for continuous labels and used DCI metrics from (Do & Tran, 2021). Our evaluation, including DCI, RMIG. The 3-VAE and Factor VAE scores, be found in App. D.6.
Metric
Align-mii
Unbizsed
No. Condition
J.VAR PacteVVE
RMIG
SAP
DCI
TDS (Ours)
Table TDS Compared to SOTA met- ric. (Red row the worst, Blue the best).
However, these measures suffer from

---

Published as a conference paper at ICLR 2024
limitations with sequential data and do not provide measures of attribute alignment under ground truth variation. To overcome this, consider cross- cross-correlation between latent variables 2m (latent anchor) and its augmentation zm with respect to ground truth attribute ym. Yet, practical challenges arise with multiple attributes, as this measure can be sensitive to variations within the same attribute. To address this, we introduce the compact Time Disentanglement Score
|| 2mm 2|1
TDS = dim(z) Z))
ΣΣ TAGFATA
(4.5)
Varl [m
where * is an augmentation of Zm and Var[zm] variance of Zm- When a set of latent variables are not axis-aligned, each variable can contain a decent amount of information regarding two or more attributes. wide gap between unaligned variables indicates an entanglement. TDS excels in axis alignment (c.f. Table. 1), is unbiased across hyperparameters.
5 EXPERIMENTS
5.1 ARCHITECTURE SETTINGS AND DATA AUGMENTATION FRAMEWORK
Residual Blocs. We enhance our Residual model by replacing traditional components in residual blocks with Sigmoid Linear Units (SiLU) (Elfwing et al., 2017). SiLU offers advantages such as faster training, robust feature learning, and superior performance compared to weight normalization.
Our framework is given in Fig. 3, set L = 16, and we fix an time window input to 256 steps, for the latent space dimension we fix d₂ 16.
D DTOSC 8)
KL. J
RMSE
Time (s) J
ReLU
0.734
0.734
28800
Squeeze-and- E on Spatial and Temporal: The SE block improves neural networks by selectively empha- sizing important features and suppressing less relevant ones. Extending SE for time series data enhances the capture of significant temporal patterns in sequences.
SiLU
21600
ReLU-SE SiLU+SE
0.721
0.721
32760
0.582
0.582
23040
Table 2: Metrics on UK-Dale (4 lower is
better, 1 higher is better Top-2
Top-1 ).
Pipeline Augmentation for Electric Load Monitoring. Four augmentations were sequentially applied to all contrastive methods' pipeline branches. The parameters from the random search are: 1) Crop and delay: applied with a 0.5 probability and a minimum size of 50% of the initial sequence. 2) Cutout or Masking: time cutout of 5 steps with 0.8 probability. 3) Channel Masks powers: each power (reactive, active, and apparent) is randomly masked out with 0.4 probability. 4) Gaussian noise: random Gaussian noise is added to window activation zm and x with a standard deviation of 0.1. The impact of each increase is detailed the Appendix. D.2.
Pipeline Correlated Sampling Attributes. We evaluate the model's robustness to data correlations by examining various pairs, primarily focusing on linear correlations between two appliances and scenarios where one device correlates with two others. For this, we parameterize these correlations by sampling from a common distribution p(y), 3/2) x exp -| ayaal //////22. where a is a scaling factor, and indicates the strength of the correlation. We extends the (Träuble et 2021) framework beyond time series and adapts it to cover correlations between multiple devices operating in a T'' time window. Scenario examples include: No Correlation (o oo); Pair: 1 (clothes dryer/oven, 0.3): Pair: 2 (washing machine/dishwasher, a 0.4), and Random pair (randomly selected pairs, o 0.8). Additional correlation pairs are detailed in Appendix. D.3.
5.2 EXPERIMENTAL SETUP
Datasets. We conducted experiments on three public datasets: UK-DALE (Kelly & Knottenbelt, 2015), REDD (Kolter & Johnson, 201 1), and REFIT (Murray et al., 2017) providing power measure- ments from multiple homes. We focus on six appliances: Washing Machine, Oven, Dishwasher, Cloth Dryer, Fridge. We performed cross-tests on different dataset scenarios, each with varying sample sizes. Specifically, scenario A involved training REFIT and testing on UK-DALE, 18.3k samples with time window T = 256, and frequencey of 60Hz, the test set consisted of 3.5k samples, scenario B involved training on UK-DALE and testing on REFIT with 13.3k samples, and scenario C involved training on REFIT and testing on REDD with 9.3k samples. The augmentation pipeline is applied for all scenarios. For training and testing under correlation, we the corresponding sampling.
7

---

Published as a conference paper at ICLR 2024
Power Reaccrre Q
Power Apparent
Power Adtive P
Oven
GT.deeeees Dryer
BerTANILM
DI0SC
Apprepaca Powera (P.0.3)
D-VAE
DIOSC
No Cor In 43
1744
0.31
Power WV
3000
30.00
Time
7lmm
Time
Figure 4: Prediction Clothes dryer under in correlated case (top) and uncorrelated case (bottom) over time window of 256min. Moving from left right, the graph illustrates the aggregated power (P.Q.S) alongside the ground-truth Clothes-dryer to be identified under correlation with Oven.
Baseline and Evaluation. We compare DIOSC with downstream task models in energy, Bert4NILM (Yue et al., 2020) as a baseline and S2P (Yang et al., 2021), S2S (Chen et al., 2018a), for those model we keep the same configuration as the original implémentation, We provide also variant 8-TC/Factor/-Va. VAE implemented for time series, compared to D3VAE (Li et al., 2023) and NVAE (Vah- dat & Kautz, 2021), 1), and RNN-VAE (Chung et al., 2015). We compare these model using RMSE, and compute disentanglement metries: RMIG, DCI, TDS. The metries have been evaluated by both, either sampling from the correlated data or from the uncorrelated distribution.
Experimental Platform. We conduct 5 seeds of experiments, reporting the averaged results and standard deviation. Based on the grid search, we found that D DOSC's best performance is obtained by (A 2.3, 3 1.5). The experiments are performed on four NVIDIA A100 GPUs. Hyperparameter settings are available in Appendix D.
5.3 PERFORMANCE AND INFORMATIVITY OF CONTRASTIVE
Finding: DIOSC performs better in Out-of-distributio (under correlated data).
To assess DIOSC 's regularization robustness to correlated appliances, we examine scenarios involving pairs defined in $ 5.1. From Fig. the increased disentanglement through DTOSC gives consistent improvements in all cases, and gets more pronounced in the low data regime, indicating higher sample efficiency, as expected from better disentanglement even the correlated in pairs. Fig. 3 show the regression results as we see even when signals correlated the disentangling is acheived and relative improvements up to +20% in RMSE. We find factorization of supports using DIOSC on the training data is strongly relate to downstream disentanglement even when experiencing strong correlation during training.
Randon Pair
Menkos -g
DCT.
TTS
RMSEE
DC
THs
RMSSK
DC3
ITS
RMSSR
DCE
TBS.
RMSR: 1
825
54.5
14.563.56
72.31=2.4
69:55-3266
JAMAE
73:4:3.0
0.962: 15
48.51
23. 32
72 483 10
0.562-15
53.6 03.31
72 42.433.1 10
D.56c.15
54.73 51
74 2.2922200 04
1.06a.00 (0:
52.99-151 1.0
JUTOVEE
75.0-1005
0341: 13
3 23
11968.13
49.2 -1.13
1554..1
50:57-1117 17
74:30
1,07:: 11
45.5-1141
MRS
0.64
79.5
79.6630 25
0:644-05
50.37:
0.725.00
616441155
VAVE HRS
11 m
0.09 m
34. 89
73 1116101 01
0.094-02
38 14
73 71.9-402 56
04
39/5--0.1
14.554055 59
0.776 04
15.15-100 100
AUTOVAE HRS
61.2-241
0.52--11
24.5 1.51
67.242.1
27.4 11.35
67.51:184
0.53-
68.79:1.37
0.56::84
21:778633
D2090
6333:135
02
19891195
69.3:12
0.41:22
22391179
70.3:032
23.97:1.9
67.32:0991
0.11::11
2260:149
BasNNLM
51.55-2 11.85
66.821.12
13.41=1.35
72.75-0.3
S15
50.5 22:32
078-2276
73:5-111
7000-1125
51.741.79
1.16a.06
79.570084
10
0512.09
n.55..10
50. 10.555055
Face VAB
70.4451.0
1002
49 -1 1055
T1 1159113
02
68 Taall
5534-042
51.775127
10005.01
MRS
0012-005
15
20 06
05
604144
74 Ras
0.5
30
0500:000
AITCVAE HES
05544:136
267:6.8
6675151
25.52 631
7045095
D109C
61422-996
0:51:111 m1
21.35 11.00
31.5:144
0853043
6585:0711
055:::1
240F 1:30
Bes4NILM
15.62155
828
22-43
1344
2417-20
LA3 m
30.15 00.18:11 1152
T3.441150 56
0.26-12
5577224T
163363
167
5631-131
1355
Life0.11
$590:140
JUTOVE
292120089
0965 10
45.11 22005
7042400 75
1052.5
5119:184
505774055
110-0011
5117-1155
VAE
0.99-122
1.06a.00
MRS
604
10
0.00:04
62991156
70918035
0794-00
63.52
63.3-200
VAVE HRS
0.702 01
15 15.65.155
14.1450 22
0142.00
4n5a1140 40
74 95
0115.66
40332-13
36
0776 05
39.35-19
JATCVNE HRS
0984-1
8.54-ml
25 1145
083751131
0.475.01
289:11.8
00875200
38.30-1.44
49345143
042::4
25:29-113
PXDSC
6437
0.
19661.95
0.50::0
31.191.95
7123009
0445:03
2637:1144
677261011
057:001
3412:1.38
Table 3: Average scores DCI, TDS, and RMSE vary from No. Correlation (left) to every appliance correlated with one confounder (right) on uncorrelated test data. Red blue, with bold indicating the best performance per
correlation. (4 lower is better, higher is better Top-2
Top-1
8

---

Published as a conference paper at ICLR 2024
6 ABLATION STUDIES
6.1 DIOSC PRESERVES ITS ROBUSTNESS IN CORRELATED SCENARIOS
Cart 20 +]
PAF [6o3.3)
Pal (o-0.)
Rontom Fair jst mu02)
60
20
Traniing (7)
Tamna (76)
Traiing 1%1
Taaning (5)
Figure 5: Relative RMSE (%) improvement over Bert4NILM for six devices using DIOSC, / VAE, and Factor with the amount of labelled training data as variable parameter.
wiiting Machine
Frigee
0.00
0.0
HONTOS
Dven
Cetthes
Dishwasher
4.0
0.75
6.7
HON-MM1
3
066
0.00
0.25
350
6.55
1.00
002
0.25
850
0.55
3.30
3.00
Figure 6: We find strong correlation between D DICSC and disentanglement metrics vary with Mf (right), linked to classifcation accuracy of each compenents Zm, on ym labeled test data (Left), Darker Blue -> High accuracy.
Finding: DIOSC preserves its robustness in correlated scenarios and achieves comparable perfor- mance to baseline models with less training sample data.
Training with the same I variational inference model with the different regularisation variants results shows that DIOSC outperforms SOTA as shown in Fig. 5, mainly in the uncorrelated cases with only 50% of labelled data, which corresponds to HDF (Roth et al., 2023). With 80% of data, DIOSC scores 14% better than HDF and 61.4% better than the baseline Mert4NILM. In the correlated scenarios (pair 1 and 2). //Factor/TCC TC- VAE shows weaker performance, while DIOSC consistently outperforms HDF and the baseline.
6.2 IN-DEPTH ATTENINTNN I-VAES LEARN AN EFFECTIVE REPRESENTATION.
Finding: DIOSC deepens, representation increases over 20% (40% in TDS), downtasking boosts perfor- mance, and attention mechanisms contribute 10% improvement.
Method
Depth (2)
NRMSE
RMBG 1
TDS 1_
(buseline)
0.928
0.911
0.35
0.920
0.34
0.9)
FRCOrVAE
0.37
DUNAE-DIOSC
In Table 7, we employ I-Variational Inference with the DIOSC regularizer, both with and without self- attention, and explore its application with alternative structures tailored for time series, particularly those residual in D3VAE. Our observations reveal two key findings. Firstly, incorporating DIOSC with another regularization method slightly enhances results, as the alternative regularizer assumes independent factoriza- tion, potentially compromising the relaxing effect. Secondly, DIOSC demonstrates improved per- formance with increasing values of L, and the TDS correlates positively with performance, while RMIG suggests that using DIOSC with attention leads to well- dell-disentangled representations. Notably, the attention mechanism proves efficient by enhancing the overall model performance.
DIOSC
DROSC
DROSC
0.74
0.70
DROSC
0.73
DROSC
0.75
Table 4: Average Normalized RMSF, RMIG,
and TDS Scores for Variants DIOSC w/.w/o At- tention, L Increases. (4 lower is better, higher better Top-2. Top-1), ).
6.3 ROBUSTNESS, DISENTANGLEMENT, AND STRONG GENERALIZZATIO
Finding: DIOSC demonstrates robust disentanglement performance across varying dimensions, while FactorVAl VAE exhibits degradation as dimensionality increases M t.

---

Published as a conference paper at ICLR 2024
In Fig. 6 (right), we we report the disentanglement performance of DIOSC and FactorVAE the dkld dataset as M is increased. FactorVAE (Higgins et al., 2016) is the closest TC-based method it uses a single monolithic Discriminator and the density-ratio trick to explicitly approximate TC. Computing TC(2) is challenging to compute as M increases. The results for M = 10 (scalable R x3) are included for comparison. The average disentanglement scores for D DISC M 7 and M 10 are lower and very close, indicating that its performance is robust in M. This is not the case for HDF Factor/3-VAE. performs worse on all metrics when increases. Interestingly, HDF M seems to recover its performance Factor/B-VV VAE are not robust to changes in M. The optimal M for HDF and TC/3-VA VAE, shown in Fig. 6 (left), indicates promising accuracy for HDF, despite being no better than DIOSC.
most metrics. Despite this, the difference suggests that HDF and
7 DISCUSSION AND CONCLUSION
To address the limitation of assuming independence in existing time series disentanglement methods, which may not accurately reflect real-world correlated data, our approach focuses on recovering correlated data. By relaxing the independence factorization assumption to sodependeece--o-ssuppooo via contrastive learning, our method achieves identification and disentanglement, enabling the model to encode attribute variability in the latent space. DIOSC, combines contrastive regularization and I- Variational Autoencoder for time series, we demonstrate that promoting pairwise factorized support is sufficient for disentangling time series. Consistent with our theoretical findings, DIOSC outperforms baseline methods by more than +61.4% in downstream tasks of NILM across datasets with various correlation shifts, highlighting the benefits of enhanced disentanglement for distr--iiittiuutioo generalization in representation learning. Future work should explore support factorization for time series with causal notions rather than independence.
Limitations of Theory. Although posit that our theoretical assumptions encapsulate crucial aspects of time series representation learning under strong correlation, they may be subject to varying degrees of violation in practical scenarios characterized by correlations. For instance, the relaxation assumption to IOS (Asm. 4. regarding the batch is influenced by its size, which we consider as hyperparameter during training.
BROADER IMPACT
Our proposed method enables effective representation learning for time series data related to energy load, offering broad applicability across various downstream tasks. In this context, we showcase its efficacy in scenarios characterized by strong correlations. The scalability of our approach, particularly when applied to scaled versions featuring a large number of appliances, facilitates its generalization across domains, establishing foundational models for energy disaggregation. The potential societal benefits, such as enabling household consumption determination, are particularly notable within the context of smart grid systems. As evidenced by its successful implementation in smart grid management, our method readily adapts to efficiently detect appliance consumption patterns. This capability not only aids in energy management but also provides users with valuable feedback regarding optimal utilization during off-peak hours, thereby optimizing energy consumption and consequently reducing carbon footprint. Such contributions underscore the significant societal and environmental ad advantages afforded by Al-driven models.
ACKNOWLEDGEMENTS
This work was granted access to the HPC resources of IDRIS under the allocation ADO1 1014921 made by GENCI (Grand Equipement National de Calcul Intensif). Part of this work was funded by the Tntaienergies Individual Fellowship through One Tech. Special appreciation is given to Thierry Luci, head of the Applied Scientist AI Team at One Tech, for his leadership and support, and to the team for their active participation in insightful discussions.
REFERENCES
Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional Causal Representation Learnin. In Proceedings of the 40th International Conference on Machine Learning, pp. 372-407.
10

---

Published as a conference paper at ICLR 2024
PMLR, July 2023. URL WRLLttt:///ppceeeiiiii http:s///poooooeiiggg..rr///s/// html.
ISSN: 2640-3498.
Ifigeneia Apostolopoulou, lan Char, Elan Rosenfeld, and Artur Dubrawski. Deep attentive variational inference. In International Conference on Learning Representations, 2021.
Robert B Ash. Information theory. Courier Corporation, 2012.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation Learning: A Review and New Perspectives, April 2014. URL http:: /ttp://aaiiiv... wwtp:///g/bbbb/eee 5538.
[es].
Giovanni Bucci, Edoardo Fiorucci, Simone Mari, and Andrea Fioravanti. New Convolutional
Neural Betwork-Based System for NILM Applications. IEEE Transactions on Instrumentation and Measurement, 2021. doi: 10.1 109/TIM.2020.003...
Marc-André Carbonneau, Julian Zaidi, Jonathan Boilard, and Ghyslain Gagnon, Measuring Disentan- glement: A Review of Metrics, May 2022. URL http:://arrdi..orr/abb//// arXiv:2012.09276 [es].
09276.
Kunjin Chen, Qin Wang, Ziyu He, Kunlong Chen, Jun Hu, and Jinliang He. Convolutional sequence to sequence non-intrusive load monitoring. the Journal of Engineering, 2018(17)1180018661 2018a. Publisher: Wiley Online Library.
Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating Sources of Disentanglement in Variational Autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc., 2018b.
Ricky Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating Sources of Disentanglement in Variational Autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- and R. Garnett (eds.), Advances Neural Information Processing Systems,
man, N. volume 31. Curran Associates, Inc., 2018c.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A Recurrent Latent Variable Model for Sequential Data. In Ad- vances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL ottps::/prroeeedigg...eeoo wwwpeurrips.cppppp....e. wwwp::ee22aattttttttteeaaa html.
cctpaper_fiieeeeeeeeeeeeeeeeeeee wileerr_fileeeeeeeeeeeeeeeeessee /tales/papee/2015/
Pierre Comon. Independent component analysis, concept? Signal processing, 36(3):287-314, 1994.
Kien Do and Truyen Tran. Theory and Evaluation Metrics for Learning Disentangled Representations, 09961. arXivv1908.099) 19991999889999 [cs, stat].
March 2021. URL http: aawivv. http//axw.....gaab/////a//
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning, November 2017. URL
borg/bb/// /102. 03118. arXiv:1702.03111 [cs].
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick. Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. International conference on learning representations, 2016.
Aapo Hyvärinen. Independent component analysis: recent advances. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, (71 10534, 2013.
Christian Jutten and Juha Karhunen. Advances in blind source separation (bss) and independent component analysis (ica) for nonlinear mixtures. International journal of neural systems, 14(05): 267-292, 2004.
Christian Jutten, Massoud ZZaeh, and Juha Karhunen. Nonlinear mixtures. Handbook of Blind Source Separation, pp. 549-592, Elsevier, 2010.

---

Published as a conference paper at ICLR 2024
Jack Kelly and William Knottenbelt. The UK-DALE dataset, domestic appliance-level electricity demand and whole-house demand from five UK homes. Scientific data, 2, 2015. Publisher: Nature Publishing Group.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarn,, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised Contrastive Learning, March 2021. URL http://arxii org/ abs /2004. 11362. arXiv:2004. 11362 [es, stat].
Hyunjik Kim and Andriy Mnih. Disentangling by Factorising, July 2019. URL htt.t:http/iiiiiee org/ arg//bs// /1011.00.00 059835988. arXiv:1802.05983 [cs, stat].
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: arXiv:1412.6940 1412.69800 2014.
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprini
20t::2000..19330 2020.
J Zico Kolter and Matthew Johnson. REDD: A public data set for energy disaggregation research. In Workshop on data mining applications sustainability (SIGKDD), San Diego, CA, volume 25, 2011. Issue: Citeseer.
Yan Li, Xinjiang Lu, Yaqing Wang, and Dejing Dou. Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement, January 2023, URL http: /ttp:://axivv.oo org/ / 2301. 03008.008.0 arXiv:2301 ..0028 [cs].
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. In international conference on machine learning, pp. 4114-4124. PMLR, 2019.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling, November 2019. URL http: httpt::///xxx..oor/eeeooooo org/ 1902. 0210011100 02. 1rXiv:1902.02102 [cs, stat].
David Murray, Lina Stankovic, and Vladimir Stankovic. An electrical load measurements dataset of united kingdom households from a two-year longitudinal study. Scientific data, 4(1):1-12, 2017.
Christoforos Nalmpantis and Dimitris Vrakas. On time series representations for multi-label NILM Neural Computing and Applications, 32(23), 2020. ISSN 0941-0643. doi: 000770000521-00-44991--- URL https: htap::///inke www.iinkkspiige...... wwm/epdf/10..10000000000
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Learning disentangled representa- tion by exploiting pretrained generative models: A contrastive learning view. arXiv preprint 102.:33002.10433. 2021.
Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disen- tanglement of Correlated Factors via Hausdorff Factorized Support, February 2023. URL http:/ //tp://aaaa /awxiv.iivvvv.////////a///aaaav. htt//bbs/2211 07347. arXiv:2210.07347 [cs, stat].
Ramalingam Shanmugam. Elements of causal inference: foundations and learning algo- rithms. Journal of Statistical Computation and Simulation, 88(16):2488 3244. November 2018. ISSN 0094-9655, 1563-5163, doi: 10.108000099995552211..888 201. 197. URL https: // tand i com doi htt///ulll//../ 101100000000000 2018 1505197.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International Conference Machine Learning, pp. 10401-10412, PMLR, 2021.
12

---

Published as a conference paper at ICLR 2024
Arash Vahdat and Jan Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Process- ing Systems, 2020.
Arash Vahdat and Jan Kautz. NVAE: A Deep Hierarchical Variational Autoencoder, January 2021.
URL http::ttt//. /ttp:://rxxive /twpvv.iivvv.///..//.../aaaaaav. org/ abs/2007. 03898. arXiv:2007.03899 [cs, stat].
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: httpps:/pproceeednggs http://ece222566eeeee.oooodaaaaa
wtwp:/ceeeeninnnnnnnnnerreeeeeee neurips wwwpurupssccpppprrrree. wwtpaper_fileeeeeeeeeeeeeeesssee pttess/pppee//11.. htt.://ngg...rrr.....ceoeeeeeeee
Yixin Wang and Michael I. Jordan. Desiderata for Representation Learning: A Causal Perspective,
February 2022a. URL http:::/aarzvv...ggaaa////////// 03795 arXiv:2109.03795 [cs, stat].
Yixin Wang and Michael I. Jordan. Desiderata for Representation Learning: A Causal Perspective, February 2022b. URL http:://arxlw..orgaab//////
03795. arXiv:2109.03795 [es,
stat].
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting, arXiv preprint arXiv:2202.01575, 2022.
Mingzhi Yang, Xinchun Li, and Yue Liu. Sequence Point Learning Based Attention Neural Network for Nonintrusive Load Decomposition. Electronics, 2021.
Weiran Yao, Guangyi Chen, and Kun Zhang. Temporally disentangled representation learning.
Advances in Neural Information Processing Systems, 15:26492-265033 2022.
Zhenrui Yue, Camilo Requena Witzig, Daniel Jorde, and Hans-Arno Jacobsen. BERTANILM: A
Bidirectional Transformer Model for Non-Intrusive Load Monitoring. In Proceedings of the Sth International Workshop on non-Intrusive Load Monitoring, NILM'20, pp. 89-93, New York, NY, USA, November 2020. Association for Computing Machinery. ISBN 978-1-4503-8191-8. doi: 10. 1 15555322777134229999 URL https: /d1 acmm org /ttp://oo/o..111144/2277070 3429390.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow Twins: Self-Supervised Learning via Redundancy Reduction, June 2021. URI. http:://axxi..ee/ 03230. arXiv:2103.03250 [cs, q-bio].
03.
Wentian Zhao, Yanyun Gao, Tingxiang Ji, Xili Wan, Feng Ye, and Guangwei Bai. Deep temporal convolutional networks for short-term traffic flow forecasting. leee Access, 7:114496-114509 2019.
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive Learning Inverts the Data Generating Process, April 2022. URL http:://axti erq/abs// /ttss/22122 2202. 08850. urXiv:2102.0888 [cs].
13

---

Published as a conference paper at ICLR 2024
Supplementary Material
To facilitate a comprehensive examination of our paper, we present additional results and furnish complete proofs for the assumptions articulated in the main manuscript. This supplementary material is meticulously organized as follows:
Table of Contents
A Proofs
B Algorithm The dual approaches for assessing Identifiability and Disentanglement
17
C I-Variational Autoencoder Procedure (Inference and Generative)
18
C.I Encodeur (Infrence Model qa)
18
C.2 Decodeur (Generative Model po).
19
Implementation Details and Extended Ablation Studies
20
D.I Data sets
20
D.2 Pipeline Augmentation for Electric Load Monitoring
20
D.3 Pipeline Correlated samples.
21
D.4 Further Experimental Details
22
D.5 Impact of 1/. A, 3 and Training stability
23
D.6 Implementation Metrics and study case
23
D.7 Method Ablation
24
D.8 PSEUDOCODE DIOSC Cosine Similarity
25
E Connection between DIOSC, Causality-Reprrsentatton and the Information Bottle- neck Principle
27
E.1 Disentanglement Based ofdependencceoo--Spppoor via Contrastive
27
14


Reviews:
[{"review_id": "HPGIuqa3tS", "reviewer": "Reviewer_a2rd", "summary": "The paper seems to be an application of disentangled representation learning for home appliance electricity usage. The authors propose to combine contrastive and variational losses. Unfortunately, the paper falls somewhere between methodological novelty and application, making it difficult to understand where the main contributions of the paper will lie. In general, I found the paper very hard to read.", "questions": "- I strongly suggest the authors make their main contributions clear at the end of the introduction. \n\n- There is no related work section.", "limitations": "", "rating": 1, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "The main strength of the paper is its approach to tackling the important problem of disentangled representation learning, which may contribute to reducing carbon footprint.", "weaknesses": "The paper lacks proper organization and has a tendency to include some unproven (or wrong) claims. For instance, in the introduction, the authors mention that \"Disentanglement via VAE **can be achieved** by a regularization term of the Kullback-Leibler divergence[],\" which is not necessarily true without certain strong assumptions and underlying conditions. The Beta-VAE paper has some qualitative evidence showing how the images are more disentangled compared to VAE. The authors also claim, \"Rather than training separate auto-encoders for individual appliances,\" which requires empirical validation with proper citations.\n\nIn addition to these issues, the notations used are very confusing and are not defined before they are referenced. For example, in the proposed method section, the notation $z_m^+$ is used without description it. It is also unclear how it differs from $\\bf{z}$ or $z$.\n\nThe main goal of this paper remains unclear to me. For example, the authors mentioned, \"The primary goals of this work are twofold: to effectively address the NILM problem and to obtain a disentangled representation of input data.\" However, it is unclear what the NILM problem is, what the nature of the input data is, and how the authors plan to achieve a disentangled representation that distinguishes itself from previous works. One issue might be that the problem statement and preliminaries are somewhat intertwined.\n\nThe color meaning used in the tables of result section is not clear. Even it is not clear how TDS (as a metric) has been compared with VAE and Beta-VAE in Table 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jnzJZdPgw3", "reviewer": "Reviewer_jyEy", "summary": "This study explores disentangled representations for time series data, with a primary emphasis on achieving representation generalization across diverse, interrelated scenarios. They focused on a specific application of electric load monitoring application where computing different household appliances contribution in a total load is the task. \nIn the context of Variational Autoencoders (VAE), this study draws inspiration from Roth et al., 2023, who addressed correlated attributes in an image processing context by replacing the independence constraint over attributes in the latent space (by a regularization term of the Kullback-Leibler divergence between the posterior of the latent attributes and a standard Guassian distribution), with the Hausdorff Factorized Support (HFS) assumption. The authors have adapted this idea for time series data and introduced the use of cosine similarity instead. Consequently, this approach no longer necessitates independent latent activations for different appliances. \nThe main idea is to address appliance correlations with weakly supervised contrastive disentanglement, promoting similarity for the same appliances and dissimilarity for absent appliances in latent representations. This is achieved through a loss function composed of two terms, one for alignment based on correlation and another to minimize redundancy between latent variables.\nIn addition, the authors proposed l-variational inference layers with self-attention mechanism to address temporal dependencies. Additionally they propose a metric of  Time Disentangling Score (TDS) to evaluate the  disentanglement performance in time series data.", "questions": "-Using cosine similarity instead of HFS needs more elaboration.\n\n-Section 3.2 would benefit from a dedicated illustration demonstrating ATTENTIVE l-VARIATIONAL AUTO-ENCODERS, along with the corresponding notations used in the text.\n\n-The authors have effectively presented the formulation for the usecase; however, in the experimental results, which I find somewhat abstract, there's a lack of a specific example illustrating how X and Y values for a time window are displayed, along with different rows of Y, etc.\n\n-In section 4.1, should be included how exactly augmentation is performed and how many, it is very abstract now. \n\n-In Section 2, specifically concerning contrastive learning, the evaluation of appliance dissimilarity in \"x\" and \"x-\" is not explicitly clarified. Is labeling used for this purpose? What if the appliances are not the exact same but should exhibit similar behavior? How are such cases addressed? Additionally, the preparation of negative and positive samples is not detailed. Have you considered ensuring that there are no common or similar appliances in these two sets, and if so, how was this determined? Providing further explanation or an illustration could enhance the clarity of data preparation, which is a crucial aspect of the methodology.\n-How many training examples did you use for linking?\n“We link the learned latent representation to ground-truth attributes using a limited number of pair labels”\n-After equation 2, In this test, the latent variable is represented as \"z,\" which is defined as a matrix of dimensions (M + K) × dz, where \"K\" and \"dz\" should be introduced and define.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents several intriguing novelties.\n1) Using pairwise similarity rather than independence assumption in VAE, to consider the correlated representations. \n2) l-variational inference layers with self-attention mechanism\n3) A metric of  Time Disentangling Score (TDS) to evaluate the  disentanglement performance in time series data\n\n The authors have tackled a captivating problem, successfully adapting image processing techniques to the more complex domain of time series data.", "weaknesses": "The paper needs some modifications to make it easier to read (some suggestions given in the Questions).\nThe experimental results are very abstract (some suggestions in Questions part)\nThe application worth more explanation, the description lacks either an illustration or it is abstract.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6yu8w5ZcGV", "reviewer": "Reviewer_cDdv", "summary": "The paper discusses the importance of learning disentangled representations for Time Series data, specifically in the context of home appliance electricity usage. The goal is to enable users to better understand and optimize their energy consumption, thereby reducing their carbon footprint. The authors frame the problem as one of disentangling the role of each appliance (e.g., dishwashers, fridges) in total electricity usage.\n\nUnlike existing methods that assume attributes (appliances in this case) operate independently, this work acknowledges that real-world time series data often show correlations between attributes. For instance, dishwashers and washing machines might be more likely to operate simultaneously during the winter season.\n\nTo address these challenges, the authors propose a method called DisCo (Disentangling via Contrastive), which employs weakly supervised contrastive disentanglement. This approach allows the model to generalize its representations across various correlated scenarios and even to new households. The method incorporates novel VAE layers equipped with self-attention mechanisms to effectively tackle temporal dependencies in the data.\n\nTo evaluate the quality of disentanglement, the authors introduce a new metric called TDS (Time Disentangling Score). The TDS proves to be a reliable measure for gauging the effectiveness of time series representation disentanglement, thereby making it a valuable tool for evaluation in this domain.\n\nOverall, the paper argues that disentangled representations, particularly those achieved using their DisCo method, can enhance the performance in tasks like reconstructing individual appliance electricity consumption.", "questions": "Please discuss how sensitive the method is to hyperparameter selection.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The method is very sound with mathmaticaly correct derivations. \nThe addressed problem of disentagling latent factors in VAE type of models is very important.\nSpecifically, he paper addresses the unrealistic assumption of independence among generative attributes that is often present in traditional untangling methods. In contrast to these traditional approaches, DisCo focuses on recovering correlated data by encoding a wide range of possible combinations of generative attributes in the learned latent space.\n\nThe authors assert that simply encouraging pairwise factorized support in the latent space is sufficient for achieving effective disentanglement, even when data attributes are correlated. This is an important finding. \n\nIn terms of performance, DisCo is shown to be competitive with downstream task methods, exhibiting significant improvements of over +60% across a variety of benchmarks in three different datasets undergoing correlation shifts (Finding 5.1). This is a strong aspect of the work.\n\nAdditionally, the capability of DisCo to adapt across correlation shifts leads to better out-of-distribution generalization, especially when these shifts are more severe. This fulfills one of the key promises of learning disentangled representations, which is to improve the model's robustness and generalizability.", "weaknesses": "I enjoyed the paper and did not find important weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)