Contextually Supervised Source Separation with Application to Energy
Disaggregation
Matt Wytock and J. Zico Kolter
Machine Learning Department
Carnegie Mellon University
Pittsburgh, Pennsylvania 15213

Abstract
We propose a new framework for single-channel source
separation that lies between the fully supervised and unsupervised setting. Instead of supervision, we provide
input features for each source signal and use convex
methods to estimate the correlations between these features and the unobserved signal decomposition. Contextually supervised source separation is a natural fit for
domains with large amounts of data but no explicit supervision; our motivating application is energy disaggregation of hourly smart meter data (the separation ofwhole-home power signals into different energy uses).
Here contextual supervision allows us to provide itemized energy usage for thousands homes, a task previously impossible due to the need for specialized datacollection hardware. On smaller datasets which include
labels, we demonstrate that contextual supervision improves significantly over a reasonable baseline and existing unsupervised methods for source separation. Finally, we analyze the case of `2 loss theoretically andshow that recovery of the signal components depends
only on cross-correlation between features for different signals, not on correlations between features for thesame signal.

Introduction
We consider the single-channel source separation problem,in which we wish to separate a single aggregate signal intoa mixture of unobserved component signals. Traditionally,
this problem has been approached in two ways: the supervised setting (Kolter, Batra, and Ng 2010; Roweis 2001;
Schmidt and Olsson 2006),where we have access to training
data with the true signal separations and the unsupervised(or “blind”) setting (Blumensath and Davies 2005; Davies
and James 2007; Lewicki and Sejnowski 2000; Schmidt and
Mørup 2006), where we have only the aggregate signal.
However, both settings have potential drawbacks: for many
problems, including energy disaggregation—which looks toseparate individual energy uses from a whole-home powersignal (Hart 1992)—it can be difficult to obtain training datawith the true separated signals needed for the supervised setting; in contrast, the unsupervised setting is an ill-defined
Copyright c 2014, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.

problem with arbitrarily many solutions, and thus algorithms
are highly task-dependent.
In this work, we propose an alternative approach that lies
between these two extremes: with contextual supervision,
along with the input signal to be separated, we provide contextual features correlated with the unobserved component
signals. In practice, we find that this is often much easierthan providing a fully supervised training set, yet it also allows for a well-defined problem, unlike the unsupervised
setting. The approach is a natural fit for energy disaggregation, since we have strong correlations between energy usage and easily observed context—air conditioning spikes inhot summer months, lighting increases when there is a lackof sunlight, etc. We formulate our model directly as an optimization problem in which we jointly estimate these correlations along with the most likely source separation. Theoretically, we show that when the contextual features are relatively uncorrelated between different groups, we can recover
the correct separation with high probability.
Contextually supervised source separation provides acompelling framework for energy disaggregation from“smart meters”, communication-enabled power meters thatare currently installed in more than 32 million homes (Institute for Electric Efficiency 2012), but are limited to recording whole home energy usage at low frequencies (every 15
minutes or hour). This is an important task since many studies have shown that consumers naturally adopt energy conserving behaviors when presented with a breakdown of theirenergy usage (Darby 2006; Neenan and Robinson 2009;
Ehrhardt-Martinez, Donnelly, and Laitner 2010). There areseveral possible ways that such a breakdown could beachieved; for example, by installing current sensors on eachdevice we could monitor electricity use directly. But, as custom hardware installation is relatively expensive (and requires initiative from homeowners), algorithmic approaches
that allow disaggregation of energy data already being collected are appealing. However, existing energy disaggregation approaches virtually all use high-frequency sampling(e.g. per second or faster) which still requires the installationof custom monitoring hardware for data collection. In contrast, by enabling disaggregation of readily available lowresolution smart meter data, we can immediately realize thebenefits of observing itemized energy use without the needfor additional monitoring hardware.

The main contributions of this paper are 1) the proposedcontextually supervised setting and the optimization formulation; 2) the application of this approach to the problem
of energy disaggregation from low-resolution smart meterdata without explicit supervision; and 3) theoretical analysis
showing that accurate separation only requires linear independence between features for different signals.

Related work

As mentioned above, work in single-channel source separation has been separated along the lines of supervised andunsupervised algorithms. A common strategy is to separate the observed aggregate signal into a linear combinationof several bases, where different bases correspond to different components of the signal; algorithms such as Probabilistic Latent Component Analysis (PLCA) (Smaragdis,
Raj, and Shashanka 2006), sparse coding (Olshausen and
Field 1997), and factorial hidden Markov models (FHMMs)(Ghahramani and Jordan 1997) all fall within this category,with the differences concerning 1) how bases are represented
and assigned to different signal components and 2) how thealgorithm infers the activation of the different bases given
the aggregate signal. For example, PLCA typically uses
pre-defined basis functions (commonly Fourier or Wavelet
bases), with a probabilistic model for how sources generate different bases; sparse coding learns bases tuned to data
while encouraging sparse activations; and FHMMs use hidden Markov models to represent each source. In the supervised setting, one typically uses the individual signals tolearn parameters for each set of bases (e.g., PLCA will learn
which bases are typical for each signal), whereas unsupervised methods learn through an EM-like procedure or by
maximizing some separation criteria for the learned bases.
The method we propose here is conceptually similar, but
the nature of these bases is rather different: instead of fixedbases with changing activations, we require features that effectively generate time-varying bases and learn activations
that are constant over time.
Orthogonal to this research, there has also been a great
deal of work in multi-channel blind source separation problems, where we observe multiple mixings of the same
sources (typically, as many mixings as there are signals)
rather than in isolation. These methods can exploit significantly more structure and algorithms like Independent Component Analysis (Comon 1994; Bell and Sejnowski 1995)
can separate signals with virtually no supervised information. However, when applied to the single-channel problem(when this is even possible), they typically perform substantially worse than methods which exploit structure in the
problem, such as those described above.
From the applied point of view, algorithms for energy disaggregation have received growing interest in recent years(Kolter, Batra, and Ng 2010; Kim et al. 2011; Ziefman andRoth 2011; Kolter and Jaakkola 2012; Parson et al. 2012)
but these approaches all use either high-frequency samplingof the whole-building power signal or known (supervised)breakdowns whereas the focus of this work is disaggregating low-resolution smart data without the aid of explicit supervision, as discussed in the previous section.

Contextually supervised source separation
We begin by formulating the optimization problem for contextual source separation. Formally, we assume there is some
unknown matrix of k component signals
"
#
|
|
|
T ⇥k
Y 2R
= y1 y 2 · · · y k
(1)
|
|
|
Pk
from which we observe the sum ȳ = i=1 yi . For example,
in our disaggregation setting, yi 2 RT could denote a power
trace (with T total readings) for a single type of appliance,such as the air conditioning, lighting, or electronics, and ȳ
denotes the sum of all these power signals, which we observefrom a home’s power meter.
In our proposed model, we represent each individual component signal yi as a linear function of some componentspecific bases Xi 2 RT ⇥niyi ⇡ X i ✓ i(2)where ✓i 2 R are the signal’s coefficients. The formal
objective of our algorithm is: given the aggregate signal ȳand the component features Xi , i = 1, . . . , k, estimate boththe parameters ✓i and the unknown source components yi .
We cast this as an optimization problem
niminimizeY,✓subject to

k
X
i=1

k
X

{`i (yi , Xi ✓i ) + gi (yi ) + hi (✓i )}(3)

yi = ȳi=1where `i : RT ⇥ RT ! R is a loss function penalizing
differences between the ith reconstructed signal and its linear representation; gi is a regularization term encoding the“likely” form of the signal yi , independent of the features;and hi is a regularization penalty on ✓i . Choosing `i , gi andhi to be convex functions results in a convex optimization
problem.
A natural choice of loss function `i is a norm penalizing
the difference between the reconstructed signal and its features kyi Xi ✓i k, but since our formulation enables loss
functions that depend simultaneously on all T values of thesignal, we allow for more complex choices as well. For
example in the energy disaggregation problem, air conditioning is correlated with high temperature but does not respond to outside temperature changes instantaneously; thermal mass and the varying occupancy in buildings often results in air conditioning usage that correlates with high temperature over some window (for instance, if no one is in aroom during a period of high temperature, we may not use
electricity then, but need to “make up” for this later whensomeone does enter the room). In this case, the loss function`i (yi , Xi ✓i ) = k(yiXi ✓i )(I ⌦ 1T )k22

(4)

which penalizes the aggregate difference of yi and Xi ✓i overa sliding window, can be used to capture such dynamics. In
many settings, it may also make sense to use `1 or `1 rather
than `2 loss, depending on the nature of the source signal.

Likewise, since the objective term gi depends on all T
values of yi , we can use it to encode the likely dynamics ofthe source signal independent of Xi ✓i . For air conditioning
and other single appliance types, we expect sharp transitions
between on/off states which we can encode by penalizing
the `1 norm of Dyi where D is the linear difference operator subtracting (yi )t 1 (yi )t . For other types of energy
consumption, for example groups of many electronic appliances, we expect the signal to have smoother dynamics andthus `2 loss is more appropriate. Finally, we also include
hi for statistical regularization purposes—but for problems
where Tni , such as the ones we consider in energy disaggregation, the choice of hi is less important.

Theoretical analysis
Next, we consider the ability of our model to recover thetrue source signals as T grows while k and ni remain fixed.
For the purposes of this section only, we restrict our attention to the choice of `2 loss, no gi or hi terms, and Gaussian
noise (the extension to the sub-Gaussian case is straightforward). We show that under this specialization of the model,
the optimization problem recovers the underlying signals at
a rate dependent on the linear independence between blocks
of input features Xi . In practice, the choice of `i , gi and hi
is problem-specific and as we see in our experiments, thischoice has a significant impact on performance. As we show
in this section, for the special case of `2 loss with no regularization, the estimate of the ✓i reduces to the least-squares
estimate which simplifies theoretical analysis significantly.
However, while this provides intuition on the essential behavior of the model in the large T regime, we note that thisis a special case of the framework and that in general moresophisticated loss functions will result in more complex algorithms with better performance.
Formally, for this section we assume the source signals
have Gaussian noise
yi = Xi ✓i? + wi

(5)

for some ✓i? 2 Rni and wi ⇠ N (0, i2 I). Under the choiceof `2 loss, our optimization problem becomesminimize
Y,✓

k
X
1

2
i=1

k
X

subject tokyi

Xi ✓i k22

(6)

yi = ȳi=1

and by taking gradients we have the optimality conditionsyi
XiT yiXi ✓i +

= 0 for i = 1 . . . k

XiT Xi ✓i = 0

for i = 1 . . . k(7)

where 2 RT is the vector of Lagrange multipliers corresponding to the constraints. These constraints imply that
XiT = 0 and summing over k we haveȳX✓ + k = 0
XT

=0

(8)

where ✓ 2 Rn is a concatenation of all the ✓iP’s, X 2 RT ⇥nkis a concatenation of all the Xi ’s and n = i=1 ni is thetotal number of features. This system of equations has thesolution✓ˆ = (X T X) 1 X T ȳ
(9)which is simply the least-squares solution found by regressing ȳ on all of the features X.
Since each yi has it’s own noise term, we can never expect to recover yi exactly, but we can recover the true ✓?
with analysis that is the same as for standard linear regression. However, in the context of source separation, we areinterested in the recovery of the “noiseless” yi , Xi ✓i? , as thiscorresponds to the recovery of the underlying signals. Thiscan be a significant advantage as it is often much easier to recover the product of Xi ✓i? than the individual ✓i? parameters
themselves. In particular, as we show in our analysis, accurate signal recovery depends only on the degree to whichfeatures in different groups are correlated, not on the correlations contained within a particular group. Concretely, ouranalysis considers how the root mean squared errorr2
1
RMSE(Xi ✓ˆi ) =Xi ✓ˆi Xi ✓i?
(10)T2
vanishes for large T .
Theorem 1. Given data generated by the model (5), andestimating ✓ˆ via (9), we have thathi
E kXi ✓ˆi Xi ✓? k22 = 2 tr XiT Xi (X T X)ii 1  2 ni ⇢i
(11)
Pk
1
2
T
T
where 2 =
and
⇢
=
(X
X
(X
X)
i
max
i
i
ii ).
i=1 i
Furthermore, for  0.1, with probability greater than 1r
4 2 ni ⇢i log(1/ )ˆ.
(12)
RMSE(Xi ✓i ) TA key quantity in this theorem is the matrix
XiT Xi (X T X)ii 1 2 Rni ⇥ni ; (X T X)ii 1 denotes thei, i block of the full inverse (X T X) 1 (i.e., first inverting
the joint covariance matrix of all the features, and thentaking the i, i block), and this term provides a measure of
the linear independence between features corresponding
to different signals. To see this, note that if features acrossdifferent signals are orthogonal, X T X is block diagonal,
and thus XiT Xi (X T X)ii 1 = XiT Xi (XiT Xi ) 1 = I, so⇢i = 1. Alternatively, if two features provided for different
signals are highly correlated, entries of (X T X)ii 1 will havelarge magnitude that is not canceled by XiT Xi and ⇢i will
be large. This formalizes an intuitive notion for contextually
supervised source separation: for recovering the underlying
signals, it does not matter if two features for the same signal
are highly correlated (this contrasts to the case of recovering✓? itself which depends on all correlations), but two correlated signals for different features make estimation difficult;intuitively, if two very similar features are provided for two
different source signals, attribution becomes difficult. A
particularly useful property of these bounds is that all terms
can be computed using just Xi , so we can estimate recovery
rates when choosing our design matrix.

x1

w1

w2

2

1.5

1

1

0

0.5

−1

0

−2

ActualEstimate

5
4
3
2
1
0

Experimental results

y1

2
1
0
3
2
y2

In this section we evaluate contextual supervision for energy disaggregation on one synthetic dataset and two real
datasets. On synthetic data we demonstrate that contextual
supervision significantly outperforms existing methods (e.g.
nonnegative sparse coding) and that by tailoring the loss
functions to the expected form of the component signals(as is a feature of our optimization framework), we cansignificantly increase performance. On real data, we begin with a dataset from Pecan Street, Inc. (http://www.
pecanstreet.org/) that is relatively small (less than
100 homes), but comes with labeled data allowing us tovalidate our unsupervised algorithm quantitatively. Here weshow that our unsupervised model does remarkably well indisaggregating sources of energy consumption and improves
significantly over a reasonable baseline. Finally, we apply
the same methodology to disaggregate large-scale smart meter data from Pacific Gas and Electric (PG&E) consisting ofover 4000 homes and compare the results of our contextually
supervised model to aggregate statistics from survey data.
In all experiments, we tune the model using hyperparameters that weigh the terms in the optimization objective; inthe case of energy disaggregation, the model including hyperparameters ↵ and is shown in Table 3. We set these
hyperparameters using a priori knowledge about the relative
frequency of each signal over the entire dataset. For energy
disaggregation, it is reasonable to assume that this knowledge is available either from survey data (e.g. (U.S. Energy
Information Administration 2009)), or from a small numberof homes with more fine-grained monitoring, as is the casefor the Pecan Street dataset. In both cases, we use the same
hyperparameters for all homes in the dataset.
Disaggregation of synthetic data. The first set of experiments considers a synthetic generation process that mimics
signals that we encounter in energy disaggregation. The process described visually in Figure 1 (top) begins with two signals, the first is smoothly varying over time while the otheris a repeating step function
X1 (t) = sin(2⇡t/⌧1 ) + 1, X2 (t) = I(t mod ⌧2 < ⌧2 /2)
(14)

x2

2

y1 + y2

The proof of Theorem 1 proceeds in two steps. First, using rules for linear transformations of Gaussian random variables, we show that the quantity Xi (✓ˆi ✓i? ) is also (zeromean) Gaussian, which immediately leads to (11). Second,
we derive a tail bound on the probability that Xi (✓ˆi ✓i? ) exceeds some threshold, which leads to the sample complexity
bound (12); because this quantity has a singular covariancematrix, this requires a slightly specialized probability bound,given by the following lemma
Lemma 1. Suppose x 2 Rp ⇠ N (0, ⌃) with rank(⌃) = n.
Then✓ ◆n/2⇢t
1
2
P kxk2 t 
exp
(t/
n)
(13)
n
2
where is the largest eigenvalue of ⌃.
The proof, along with the complete proof of Theorem 1,
is deferred to Appendix A.

10

Figure 1: Synthetic data generation process starting with two
underlying signals (top left), corrupted by different noisemodels (top right), summed to give the observed input (row2) and disaggregated (rows 3 and 4).
where I(·) is the indicator function and ⌧1 , ⌧2 are the periodof each signal. We also use two different noise models: forthe smooth signal we sample Gaussian noise from N (0, 2 )while for the step function, we sample a distribution with apoint mass at zero, uniform probability over [ 1, 0) [ (0, 1]
and correlate it across time by summing over a window ofsize . Finally, we constrain both noisy signals to be nonnegative and sum them to generate our input.
We generate data under this model for T = 50000 time
points and consider increasingly specialized optimization
objectives while measuring the error in recovering Y ? =XD(✓? ) + W , the underlying source signals corrupted byModelMean prediction
Nonnegative sparse coding`2 loss for y1`2 loss for y1 , `1 loss for y2`2 loss for y1 , `1 loss for y2 , penalty on kDyi k

MAE
0.37760.28430.12050.09940.0758

Table 1: Performance on disaggregation of synthetic data.

Total (kWh)ActualCategory
Base
A/C
Appliance
Average

Estimate

6
4

4Contextual
0.18490.19190.19000.1889Base

A/C

Appliance

100

2
80
Energy (kWh)Base (kWh)

NNSC
0.27930.28940.24160.2701

Table 2: Comparison of performance on Pecan Street
dataset, measured in mean absolute error (MAE).

2
0

0
4
A/C (kWh)Mean
0.25340.28490.22620.2548

60
40
20

2

0
100

0

200

Appliance (kWh)

Day

300

100

200

300

Day

4

Figure 3: Energy disaggregation results over entire time period for a single home from the Pecan Street dataset withestimated (left) and actual (right).

2

0
24

48

72

96

120

144

168

Hour

Figure 2: Energy disaggregation results over one week anda single home from the Pecan Street dataset.
noise. As can be seen in Table 1, by using `1 loss for y2 andadding gi (yi ) terms penalizing kDy1 k22 and kDy2 k1 , error
decreases by 37% over just `2 loss alone; in Figure 1, we
observe that our estimation recovers the true source signals
closely with the gi terms helping to capture the dynamics ofthe noise model for w2 .
As a baseline for this result, we compare to the mean prediction heuristic (predicting at each time point a breakdown
proportional to the overall probability of each signal) and toa state-of-the-art unsupervised method, nonnegative sparse
coding (Hoyer 2002). We apply sparse coding by segmenting the input signal into 1000 examples of 50 time points(1/4 the period of the sine wave, X1 (t)) and fit a sparsemodel of 200 basis functions. We report the best possible
source separation by assigning each basis function according to an oracle measuring correlation with the true source
signal and using the best value over a grid of hyperparameters. As can be seen in Table 1, the mean prediction heuristic
is nearly 5 times worse and sparse coding is nearly 4 times
worse than our best contextually supervised model.
Energy disaggregation with ground truth. Next we
consider the ability of contextual supervision to recover thesources of energy consumption on a real dataset from Pecan
Street consisting of 84 homes each with at least 1 year worthof energy usage data. As contextual information we construct a temperature time series using data from Weather Underground (http://www.wunderground.com/) measuring the temperature at the nearby airport in Austin, Texas.

The Pecan Street dataset includes fine-grained energy usage
information at the minute level for the entire home with anenergy breakdown labeled according to each electrical circuit in the home. We group the circuits into categories representing air conditioning, large appliances and base load andaggregate the data on an hourly basis to mimic the scenariopresented by smart meter data.
The specification of our energy disaggregation model isgiven in Table 3—we capture the non-linear dependence on
temperature with radial-basis functions (RBFs), include a“Base” category which models energy used as a functionof time of day, and featureless “Appliance” category representing large spikes of energy which do not correspondto any available context. For simplicity, we penalize eachcategory’s deviations from the model using `1 loss; but for
heating and cooling we first multiply by a smoothing matrix
Sn (1’s on the diagonal and n super diagonals) capturing
the thermal mass inherent in heating and cooling: we expect
energy usage to correlate with temperature over a windowof time, not immediately. We use gi (yi ) and the difference
operator to encode our intuition of how energy consumption in each category evolves over time. The “Base” category
represents an aggregation of many sources which we expect
to evolve smoothly over time, while the on/off behavior inother categories is best represented by the `1 penalty. Finally we note that in the Pecan Street data, there is no labeled
circuit corresponding exclusively to electric heating (“Heating”), and thus we exclude this category for this dataset.
In Table 2, we compare the results of contextual supervision with the mean prediction heuristic and see that contextual supervision improves by 26% over this baseline whichis already better than nonnegative sparse coding. Qualitatively we consider the disaggregated energy results for a sin-Category
Base
Heating
A/C
Appliance

FeaturesHour of day
RBFs over temperatures < 50 F
RBFs over temperatures > 70 F
None`i
↵1 ky1 X1 ✓1 k1↵2 kS2 (y3 X3 ✓3 )k1↵3 kS2 (y2 X2 ✓2 )k1↵4 ky4 k1

gi

2
1 kDy1 k2
2 kDy3 k1
3 kDy2 k1
4 kDy4 k1

Table 3: Model specification for contextually supervised energy disaggregation.

Base

A/C

Heating

Appliance

25

Energy (kWh)

20
15
10
5
0

Average energy (kWh)24

48

72 96 120 144 168
Hour

24

48

72 96 120 144 168
Hour

20

40

100 120
Week

140

160

1000

500

0
60

80

180

200

Figure 4: Disaggregated energy usage for a single homenear Fresno, California over a summer week (top left) and awinter week (top right); aggregated over 4000+ homes overnearly four years (bottom)
gle home over one week in Figure 2 and see that contextual
supervision correctly assigns energy usage to categories—alarge amount of energy is assigned to A/C which cycles on
and off roughly corresponding to external temperature, large
spikes from appliances happen at seemingly random times
and the smoothly varying base load is captured correctly. In
Figure 3, we consider the disaggregation results for the same
home across the entire time period and see that the contextually supervised estimates correspond very closely to theactual sources of energy consumption.
Large-scale energy disaggregation. Next, we turn to themotivating problem for our model: disaggregating largescale low-resolution smart meter data into its component
sources of consumption. Our dataset consists of over 4000
homes and was collected by PG&E from customers inNorthern California who had smart meters between 1/2/2008
and 12/31/2011. According to estimations based on survey
data, heating and cooling (air conditioning and refrigerators)
comprise over 39% of total consumer electricity usage (U.S.
Energy Information Administration 2009) and thus are dominant uses for consumers. Clearly, we expect temperature tohave a strong correlation with these uses and thus we provide
contextual supervision in the form of temperature informa-tion. The PG&E data is anonymized, but the location of individual customers is identified at the census block level and
we use this information to construct a parallel temperaturedataset as in the previous example.
We present the result of our model at two time scales,
starting with Figure 4 (top), where we show disaggregatedenergy usage for a single home over a typical summer andwinter week. Here we see that in summer, the dominantsource of energy consumption is estimated to be air conditioning due to the context provided by high temperature.
In winter, this disappears and is replaced to a smaller extent
by heating. In Figure 4 (bottom), itemized energy consumption aggregated across all 4000+ homes demonstrates thesebasic trends in energy usage. Quantitatively, our model assigns 15.6% of energy consumption to air conditioning and7.7% to heating, reasonably close to estimations based on
survey data (U.S. Energy Information Administration 2009)(10.4% for air conditioning and 5.4% for space heating). We
speculate that our higher estimation may be due to the model
conflating other temperature-related energy usages (e.g. refrigerators and water heating) or to differences in populations between the survey and smart meter customers.

Conclusion and discussion
The disaggregation of smart meter data into itemized energy
uses creates large opportunities for increases in efficiency;
as smart meters are already widely deployed and have beencollecting data for the past several years, millions of homes
stand to benefit. However, disaggregating smart meter data
is a challenging task due to its low-resolution sampling andlack of supervised information. We believe that with the development of contextual supervision described in this paper,
we have made a significant advancement in this area thathas been previously dominated by methods that rely on either high-resolution or supervised data that, unlike the smart
meter data, is not readily available.
An interesting direction for future work is the explicit
connection of our large-scale low-resolution methods withthe more sophisticated appliance models developed on
smaller supervised datasets with high-frequency measurements. However, there are clear limitations as to what can beobserved in a whole home power trace that is only sampled
once an hour. The development of refined statistical models
that produce confidence intervals around their estimations is
one avenue for dealing with this uncertainty. Still, the largest
gains are likely to come from an increase in sampling frequency, perhaps in a hybrid approach that varies the sampling rate in order to capture more accurate high-frequency
snapshots during periods of high activity.

References
Bell, A. J., and Sejnowski, T. J. 1995. An informationmaximization approach to blind separation and blind deconvolution. Neural computation 7(6):1129–1159.
Blumensath, T., and Davies, M. 2005. Shift-invariant sparse
coding for single channel blind source separation. SPARS
5:75–78.
Comon, P. 1994. Independent component analysis, a newconcept? Signal processing 36(3):287–314.
Darby, S. 2006. The effectiveness of feedback on energy
consumption. Technical report, Environmental Change Institute, University of Oxford.
Davies, M., and James, C. 2007. Source separation using
single channel ica. Signal Processing 87(8):1819–1832.
Ehrhardt-Martinez, K.; Donnelly, K. A.; and Laitner, S.
2010. Advanced metering initiatives and residential feedback programs: a meta-review for household electricitysaving opportunities. In American Council for an EnergyEfficient Economy.
Ghahramani, Z., and Jordan, M. I. 1997. Factorial hidden
markov models. Machine learning 29(2-3):245–273.
Hart, G. W. 1992. Nonintrusive appliance load monitoring.
Proceedings of the IEEE 80(12):1870–1891.
Hoyer, P. O. 2002. Non-negative sparse coding. In Neural
Networks for Signal Processing, 2002. Proceedings of the2002 12th IEEE Workshop on, 557–565. IEEE.
Institute for Electric Efficiency. 2012. Utility-scale smart
meter deployments, plans, and proposals. Technical report,
Institute for Electric Efficiency.
Kim, H.; Marwah, M.; Arlitt, M. F.; Lyon, G.; and Han, J.
2011. Unsupervised disaggregation of low frequency power
measurements. In SDM, volume 11, 747–758. SIAM.
Kolter, J. Z., and Jaakkola, T. 2012. Approximate inference in additive factorial hmms with application to energy
disaggregation. In International Conference on Artificial Intelligence and Statistics, 1472–1482.
Kolter, J. Z.; Batra, S.; and Ng, A. Y. 2010. Energy disaggregation via discriminative sparse coding. In Neural Information Processing Systems, 1153–1161.
Lewicki, M. S., and Sejnowski, T. J. 2000. Learning overcomplete representations. Neural computation 12(2):337–
365.
Neenan, B., and Robinson, J. 2009. Residential electricity use feedback: A research synthesis and economic framework. Technical report, Electric Power Research Institute.
Olshausen, B. A., and Field, D. J. 1997. Sparse codingwith an overcomplete basis set: A strategy employed by V1?
Vision research 37(23):3311–3326.
Parson, O.; Ghosh, S.; Weal, M.; and Rogers, A. 2012. Nonintrusive load monitoring using prior models of general appliance types. In 26th AAAI Conference on Artificial Intelligence.
Roweis, S. T. 2001. One microphone source separation. Advances in neural information processing systems 793–799.

Schmidt, M. N., and Mørup, M. 2006. Nonnegative matrix factor 2-d deconvolution for blind single channel source
separation. In Independent Component Analysis and Blind
Signal Separation. Springer. 700–707.
Schmidt, M., and Olsson, R. 2006. Single-channel speech
separation using sparse non-negative matrix factorization.
Smaragdis, P.; Raj, B.; and Shashanka, M. 2006. A probabilistic latent variable model for acoustic modeling. Advances in models for acoustic processing, NIPS 148.
U.S. Energy Information Administration. 2009. 2009
RECS survey data. Available at http://www.eia.
gov/consumption/residential/data/2009/.
Ziefman, M., and Roth, K. 2011. Nonintrusive applianceload monitoring: Review and outlook. IEEE Transactions
on Consumer Electronics 57(1):76–84.

Appendices to “Contextually Supervised Source Separation with Application to
Energy Disaggregation”

A

ProofsIn this section we prove the theoretical result of the paper, that when the contextual features are relatively uncorrelated
between different groups, we can recover the correct separation with high probability. We begin with a tail bound on
Gaussian random variables which is slightly specialized to handle the case of a singular covariance matrix.
Lemma 1. Suppose x 2 Rp ⇠ N (0, ⌃) with rank(⌃) = n. Then✓ ◆n/2⇢t
1
2
P kxk2 t 
exp
(t/
n
2
where

n)

(1)

is the largest eigenvalue of ⌃.

Proof. By Chernoff’s boundP kxk22for any ↵

t 

h
i
2
E e↵kxk2
e↵t

(2)

.

0. For any ✏ > 0, z ⇠ N (0, ⌃ + ✏I),
⇢
Z
h
i
2
1 T
E e↵kzk2 = (2⇡) p/2 |⌃ + ✏I| 1/2 exp
z (⌃ + ✏I) 1 z + ↵z T z dz
2
⇢
Z
1 T
= (2⇡) p/2 |⌃ + ✏I| 1/2 exp
z (⌃ + ✏I) 1 (I 2↵(⌃ + ✏I) 1 )z dz
2= (2⇡) p/2 |⌃ + ✏I| 1/2 (2⇡)p/2 |⌃ + ✏I|1/2 |I
2so taking the limit ✏ ! 0, we have that E[e↵kxk2 ] = |I|I2↵⌃| =nY(3)2↵(⌃ + ✏I)| 1/2

2↵⌃| 1/2 . Since ⌃ has only n nonzero eigenvalues,(1

2↵ i )

t 

(1

(1

2↵ )n

(4)

i=1

and soP kxk221.
2↵ )n/2 e↵t(5)Minimizing this expression over ↵ gives ↵ = t 2tn and substituting this into the equation above gives the desired
bound.
Now we are ready to prove the main result. First, recall that our theoretical analysis is concerned with the special caseof `2 loss, no regularization and Gaussian noise; we assume
yi = Xi ✓i? + wi
for some ✓i?

2 R
problem becomesni

and wi ⇠ N (0,

2
i I).

(6)

Furthermore, recall that under the choice of `2 loss, our optimization

minimizeY,✓subject to

k
X
1

2
i=1

k
X

kyi

yi = ȳi=1

1

Xi ✓i k22

(7)

and by taking gradients we have the optimality conditions
yi Xi ✓i + = 0 for i = 1 . . . k
XiT yi(8)

XiT Xi ✓i = 0 for i = 1 . . . k

where 2 RT is the vector of Lagrange multipliers corresponding to the constraints. These constraints imply that
XiT = 0 and summing over k we haveȳ X✓ + k = 0(9)
XT = 0Pk
where ✓ 2 Rn is a concatenation of all the ✓i ’s, X 2 RT ⇥n is a concatenation of all the Xi ’s and n = i=1 ni is thetotal number of features. This system of equations has the solution✓ˆ = (X T X) 1 X T ȳ(10)which is simply the least-squares solution found by regressing ȳ on all of the features X. Finally, note that our theorem
is concerned with the recovery of the product Xi ✓i which is often significantly easier than the recovery of ✓i itself.
Theorem 1. Given data generated by the model (6), and estimating ✓ˆ via (10), we have thathi
E kXi ✓ˆi Xi ✓? k22 = 2 tr XiT Xi (X T X)ii 1  2 ni ⇢i(11)Pkwhere 2 = i=1 i2 and ⇢i = max (XiT Xi (X T X)ii 1 ). Furthermore, for  0.1, with probability greater than
1r
4 2 ni ⇢i log(1/ )ˆ
RMSE(Xi ✓i ) .
(12)TProof. To write the problem more compactly, we define the block matrix W 2 RT ⇥k with columns wi , and define the“block-diagonalization” operator B : Rn ! Rn⇥k as
2
3✓1 0 · · · 0
6 0 ✓2 · · · 0 7
B(✓) = 6
.
(13)
..
.. 7
..
4 ...
.
. 5
.
0

···

0

Since Y = XB(✓? ) + W ,✓kˆ = XB (X T X) 1 X T (XB(✓? ) + W )1
XB(✓)
(14)

= XB(B(✓? ))1 + XB (X T X) 1 X T W 1
?

T

1

T

= XB(✓ ) + XB (X X) X W 1
For simplicity of notation we also denoteu 2 Rn ⌘ (X T X) 1 X T W 1.
Thus
ˆ = XB(u).
XB(✓? ) XB(✓)
Now, using rules for Gaussian random variables under linear transformations W 1 ⇠ N (0,N (0, 2 (X T X) 1 ). Finally, partioning u1 , u2 , . . . , uk conformally with ✓,Xi ✓? Xi ✓ˆi = Xi ui ⇠ N (0, 2 Xi (X T X) 1 X T )EXi ✓ ?

Xi ✓ˆi

2
2

=

2

2

(16)
IT ) and so u ⇠

i

(17)

tr XiT Xi (X T X)ii 1 .

(18)

ii

so

(15)

Xi (X T X)ii 1 XiT is a rank ni matrix with maximum eigenvalue equal to 2 ⇢i , applying Lemma 1 above gives✓✓◆ni /2⇢◆⇣⌘T ✏2
1 T ✏2
2
2
ˆP RMSE(Xi ✓i ) ✏ = P kXi ui k2 T ✏ 
exp
ni
.
(19)
2⇢
n i 2 ⇢i
2
i
Setting the right hand side equal to and solving for ✏ givesr
W ( 2/n /e)ni ⇢i 2
✏=
(20)
T
x
where W denotes the Lambert W function (the inverse of f (x) = xe ). The theorem follows by noting thatW ( 2/n /e)  4 log 1 for all n
1 when  0.1, with both quantities always positive in this range (notethat leaving the W term in the bound can be substantially tighter in some cases).
Since2