Self-Supervised Learning of Appliance Usage
Authors: Chen-Yu Hsu
Abbas Zeitoun
Guang-He Lee
Dina Katabi
Tommi Jaakkola
Venue:  (2020)
Keywords: Appliance usage
self-supervised learning
multi-modal learning
unsupervised learning

Abstract: Learning home appliance usage is important for understanding people's activities and optimizing energy consumption.  The problem is modeled as an event detection task, where the objective is to learn when a user turns an appliance on, and which appliance it is (microwave, hair dryer, etc.). Ideally, we would like to solve the problem in an unsupervised way so that the method can be applied to new homes and new appliances without any labels. To this end, we introduce a new deep learning model that takes input from two home sensors: 1) a smart electricity meter that outputs the total energy consumed by the home as a function of time, and 2) a motion sensor that outputs the locations of the residents over time.  The model learns the distribution of the residents' locations conditioned on the home energy signal. We show that this cross-modal prediction task allows us to detect when a particular appliance is used, and the location of the appliance in the home, all in a self-supervised manner, without any labeled data. 

Published as a conference paper at ICLR 2020
SELF-SUPERVISER LEARNING OF APPLIANCE USAGE
Chen-Yu Yu Hsu, Abbas Zeitoun, Guang-He Lee, Dina Katabi & Tommi Jaakkola Computer Science and Artificial Intelligence Lab Massachusetts Institute of Technology Cambridge, MA 02139, USA {cyhsu, zeitoun, guanghe, dk @mit. edu, tomni wwt.ssiaaaiiii........e......eie edu
ABSTRACT
Learning home appliance usage is important for understanding people's activities and optimizing energy consumption. The problem is modeled as event detection task, where the objective is to learn when a user turns an appliance on, and which appliance it is (microwave, hair dryer, etc.). Ideally, problem in an unsupervised way so that the method can be applied to new homes and appliances without any labels. To this end, we introduce new deep learning model that takes input from two home sensors: 1) a smart electricity meter function of time, and 2)
would like to solve the
that outputs the total energy consumed by the home a motion sensor that outputs the locations of the residents over time. The model learns the distribution of the residents' locations conditioned on the home energy signal. We show that this cross-modas prediction task allows us to detect when a particular appliance used, and the location of the appliance in the home, all in a eeff-supervised manner, without any labeled data.
INTRODUCTION
Learning home appliance usage patterns is useful for understanding user habits and optimizing electricity consumption. For example, knowing when person their microwave, stove, oven, coffee machine or toaster provides information about their eating patterns. Similarly, understanding when they use their TV, air-conditioner, and habits. Such information can be used to encourage energy saving by optimizing appliance usage (Armel al. 2,13), to track the wellbeing of elderly living alone (Donini et al.) [0033] Debes et al.) 2016), or to provide users with behavioral analytics (Zhou & Yang 2016 Zipperer et al. (0033). This data also useful for various businesses such home insurance companies interested assessing accident risks and utility companies interested in optimizing energy efficiency (Armel et al. 2013).
washer and dryer provides knowledge of their behavior
The problem can be modeled as event detection - i.e., given the total energy consumed by the house as function of time, we want to detect when various appliances are turned on. Past work has looked at analyzing the energy signal from the home utility meter to detect when certain appliances are on 1 Most solutions, however, assume that the energy pattern for each appliance is unique and known, and use this knowledge mowledge to create labeled data for their supervised models. (Kolter et al., 2010 Zhong et al. 2014:2013 Kelly & Knottenbelt] 2015) Zhang et al.) 2018 [Bonfigli et al. 2018). Unfortunately. such solutions do not generalize well because the energy pattern an appliance depends on its brand and can differ from one home to another (Kelly & Knottenbelt 2015 Bonfigli et al. 2018 The literature also contains some unsupervised methods, but they typically have limited accuracy (Kim)
al. 2011 Kolter & Jaakkola 2012 Johnson & Willsky| 2013 Parson et al. 2014 Wytock & Kolter
2014
Zhao et al.] 2016
Lange & Berges (018)).
Unsupervised event detection in a data stream is intrinsically challenging because we do not know what patterns to look for. In our task, not only may appliance energy patterns be unknown, but also the energy signal may include many background events unrelated appliance activation, such as the fridge or HVAC power eycling events.
One way to address this challenge is to consider the self-supervised paradigm. If different stream of data also observes the events of interest, we can use this second modality to provide self-supervising
'The utility meter outputs the sum of the energy of all active appliances in a house as function of time. For example, Samsung dishwasher may have different energy pattern from that of Kenmore dishwasher.

---

Published as a conference paper at ICLR 2020
signals for event detection. To that end, we leverage the availability of new fine fenouuuuot motion sensors which track the locations of people at home (Adib et al. [013] Joshi et al. 2015 Li et al. 2016| Chourchinn et al., 2017 Hsu et al. 2017b). Such sensors operate a consumer radar, providing decimeter-level location accuracy. They do not require people to wear sensors on their bodies, can operate through walls, and track people's locations in different rooms.
These location sensors indirectly observe the events of interest. Specifically, they capture the change in user locations as they reach out an appliance to set it up or turn (e.g. put food in microwave and turn it on). Hence, the output of such sensors can provide a second modality for self-supervision.
But how should one design the model? We cannot directly use location activation events. People can be next to an appliance but neither activate it nor interact with it. Moreover, we do not assume appliance locations are known a priori. We also cannot use the two modalities to learn joint representation of the event in shared space. This is because location and energy are unrelated most of the time and become related only when the event of interest occurs. Furthermore, there are typically multiple residents in the home, making it hard to tell which of them interacted with the appliance.
label for appliance
Our model is based on cross-modal prediction. We train a neural network that, given the home energy at particular time, predicts the location of the home residents. Our intuition is that appliance activation events have highly predictable locations, typically the location of the appliance. In contrast, background energy events (c.g. power cycling of the fridge) do not lead to predictable locations. Thus, model uses this learned predictability along with the associated location and energy representation to cluster the events in the energy stream. In addition, we use a mixture distribution to disentangle irrelevant location information of other residents in the home. Interestingly, our model not only learns when each appliance is activated but also discovers the location of that appliance in the home, all without any labeled data.
We summarize the contributions of this paper as follows:
The paper introduces new method for self-supervised event detection from weakly related data streams. The method combines neural cross-modal prediction with custom clustering based on the learned predictability and representation. We apply it to the task of detecting appliance usage events using unlabeled data from two sensors in the home: the energy meter, and location sensor. To evaluate our design, we have created the first dataset with concurrent streams of home energy and location data, collected from 4 homes over a period of 7 months. For each home, data was months. Ground truth measurements are provided via smart plugs connected directly to each appliance.
collected for
Compared to past work on unsupervised learning of appliance usage and a new baseline that leverages the two modalities, our method achieves significant improvements of 67.3% and 51.9% respectively for the average detection F1 score.
We will release code and dataset to encourage future work on multi- modal models for understand- ing appliance usage patterns and the underlying user behavior.
2 RELATED WORK
Energy disaggregation Our work is related to past work on energy disaggregation, which refers to the problem of separating appliance-Ievel energy from a home's total (or aggregate) energy signal. Past work in this domain can be broadly classified into two categories: supervised and unsupervised.
Supervised methods assume that the power signatures of individual appliances are available. They use data from individual appliances to obtain models for each appliance power signature, and then use those models detect appliance events from the aggregate energy signal. Early work learns sparse codes for different appliances (Kolter al., 2010) or uses a Factorial HMM (FHMM) (Ghahramani & Jordan 1996) to model each appliance as an HMM (Zhong et al. 2014 2015). Other work uses matrix factorization approaches to estimate monthly energy breakdowns (Batra al. 2017 2018). More recently, neural networks have been used to model appliances (Kelly & Knottenbeli 2015
Zhang al. 2018 Jia et al. 2019 Bonfigli al. 2018), where extracting appliance-level energy is formulated as a de-noising problem. However, supervised solutions typically do not generalize well
Project website: http://spplee.ssailii..ed
2

---

Published as a conference paper at ICLR 2020
Power (Wattss
2000
Power (Watts)
Microwave
on off
Disposer
4000
2000
off
1000
2000
1000
22
00
02
12
14
16
20:00
20:30
20.50
Time (Hours)
Time
(a) Aggregate energy data (over one day)
(b) Zoom in around 20:30 in (a)
y (meters)
y (meters)
(meters)
20
50
Time (seconds)
X (meters)
(d) Location data for two people (top-down view)
(c) Location data for two people over one minute
Figure 1: Aggregate energy signal and people's indoor location data
to new homes (Kelly Knottenbelt 2015 Bonfigli et al. 2018). This is because two appliances of the same type (c.g. coffee machine) in different homes often manufactured by different brands, and thus have different power signatures.
Unsupervised methods do not assume prior knowledge of appliance signatures; they attempt to learn those signatures from the aggregate energy signal. Early approaches use variants of FHMM, and learn appliance HMMs with Dopectatiio-MMaxmiiaatee
(Kim et al.) 2011), approximate footprint extraction procedures (Kolter & Jaakkola, 2012). or using expert knowledge to configure prior parameters (Johnson & Willsky [0133] Parson al. 2014 Some papers propose using contextual information (such as temperature, hour of the day, and day of the week) (Wytock & Kolter 2014), or use event-based signal processing methods to cluster appliances (Zhao al. 2016). More recently,
Lange & Berges (2018) proposed using recurrent neural network as the variational distribution learning the FHMM. In contrast, our work leverages people's location data as a self-supervising
signal. We cluster appliance events through learning the relation between energy events and people's s locations, and also learn appliance locations as a by-product.
Motivated by in-home applications and continuous health monitoring. recent years have witnessed an increasing number of indoor location sensing systems (Adib et al.. 2015 Joshi et al. 2015 Li et al. 2016; Ghourchian et al. 2017). They infer people's locations passively by analyzin
Passive location sensing
how people change the surrounding radio signals (e.g. WiFi) and do not require people to wear any sensors. These sensors have been used for various applications including activity recognition (Wang et al. (001|/2155 sleep monitoring (Zhao et 2017 [ et al. 2017a), mobility and behavioral sensing (Hsu al. [017b] 2019), and health monitoring (Kaltiokallio et al.. 2012). In work, we leverage the availability of such sensors to introduce location data as an additional data modality for learning appliance usage patterns.
Self-supervised multi-modal learning Our work is related a growing body of work multi- modal learning. Most approaches learn to encode the multi-modal data into a shared space (Gomez al. 2017 Harwath al.) 2018] Owens & Eiros] 2018 Zhao al. 2018 (019). In contrast, since our two modalities are mostly unrelated and become related only when an activation event happens, we learn to predict one modality conditioned on the other. Our work also related cross-modal prediction (Krishna et al. 2017: Owens et al.) 2016) Zhang et al.) 2017) but differs from it in an essential way. Past work on cross-modal prediction typically uses the prediction as the target outcome (e.g. output text for video captioning). In contrast, activation events. Thus, we design our method to leverage the learned predictability and cross- modal mapping for clustering activation events. Furthermore, introduce mixture prediction design to disentangle unrelated information in our predicted modality (location measurements unrelated to energy events).
objective is to discover the hidden appliance
3

---

Published as a conference paper at ICLR 2020
3 PROBLEM FORMULATION
Our goal is learn appliance activation events in an unsupervised way, using two input streams: home aggregate energy and residents' location data. Figure shows the two data modalities. We describe each of them formally and define appliance "events" below.
Aggregate energy signal A household's total energy consumption is measured by a utilityy meter regularly. This measures the sum of the energy consumed by all appliances at each point in time. We denote the aggregate energy signal by y (y1, (31.32,... V2...... .rr,, where y/ E Rt. Suppose there are total of K appliances in home, and each appliance's energy signal denoted by æ (21,k, T2,k, ,., Fr,k), where Ft,k R4. Only the aggregate energy signal is observed, yr =
_k=1 Ft.k + Eg, where E (O, o²) is the background noise.
K
Figure Ia shows one day of an aggregate energy signal. The base power level shifts constantly throughout the day, depending on the background load (e.g. ceiling lights). Added on top of the base level are the various appliance events. Figure lb zooms in around 20:30, and shows examples of those events. The stove was turned on around 20:28, and its power continued cycle between few levels. While the stove was on, the microwave was also turned on and ran for few minutes, and the garbage disposer was turned on shortly.
Indoor location data We use a single location sensor similar to that in Hsu et al. 2017b) to measure people's indoor locations passively. The sensor sends out radio signals and analyzes the reflections to localize multiple people. Similarly to a regular WiFi router, the sensor has a limited coverage area of up to 40 feet. Suppose there are Pt people in the coverage area time The location data is denoted by 4 = (4,1,4,2
(.P.), where 4 E I²² is the x-y location of person p at time We can represent the location data over multiple time frames as l (41, .,,....,Zz'" Figure Ic] shows one minute of location data from two people, and Figure Id shows the data from top-down view.
Appliance activation events When an appliance is turned on, it causes a jump in energy consump- tion, i.e. a leading edge in the energy signal, shown in Figure Tb. We call such pattern an appliance activation event. On the other hand, when an appliance changes its internal state, can also cause a change in the energy signal as shown in the same figure. We call such a pattern a background event. We are interested in discovering activation events to learn appliance usage patterns. Thus, for each jump the aggregate signal, we take time window (of 25 seconds) centered around that jump, and analyze it to detect whether it is an activation event and which appliance it corresponds to.
4 MODEL
Our model operates on time windows (25 seconds) centered around jumps in aggregate energy signal, and the corresponding time windows of location data. The model aims detect appliance activation events by finding windows with highly predictable user locations conditioned on the energy signal.
Figure 2 shows our model. The idea underlying model is to first learn representation of appliance event windows that separates the information about appliance type, ct,cat= from the shape of the energy signal, 2t,c This is achieved through the appliance energy encoder E. We can then use the appliance type encoding to predict the location data through the location predictor L,, which is conditioned 2t,cat- Since people's locations have information unrelated to appliance events, the total location predictor is a mixture of T., and a second module T.y which captures ienttinneependent location information. Below, we describe the design of these modules. More details about the neural network parameters and implementation are discussed in Appendix 8.4
Appliance Energy Encoder Given window of aggregate energy signal yest+w (Yt, Dt.1........ :etw the encoder F encodes the series into an event vector 2,. We break the event vector into two parts: a categorical vector zt,eot and continuous vector 2t,cont. We aim to capture the appliance type with ct. (e.g. microwave vs. dishwasher) and use zt. to capture the variability within the signature of the same appliance. A softmax layer is applied 21,cat to ensure that it is valid distribution over appliance types. E is parametrized using convolution layers, with one fully connected layer to produce Zt,comt and another for Zt,eat- We denote by OF the parameters of the encoder.
We remove the hase power level in each window by subtracting the minimum in the window.
4

---

Published as a conference paper at ICLR 2020
Pestwa
P P(leesowy
(Warts)
D
a
(1 - a)) P(IC))
500
Eree
P.cont
Pr.cut
P(
Lio
10
19
Time (seconds)
E
Le
Lg
Ve:ewwa (energy signal)
4ec4w2 (locution)
(context)
X
Figure 2: Model architecture. The model learns encode energy sig- nals into event vectors while learning to predict the concurrent location data. The location predictor Le conditioned on the energy event ot,oats and Lg is conditioned on context features. The decoder D takes event vectors and learns to reconstruct the original energy signal.
Figure 3: Total energy
signal (top) and location information (bottom) seen by the model
Location predictors We try to predict the location data conditioned on the appliance event, i.e. we predict window locations 1or+ = (.....-...... ,,, centered around the appliance event. We handle multiple people's locations with mixture model. Specifically, use Le to predict locations related to energy events and Lg to handle other locations. The final prediction is a mixture of predictions from L. and Ly:
pe: (1 C) = a po.., (atz),
a pp:., (4cc),
where per.() (-) is parametrized by Le with parameters OL. por. (-) parametrized by Lg with parameters Or., 0, = (Or.. 0r.,) }. and e includes context features. We use the number of people in the window (reported by the location sensor), the time of day, and the day of the week as the context features. The weight a depends on the number of people in the current window
1/P..
To represent location data, we blur each location measurement with a Gaussian kernel on the x-y plane to create an image, and process the window of locations 4u+ww into frames of images (Figure 3 We reuse the notation E X to represent frames of location images, where |XX YY) the number of discretized points on the x-y and time dimensions. By presenting location data as images, we also remove the variable Pt while handling variable number of people in each frame.
We choose pe.., (4|zz,cat) to multivariate Gaussian with a diagonal covariance structure: Pe., (4) (oox)) 2 (l: Ee) = II [ WWle, Pz,y, o2,.)) where μa (z(z OI.) E
X and we choose J...t be a constant. We use 3D deconvolution networks to model L. which takes 2t,cnt as input and outputs the means of the location distributions. We model poz., (-) and Lg in similar way.
During training, given a window of data (l, B:.tt: c). we minimize the negative log likelihood of the mixture distribution in predicting the locations:
Lioe (0g. 02)
log pe, (1) c).
Note that since the gradient flows through zt,cats the likelihood is a function of both 0 and 0,,- Hence, the encoder F. also learns to encode the energy series based on the concurrent location data.
Energy Decoder The decoder D takes both Zt,cat and 2t,cont and learns to reconstruct the original input energy series by predicting - The decoder D is parametrized using deconvolution layers. We minimize the reconstruction loss during training:
Crea (0z 0p) = 1| - Westwyll
The reconstruction loss encourages the encoder E to produce good initial vectors for L, to predict lo- cations. At the same time, it serves a regularizer to prevent encoder E from generating meaningless vectors by overfitting location predictions.

---

Published as a conference paper at ICLR 2020
Training We train all components to jointly optimize the location predictions and energy recon- struction. We minimize the total loss: Liotal = Lloc + A Free over all windows of data, where is
parameter to balance the two terms. The training details are discussed in Appendix 8.4
4.1 CLUSTERING APPLIANCE EVENTS WITH GROSS-MODAL PREDICTIONS
Once the model is trained, we obtain for each window of energy data its appliance event vector ct, and its cross-modas location prediction pe,.. (-Izt,cut). Next, we use these two vectors for clustering. We design a density-based based clustering algorithm leveraging the cross-modal relation we learned. Our intuition is that activation events for the same appliance will cluster together since they have the same appliance type and the same location. We omit the cat notation below for brevity.
is typically difficult to cluster in a space learned by neural encoder because the transformation is highly non-linear and the distance metric is not well-defined. We circumvent this problem by associating the encoded space with a Euclidean space, in which we can easily measure distance. Specifically, for two event vectors 21 and Ø2, we can measure their distance in the location space using Pa... (-121) and pe... (-1z2)-
The location prediction poz. (+12,) represents the likelihood of observing any location Lz,, around the time of the appliance event. We found that for events related to human activities (c.g., turning on a kettle or microwave), P0L, (-12,) shows peak value at the location of the appliance in the x-y space at the time when person interacted with the appliance. For events not related to human activities (e.g. fridge cycles or random background events), po... (-12;) has low values and diffused.
We define the location predictability score (or the confidence of location prediction) as s(2;) = maxx.y.t PoL, (.. 122),, and the location distance Dise between two events as: Dioe (20 22) = ||(=;- 2,2 Vr v2)122, where (2,, ) arg max po.. (Lu,,, 1 Similarly, the neighborhood distance Dab between two events is defined Dub (2b x2) 21 #2112-
Our clustering algorithm starts with a z; with high predictability score s(z;). ). It expands the cluster around z, 's local neighborhood in the space. It stops expanding if neighbor's location distance Dioe is too far from the cluster center. If all neighbors of the current cluster are visited and none has cluster from another event with high predictability score. The
small enough Dioe, start algorithm is described formally in Algorithm[] |
We discuss the choice of parameters in Appendix 8.5
Algorithm 1 Clustering energy events with the learned cross-modal relations
Input: {zi} and s(-): event vectors and their location predictability scores
Ta. TJPee Ta: thresholds for predictability score, location distance, neighborhood distance Nmin: the minimum number of samples to form a valid cluster
Output: Clusters appliance activation events that are associated with consistent location
1: procedure EL-SCAN(1,.. TiD MJ., Nmia)
2: 2 + (2,) > 71 k-
3: while 2 + do
Zseed arg max s(2,)
cluster; + [Z J. <- k +1
Start a new cluster
6:
ExpandCluster((; Zseed; TiD - T)))
7: end while
8: return clusters with at least Nmwi examples
end procedure
10:
11:
function EXPANDCLUSTERIA 2. 1/D 1)2)
12:
Zug f- compute current cluster center
13:
Zub + {z, 2 Dab < n) and Droe(zi, Zue) nDwce
Find valid neighbors
14:
15:
cluster, < cluster, U Z.b
16:
Repeat ExpandCluster() for all in Znb
17:
end function
WWe choose À to be 0.1 in our experiments to put more emphasis on the location prediction. 'In the implementation, compute the location predictability score as max Pr.yt for simplicity.

---

Published as a conference paper at ICLR 2020
5 DATASET
We collected concurrent streams of aggregate energy signal and location data from 4 homes over 7 months. We use this dataset for our evaluation. To obtain ground truth labels of appliance events, we deployed programmable smart plugs on the power outlet associated with each appliance. Since not all appliances can be measured by a smart plug (e.g. some appliances do not have accessible power outlets), we also developed a labeling tool for manual labeling. The tool allows labelers to label appliance events from the aggregate energy signal, with the help of smart plug data and information collected from the home residents. The choice of sensors and their sampling rates are detailed in Appendix 8.1
6 RESULTS
We evaluate model and clustering algorithm unsupervised appliance activation event detection and their learned appliance locations. We name our approach SAPPLE suefr-sueer APPliance usage LEarning).
6.1 UNSUPERVISED APPLIANCE EVENT DETECTION
For appliance event detection, we compare with four baselines. Our method and two baselines have access to location information. EL- takes both energy and location data as input and directly clusters them using k-means (Arthur& & Vassilvitskii 2007 E-only-Kmentd signal with k-means. Methods with location information pre-filter the events and discard events without any location data, as they are unlikely to activation events. The other two baselines only take the total energy signal input: AFAMAP (Kolter & Jaakkola 2012) uses factorial HMM, and VarBOLT (Lange Berges 2018) uses a recurrent neural network to model aggregate appliance signals. We use publicly available implementations for these methods (implementation bb.
clusters only the energy
We use the same hyper-parameters for the network architecture, training, and clustering algorithm across all homes. As our clustering algorithm is non-parametric, we choose the same number of clusters that it discovers for other methods if possible. For VarBOLT, we report results using 10 clusters, since the training time grows exponentially with the number of clusters and training with more clusters is prohibitively slow. As in past unsupervised work, we report the detection Fl scores based on the best cluster assignments with the ground truth appliances.
Table T shows that SAPPLE has an average detection F1 score of 72.8%, outperforming other baselines ranging from 4.0% 20.9%. As reported by Bonfigli et al. (2018), AFAMAP performs better when appliance-level data is available for training the HMMs. In the unsupervised setting, however, its footprint extraction procedure does not always produce meaningful HMMs for individual appliances (Bonfigli et al.) 2018 8 Beckel et al., 2014), causing degraded performance. VarBOLT's training objective focuses on explaining the total amount of energy multiple components (clusters) to model appliances that are on for long period (e.g. fridge, heater, and (tryer/wahher,, These types of appliances generate many background events, making the algorithm focus less activation events of other appliances.
home. Thus, it often uses
Comparing our method with baselines that also have location information (E-only-Kmeans and EL- EL-Kmeans), our approach still outperforms them significantly.
performs better than AFAMAP and VarBOLT, showing that the presence of location data is highly related to activation events. However, naively using location data for clustering does not improve the results by much, as Emeans performs only slightly better than E-only-Kmeans. This is because not all location data is related to appliance events and vice versa. Our approach "cleans up" the data by learning the relation between the two modalities and discovers clusters with strong cross-modal predictability.
Table 2 shows break down of our results for different appliances.
6.2 ABLATION STUDY
We perform an ablation study to show our results are contributed by all components in method. As shown in Table 3, we compare our clustering algorithm (Method I) with different algorithm that concatenates the learned multi-modal embeddings (#t,cat and poz., (+(z),eat)) and directly clusters
The data collection approved by our institutional review board (IRB).
Foor each window of data, -m-Kea concatenates the energy signal, the frames of location images (flattened 1-d vector), and the context vector to create the feature vectors clustered by k-means.
7

---

Published as a conference paper at ICLR 2020
Table Unsupervised Appliance Event Detection. Averaged FI scores (%) of all appliances.
Methods w/ location information
Methods w/o location
Na Nappliances
SAPPLE
EL-Kmcans
E-only- Kmeans
AFAMAP
VarBOLT
Home 1
8
82.3
26.9
10.2
6.0
4.4
Home 2
8
69.1
19.5
19.8
5.5
3.7
Home 3
6
76.2
16.5
15.1
4.1
3.3
Home 4
63.6
20.5
20.3
6.5
4.5
Average
72.8
20.9
16.4
5.5
4.0
Table 2: Unsupervised Appliance Event Detection. Our method's Fl score (%) for each appliance.
Home
Home
Home
Home 4
Kettle
91.9
98.6
Table 3: Ablation study.
Hair dryer
88.0 98.3
1.1
Coffee machine
96.1
75.6
90.4
Methods
Avg. (%)
Microwave
81.9
82.1
88.1
96.7
Stove-activation
90.6
76.7
92.7
SAPPLE
72.8
Disposer
62.5
78.5
53.1
2
Learned emheddings
58.8
Toaster
49.1
71.1
K-means
Blender
6.2
3
Remove Lg
68.5
Dryer
88.1
4
Remove Le & Lg
27.6
Iron
71.1
Rice Cooker
0.0
Others
49.4
96.1
83.4
92.3
Average
82.3
69.1
76.2
63.6
them with k-means (Method 2). Our clustering algorithm is more effective than directly clustering the multi-modal embeddings, providing an improvement of 14.0% in the average score. This is because clustering algorithm treats the two modalities differently. For location predictions, we can leverage our understanding of physical distance to set cluster boundaries. For the energy embedding, since it is non-linear mapping with no clear distance metric, algorithm iteratively groups together events the embedding neighborhood that have approximately the same locations.
Apart from clustering algorithm, evaluate the benefits of mixture component Lg by experimenting with removing I.g from the model, which reduces the F1 score by 4.3% (Method 1 vs 3). This shows the importance of having T.g extract background motion allow the location predictor L. to focus on modeling the person who interacts with the appliance.
We also consider removing both Lg and Le, and clustering the input based only the energy embedding ot, since there is no learned location predictions. The results shown under Method demonstrate the importance of the location embedding generated by the combination of Ly and Le-
6.3 LEARNED APPLIANCE LOCATIONS
Our model also learns the locations where people interacted with appliances, which are typically close to the appliances' physical locations (we discuss remotely activated appliances in Appendix (8.6). For each appliance event, we take the location predicted by Le with the highest predictability score, and compare that with the ground truth appliance location measured by laser meter. The average location prediction error is 0.65 meters with a standard deviation of 17 meters across homes. The errors mostly due to location offsets between the person and the appliance. Figure 4agure/alshows the location predictions and their ground truth of several appliance events in Home The corresponding energy signals are shown in Figure 4b Figure 4e
The location information also helps disambiguate appliances with similar energy signals. For example, although the hair dryer and kettle (Figure and Figure 44) have very similar energy signatures, their different locations (green and orange in Figure
4a) guide the model to encode their events differently.
8

---

Published as a conference paper at ICLR 2020
Disposer Hair onyer
2200
y (meters)
Power (WWNEN))
3300
Power (Watts)
Power (Worts)
2200
Power (Wetts
touth
1200
2020
Time (saconds)
Time (seconds)
Time (saconds)
Time (seconds)
(meters)
(h) Microwave
(c) Disposer
(d) Hair dryer
(c) Kettle
(a) Learned locations
Figure 4: Energy signals of discovered activation events and their learned locations from Le-
6.4 LOCATION PREDICTIONS OF Le VS Lig
We visualize the location predictions from the event-related predictor Le and the event-independent predictor L to illustrate how they handle scenarios with multiple people. SJswos example of how the mixture design handles the two types of locations. Since Le is conditioned on energy events, it naturally learns predict locations related to appliance events. In this case, the location of the hair dryer is predicted by Le (Figure 5b). On the other hand, Lg predicts the typical locations people tend to stay (e.g., the couch in Figure Scb based the context. Having Lg explain the other locations helps La focus on learning the event-related locations.
y (meters)
(meters)
(meters)
Hair dryer
Couch
10
(meters)
(meters)
(meters)
(a) Observed locations (two people) (b) L, 's prediction (event-related) (c) Ly 's prediction (other locations)
Figure 5: Observed locations and predictions of Le and Ly at given time for hair dryer event.
6.5
CONTEXTUAL LOCATION INFORMATION AND CLUSTER VISUALIZATIONS
In Appendix 8.2 we discuss emerging contextual relations between indoor locations through learning cross-modal predictions. We also visualize the learned event vectors to shed light on the design rationales behind clustering algorithm in Appendix 8.3
7 CONCLUSION
We introduced a self-supervised solution for learning appliance usage patterns in homes. We infer appliance usage by learning from data streams of two modalities: the total energy consumed by the home and the residents' location data. Our approach improves on unsupervised appliance event detection significantly, and learns appliance locations and usage patterns without any supervision,
ACKNOWLEDGMENTS
The authors thank the members of NETMIT at MIT and the reviewers for their feedback and helpful comments. We thank the participants in our study for facilitating the sensor deployments in their homes. We also thank the various companies sponsoring the MIT Center for Wireless Networks and Mobile Computing.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint 1803::66030444 2016.
"W discuss limitations in Appendix 8.6

---

Published as a conference paper at ICLR 2020
Fadel Adib, Zachary Kabelac, and Dina Katabi. Multi Aulti-person localization via rf body reflections. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15), pP- 279-292, 2015.
K Carrie Armel, Abhay Gupta, Gireesh Shrimali, and Adrian Albert. Is disaggregation the holy grail of energy efficiency? the case of electricity. Energy Policy, 52:213-232 2013.
David Arthur and Sergei Vassilvitskii. k-means++:
The advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium Discrete algorithms, pp. 1027-1035. Society for Industrial and Applied Mathematics, 2007.
Nipun Batra, Hongning Wang, Amarjeet Singh, and Kamin Whitehouse. Matrix factorisation for scalable energy breakdown. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Nipun Batra, Yiling Jia, Hongning Wang, and Kamin Whitehouse. Transferring decomposed tensors for scalable energy breakdown across regions. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Christian Beckel, Wilhelm Kleiminger, Romano Cicchetti, Thorsten Staake, and Silvia Santini. The eco data set and the performance of non-intrusive load monitoring algorithms. In Proceedings of the 1st ACM Conference on Embedded Systems for Energy-Efficiee Buildings, pp. 80-89, ACM, 2014.
Roberto Bonfigli, Andrea Felicetti, Emanuele Principi, Marco Fagiani, Stefano Squartini, and Francesco Piazza. Denoising autoencoders for non-intrusive load monitoring: improvements and comparative evaluation. Energy and Buildings, 158:1461-1477 2018.
Christian Debes, Andreas Merentitis, Sergey Sukhanov, Maria Niessen, Nikolaos Frangiadakis, and Alexander Bauer. Monitoring activities of daily living in smart homes: Understanding human behavior, IEEE Signal Processing Magazine, 33(2)881 2016.
Lorenzo Maria Donini, Eleonora Poggiogalle, Maria Piredda, Alessandro Pinto, Mario Barbagallo, Domenico Cucinotta, and Giuseppe Sergi. Anorexia and eating patterns in the elderly. PloS one, (5)e63333,, 2013.
emonPi. Open energy monitor httppps://oeeeeeeeeyooooiiiieeee org/
Zoubin Ghahramani and Michael Jordan. Factorial hidden markov models. In Advances in Neural Information Processing Systems, pp. 472 1996.
Negar Ghourchian, Michel Allegue-Martinez, and Doina Precup. Real-time indoor localization in smart homes using semi-supervised learning. In TwentyNinth IAAI Conference, 2017.
Lluis Gomez, Yash Patel, Marçal Rusiñol, Dimosthenis Karatzas, and CV Jawahar. Self-supervised learning of visual features through embedding images into text topic spaces. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4230-4239, 2017.
David Harwath, Adria Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, and James Glass. Jointly discovering visual objects and spoken words from raw sensory input. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 649-665, 2018.
Chen-Yo Yu Hsu, Aayush Ahuja, Shichao Yue, Rumen Hristov, Zachary Kabelac, and Dina Katabi. Zero-effort in-home sleep and insomnia monitoring using radio signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(3):1-18, 2017a.
Chen- Yu Hsu, Yuchen Liu, Zachary Kabelac, Rumen Hristov, Dina Katabi, and Christine Liu. Extracting gait velocity and stride length from surrounding radio signals. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 2116-2126. ACM, 2017b.
Chen- Yu Hsu, Rumen Hristov, Guang- He Lee, Mingmin Zhao, and Dina Katabi. Enabling identifi- cation and behavioral sensing in homes using radio reflections. In Proceedings of the 2019 CHI Conference Human Factors in Computing Systems, pp. 1-13, 2019.
10

---

Published as a conference paper at ICLR 2020
AFAMAP htt.///ennnoooooo///....///////e
Mat htt.l//aaallorrttts//okerreeeeee alg a.
Variational BOLT implementation.
b.
Yiling Jia, Nipun Batra, Hongning Wang, and Kamin Whitehouse. tree-structured neural network
model for household energy breakdown. In The World Wide Web Conference, pp. 2872-2875 2019.
Matthew J Johnson and Alan S Willsky. Bayesian nonparametric hidden semi-markov models. Journal of Machine Learning Research, 14(Feb):673-701, 2013. 2013.
Kiran Joshi, Dinesh Bharadia, Manikanta Kotaru, and Sachin Katti. Wideo: fine-grained device-free motion tracing using rf backscatter. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15), pp. 189-204 2015.
Ossi Kaltiokallio, Maurizio Bocca, and Neal Patwari. Follow@ @ grandma: Long-term device-free localization for residential monitoring. In Local Computer Nenworks Workshops (LCN Workshops), 2012 IEEE 37th Conference on, pp. 991-998. IEEE, 2012.
Jack Kelly and William Knottenbelt. Neural nilm: Deep neural networks applied to energy disag- gregation. In Proceedings of the 2nd ACM International Conference on Embedded Systems for Energy-Efficient Built Environments, pp. 55-64. ACM, 2015.
Hyungsul Kim, Manish Marwah, Martin Arlitt, Geoff Lyon, and Jiawei Han. Unsupervised disag- gregation of low frequency power measurements. Proceedings of the 2011 SIAM international conference on data mining, pp. 747-758. SIAM, 2011.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
J Zico Kolter and Tommi Jaakkola. Approximate inference in additive factorial hmms with application to energy disaggregation. In Artificial intelligence and statistics, pp- 1772-148, 2012.
J Zico Kolter, Siddharth Batra, and Andrew Y Ng. Energy disaggregation via discriminative sparse
coding. In Advances in Neural Information Processing Systems, pp. 1153-1161, 2010.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference computer vision, pp. 706-715. 2017.
Henning Lange and Mario Berges. Variational bolt: approximate learning in factorial hidden markov models with application to energy disaggregation. In T'hirty-Scoond AAAI Conference Artificial Intelligence, 2018
Xiang Li, Shengjie Li, Daqing Zhang, Jic Xiong, Yasha Wang, and Hong Mei. Dynamic-music: accurate device-free indoor localization. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 196-207. ACM, 2016.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2NNo)):5579-2605 2008
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 631-648, 2018.
Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2405-2413, 2016
Oliver Parson, Siddhartha Ghosh, Mark Weal, and Alex Rogers. An unsupervised training method for non-intrusive appliance load monitoring. Artificial Intelligence, 217:1-19, 2014
TP-link. Tp-link smart plug hs1 10 httpt:tpp/// www. http:://wwwwwtwwwt/ee com uk home http:h/mmeeettto.iiieeeeeee
smart- amarttpplgg/i /hs110/1/

---

Published as a conference paper at ICLR 2020
Wei Wang, Alex X Liu, Muhammad Shahzad, Kang Ling, and Sanglu Lu. Understanding and modeling wifi signal based human activity recognition. In Proceedings of the 2 annual international conference mobile computing and networking, pp. 63-76. ACM. 2015.
Yan Wang, Jian Liu, Yingying Chen, Marco Gruteser, Jie Yang, and Hongbo Liu. E-eyes: device-free location-oriented activity identification using fine-grained wifi signatures. In Proceedings of the 20th annual international conference on Mobile computing and networking, pp. 617-628. ACM, 2014.
Matt Wytock and J Zico Kolter. Contextually supervised source separation with application to energy disaggregation. In Twenty-eighth AAAI conference on artificial intelligence, 2014.
Chaoyun Zhang, Mingjun Zhong, Zongzuo Wang. Nigel Goddard, and Charles Sutton. Sequence- to-point learning with neural networks for non-intrusive load monitoring. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1058-1067, 2017.
Bochao Zhao, Lina Stankovic, and Vladimir Stankovic. On training-less solution for non inttrussvvv appliance load monitoring using graph signal processing. IEEE Access, 4:1784-1799, 2016.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 570-586, 2018.
Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proceedings of the IEFE International Conference on Computer Vision, pp. 1735-1744, 2019.
Mingmin Zhao, Shichao Yue, Dina Katabi, Tommi Jaakkola, and Matt Bianchi. Learning sleep stages from radio signals: A conditional adversarial architecture. In International Conference Machine Learning, pp. 4100-4109, 2017.
Mingjun Zhong, Nigel Goddard, and Charles Sutton. Signal aggregate constraints in additive factorial hmms, with application to energy disaggregation. In Advances in Neural Information Processing Systems, pp. 3590-3598 2014.
Mingjun Zhong, Nigel Goddard, and Charles Sutton. Latent bayesian melding for integrating individual and population models. In Advances neural information processing systems, pp. 3618-3626, 2015.
Kaile Zhou and Shanlin Yang. Understanding household energy consumption behavior: The con- tribution of energy big data analytics. Renewable and Sustainable Energy Reviews, 56:810 80098, 2016.
Adam Zipperer, Patricia Aloise Aluise-Young, Siddharth Suryanarayanan, Robin Roche, Licko Earle, Dane Christensen, Pablo Bauleo, and Daniel Zimmerle. Electric energy management in the smart home: Perspectives on enabling technologies and consumer behavior. Proceedings of the IEEE, 101(11):2377-2408 2013.
12

---

Published as a conference paper at ICLR 2020
8 APPENDIX
8.1 SENSORS DETAILS
In this section, we describe details of the sensors used in our dataset collection.
Aggregate energy signal For flexible data collection, install sensor (emonPij the main circuit breaker in each house as a proxy for the utility meter. We programmed the sensor to collect the aggregate energy signal at 1.2 kHz. We down-sampled the data to 10 Hz for problem to emulate the achievable data rate from a utility meter hardware (Armel et al. 2013
Location data The wireless location sensor built on design similar to Hsu et al. (2017b). is single stand-alone sensor that hangs on the wall, and passively collects multiple people's ' locations with decimeter-level accuracy. We nown-sampled the location streams to Hz.
Appliance-level data (for ground truth labeling) We TP-Link smart plugs (TP-link) with energy monitoring features for collecting appliance-level data. We wrote custom software using available APIs collect appliance energy signals Hz. For appliances that cannot be connected to smart plug, we asked the residents to write down appliance usage times to help with manual labeling.
8.2 CONTEXTUAL, LOCATION INFORMATION VTA LEARNED APPLIANCE EVENTS
By analyzing the location predictions of Le conditioned different appliance events, also discover interesting contextual relations between different indoor locations. Figure 6 visualizes the location predictions at different frames around kettle event. We plot the per-frame location predictability score (or prediction confidence) over time in Figure 6a The score peaks around = Os, the time of the event. This is because when people turn on kettle, they may approach it from different locations, but the location when they push the button is consistent and can be predicted correctly shows the kettle's 's location (Figure 6d).
confidently. As result, the prediction
Interestingly, a smaller peak of predictability score shows at t = - 10s in Figure ba If we look at the location prediction from moves from the sink to the kettle TO This is because people often fill water at the sink before starting the kettle. Through learning the cross-modal relation, contextual information among locations also emerges as different appliance events are discovered.
10s to Os (Figure [b]- Figure (6d) we see how the prediction
Sink
conficen
0.5
Moving
Tumn kettle
(meters)
(meters)
(meters)
03
water sink
Kettle
03
15
15
10
10
10
time (seconds)
(meters)
(metters)
(meters)
(a) Prediction confidence time
(b) -10s
(c) -68
(d) t Os
Figure 6: Visualizing location predictions different times conditioned kettle event.
8.3 VISUALIZATION OF THE LEARNED EVENT VECTORS AND LOCATION PREDICTABILITY
To illustrate what the model learns and the design rationales behind our clustering algorithm, visualize the space of the learned event vectors 2, cut and their location predictability score s(z). Figure 7 shows the t-SNE (Maaten Hinton, 2008) visualization of the event vectors on dimensional space. We color coded the events with three metrics: location predictability scores (Figure 7a), cluster ID discovered by our algorithm (Figure and ground truth label (Figure 7cp. The predictability score depends on how strongly an appliance event co-occurred with particular
We normalize each image hetter visualize locations with lower prediction confidence.
13

---

Published as a conference paper at ICLR 2020
location. As shown in Figure 7a most appliances related to human activities have high predictability scores (e.g., kettle, hairdryer, microwave, coffee machine, etc). On the other hand, appliances that eycle in the background (e.g., heater) have very low predictability. The stove has many clusters of background events. This is because when the stove is on, it cycles between a few power levels, and the cycle durations depend the heating levels. Interestingly, also have high predictability scores, while others with cycling states ("stove-cycle") show low scores. This likely because people are next the stove more often when the heating level is high.
found that stove clusters with
higher power levels ("stove-big-cycle
We can also see that without clustering using both location predictions and event vectors, it hard to separate some of the cluster boundaries. Besides, learning to relate energy events to location data enables us to measure the distances of events a well-defined physical space.
unlabelcd
kettle
hairdryer-1
coffee
25
micromave
daposer
heiddrye-22
-25
-2
-25
stave
-30
02
-50
-50
stave cycle
stove-tig-cyuu
-75
-15
heater
dahwasher
(a) Location predictability scores
(b) Discovered clusters
(c) Ground truth appliance labels
Figure 7: t-SNE visualization of the learned event vectors colored by (a) location predictability scores
(b) discovered clusters, and (c) ground truth labels.
8.4 NETWORK IMPLEMENTATION AND TRAINING DETAILS
In this section, we provide implementation and training details of our neural network model. We use convolution and deconvolution layers for the energy encoder and decoder. Each module has 8 layers with kernel size of 3 and a stride of 2. We choose the dimensions of 21,eat and ct,c to be 128 and 3. The location predictors have 5 layers of 3D deconvolution with a kernel size of 3 and a stride of 2 each dimension. The frames of location images for each time window have 32 32 x 32 pixels. We discretize the x. y, and time dimensions into 32 points, where the range of the x-y dimensions are 10 meters and the time dimension has 32 seconds. The neural networks are implemented in Tensorflow (Abadi al. 2016). For training, use the Adam (Kingma & Ba 2014) optimizer with a learning rate of 0.001 and a batch size 64.
8.5 CLUSTERING PARAMETERS AND DETAILS
In all experiments, we set 1/ = 0.4 meters, 7. = 0.03, 7, = 0.2, and Nmin 10. These values are chosen based on physical and computational constraints. The value of 1DD. based on the minimum physical separation between two appliances. The value of only affects the search space in each iteration, and is chosen be small for computational efficiency. The minimum predictability score 7, is chosen based on a validation set from one of the homes. The value of Nmin is set to 10 to say that we need the appliance appear in the data at least times before we trust that a real appliance.
8.6 LIMITATIONS
We discuss the limitations of our approach in this section. One limitation is that some remotely activated appliances may not have predictable locations. However, from our experience collecting the dataset, the vast majority of the appliances used on a daily basis (Table 2 require human interaction. For example, person has put food into microwave before turning it on, to hold hair dryer while drying hair, and to push a button to get a coffee machine running. Even for an appliance with a remote controller, long as the person has a regular place to interact with the appliance from (c.g.. always turning the TV on while sitting on the couch), our model can still learn to predict the location of interaction. Another limitation is that our location sensor has limited coverage area (around 40 feet in radius). This is enough to cover typical one-bedroom apartment. For a larger house, one could deploy a second sensor, similarly to how a WiFi repeater extends the coverage area.
14


Reviews:
[{"review_id": "SygwxOtM5H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Review ICLR20, Self-Supervised Learning of Appliance Usage\n\nThis review is for the originally uploaded version of this article. Comments from other reviewers and revisions have deliberately not been taken into account. After publishing this review, this reviewer will participate in the forum discussion and help the authors improve the paper.\n\n\n## Overall\n\n**Summary**\n\nThe authors introduce a new method for classifying which appliance was turned on by looking at the change in total household electricity consumption and tracking people's coarse positions in the house.\n\n\n**Overall Opinion**\n\nThe paper is concise and interesting, but for now, I have to reject it because the same major things aren't clear to me:\n\n- Why are you doing this, to begin with, i.e. \"... to optimize energy consumption\"?. What does energy consumption have to do with classifying which appliance was turned on? There might be the rare edge case of a fridge door left open accidentally but that can't be the main argument here, can it?\n- Now, assuming that there is a benefit to this, why can't we use the smart plugs all around? You already use them to provide ground truth. But I imagine they are cheap and easy enough to distribute at large scale, whereas the location sensor was custom-built by you if I understood correctly and probably has to be calibrated for every house.\n\nOther than that, the paper is pretty good. Here are some minor...\n\n## Specific comments and questions\n\n- I got slightly disappointed when I picked up the paper because I was in a robotics mindset and assumed \"Self-supervised learning of appliance usage\" was some new way for a robot to analyze appliances and push buttons. I'd add a \"human\" in the title for clarity, e.g. \"Self-supervised Learning of Human Appliance Usage\" or \"Self-supervised Learning of Appliance Usage from Human Location Data\". But that's just my opinion, no pressure on this one.\n\n### Abstract\n\n- Please never repeat sentences from the abstract in the introduction or vice versa, good as they might be.\n\n### Intro & Rel. Work\n\nall good\n\n### Problem Formulation\n\n- Please always provide a proper caption for your figure so that the caption contains enough information for the reader to understand what's going on in the figure without having to search for it in the main text. This applies to Fig.1 and even more so to Fig.2 where I'd recommend explaining in broad strokes the information flow.\n- Fig. 1 (a) Why is there higher power usage between 02-06 compared to 14-18 o'clock? Is that a heater as the outside temperatures drop overnight? (c/d) I don't think that's a good way of visualizing this. I don't get anything out of diagram (c) and I'd prefer it if you could encode the time aspect in the diagram (d) as color/alpha. E.g. blue agent's oldest position is almost imperceptibly white, growing bluer and darker as the data points get closer to the final step in the recording.\n\n### Model\n\n- Doesn't the softwax require to know the number of appliances a priori?\n- How do you deal with $\\alpha$ and $P_t$ when a person is at the edge of the visible space? Does this cause problems during training or did this never occur? I could imagine, since the location is noisy, a person could jump in and out of the visible area and cause training instability.\n\n### Datasets\n\nall good\n\n### Results\n\n- What's \"Door / leaving\" in Table 2? Which appliance is this?\n- I think the standard at ICLR is to have the table and figure captions both below the respective objects\n- The last 3 sections, 6.3 to 6.5, read a bit rushed and might need a rewrite for clarity. \n- Fig. 5 is nice. But shouldn't the signal add up to explain the entire scene? There's a bit of blur next to the couch. Is this a training set vs test set artifact?\n\n### Conclusion\n\n- This needs to be developed a bit more: what are some downsides to your method? Where does it NOT work or what makes it prone to error? And what are future directions based on this work? What's worth researching further or have you solved appliance use detection for good?\n\n---\n\nI'm sure there are some good answers for my main concerns outlined at the beginning and after that, I'd be happy to adjust my rating. The rest are minor issues. Like I mentioned, save for page 8, the paper is well-written and focused.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxczJKYtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors proposed a multi-modal unsupervised algorithm to uncover the electricity usage of different appliances in a home. The detection of appliance was done by using both combined electricity consumption data and user location data from sensors. The unit of detection was set to be a 25-second window centered around any electricity usage spike. Authors used a encoder/decode set up to model two different factors of usage: type of appliance and variety within the same appliance. This part of the model was trained by predicting actual consumption. Then only the type of appliance was used to predict the location of people in the house, which was also factored into appliance related and unrelated factors. Locations are represented as images to avoid complicated modeling of multiple people.\n\nAs the final step, a customized clustering algorithm was used to turn appliance related events into clusters that each represent an appliance.\n\nAuthors trained the model with real world data collected from 4 homes over a few months each. Smart plugs and some human labeling served as ground truth. The results showed good performance of the proposed unsupervised algorithm in recovering the actual appliances in the homes.\n\nReviewer acknowledge the difficulty in collecting such data and the careful analysis into data characteristics which led to the algorithm design. Reviewer also found the illustrations (esp Fig 1) quite intuitive for understanding the problem. Proposed algorithm is quite reasonable. The release of code and data could also be helpful to the community.\n\nReviewer also saw some non-trivial issues that Authors may need to fix or explain:\n\n1) Baseline setup seems to be too weak. Authors didn't explain how EL-KMeans was conducted which is non-trivial. The given citation (Arthur & Vassilvitskii, 2007) is a very generic paper discussing KMeans++, not EL-KMeans. In Reviewer's opinion this needs to be fixed for acceptance.\n\n2) in Section 6.2 Ablation study, Author mentioned \"we adopt a density propagation algorithm that only uses local neighborhood distance\" but Reviewer didn't find any content related to the algorithm. Is this the cluster algorithm?\n\nOther issues:\n\n1) In table 2,  What happened to Home 4's hair dryer and rice cooker? Their F1 are as low as 1.1.\n\n2)  The data size is quite small, which makes the conclusion less strong.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BygE9mFVqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a learning algorithm to recover the events of using an appliance and as well as the location of the appliance in a home by using smart electricity meter and a motion sensor installed a home. In the model, the input is a window of electricity energy consumption and context and the output of the model is the location collected by the motion sensor. The appliance activation as the latent variables is learned using a autoencoder architecture. \nThis work is very interesting and new to the energy efficiency community and non-intrusive load monitoring. This provide an alternative approach to understand how energy was used in a house and so could be used potentially for energy efficiency and malfunction detection of appliances. The data is also new to the community. The paper is well written and the experiments are clear to me. I have some more detailed concerns:\n1) One thing is surprising to me: in the autoencoder, you want to learning the latent variables {Z_t, Z_{t,cat}}. It is surprising that for example Z_{t,cat} was exactly the appliance category. I suppose it was aided by the location information presented in P(l|y,c)? I had an experience training an autoencoder for energy disaggregation, but it never worked well because the latent variables Z could be arbitrary.\n2) It would be more readable if you the model was provided with more details, for example, the detailed model for L_e(Z_{t,cat};\\theta_{L_e}). Although p_{\\theta_{L_g}} and L_g were defined in a similar way, they should be explicitly given.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors proposed a multi-modal unsupervised algorithm to uncover the electricity usage of different appliances in a home. The detection of appliance was done by using both combined electricity consumption data and user location data from sensors. The unit of detection was set to be a 25-second window centered around any electricity usage spike. Authors used a encoder/decode set up to model two different factors of usage: type of appliance and variety within the same appliance. This part of the model was trained by predicting actual consumption. Then only the type of appliance was used to predict the location of people in the house, which was also factored into appliance related and unrelated factors. Locations are represented as images to avoid complicated modeling of multiple people.

The reviewers were satisfied with the discussion after the authors, and therefore believe this work is of general interest to the ICLR community.
Decision: Accept (Poster)