Self-Supervised Learning of Appliance Usage

1 INTRODUCTION . Learning home appliance usage patterns is useful for understanding user habits and optimizing electricity consumption . For example , knowing when a person uses their microwave , stove , oven , coffee machine or toaster provides information about their eating patterns . Similarly , understanding when they use their TV , air-conditioner , or washer and dryer provides knowledge of their behavior and habits . Such information can be used to encourage energy saving by optimizing appliance usage ( Armel et al. , 2013 ) , to track the wellbeing of elderly living alone ( Donini et al. , 2013 ; Debes et al. , 2016 ) , or to provide users with behavioral analytics ( Zhou & Yang , 2016 ; Zipperer et al. , 2013 ) . This data is also useful for various businesses such as home insurance companies interested in assessing accident risks and utility companies interested in optimizing energy efficiency ( Armel et al. , 2013 ) . The problem can be modeled as event detection – i.e. , given the total energy consumed by the house as a function of time , we want to detect when various appliances are turned on . Past work has looked at analyzing the energy signal from the home utility meter to detect when certain appliances are on.1 Most solutions , however , assume that the energy pattern for each appliance is unique and known , and use this knowledge to create labeled data for their supervised models . ( Kolter et al. , 2010 ; Zhong et al. , 2014 ; 2015 ; Kelly & Knottenbelt , 2015 ; Zhang et al. , 2018 ; Bonfigli et al. , 2018 ) . Unfortunately , such solutions do not generalize well because the energy pattern of an appliance depends on its brand and can differ from one home to another ( Kelly & Knottenbelt , 2015 ; Bonfigli et al. , 2018 ) .2 The literature also contains some unsupervised methods , but they typically have limited accuracy ( Kim et al. , 2011 ; Kolter & Jaakkola , 2012 ; Johnson & Willsky , 2013 ; Parson et al. , 2014 ; Wytock & Kolter , 2014 ; Zhao et al. , 2016 ; Lange & Berges , 2018 ) . Unsupervised event detection in a data stream is intrinsically challenging because we do not know what patterns to look for . In our task , not only may appliance energy patterns be unknown , but also the energy signal may include many background events unrelated to appliance activation , such as the fridge or HVAC power cycling events . One way to address this challenge is to consider the self-supervised paradigm . If a different stream of data also observes the events of interest , we can use this second modality to provide self-supervising 1The utility meter outputs the sum of the energy of all active appliances in a house as a function of time . 2For example , a Samsung dishwasher may have a different energy pattern from that of a Kenmore dishwasher . signals for event detection . To that end , we leverage the availability of new fine-resolution motion sensors which track the locations of people at home ( Adib et al. , 2015 ; Joshi et al. , 2015 ; Li et al. , 2016 ; Ghourchian et al. , 2017 ; Hsu et al. , 2017b ) . Such sensors operate as a consumer radar , providing decimeter-level location accuracy . They do not require people to wear sensors on their bodies , can operate through walls , and track people ’ s locations in different rooms . These location sensors indirectly observe the events of interest . Specifically , they capture the change in user locations as they reach out to an appliance to set it up or turn it on ( e.g . put food in a microwave and turn it on ) . Hence , the output of such sensors can provide a second modality for self-supervision . But how should one design the model ? We can not directly use location as a label for appliance activation events . People can be next to an appliance but neither activate it nor interact with it . Moreover , we do not assume appliance locations are known a priori . We also can not use the two modalities to learn a joint representation of the event in a shared space . This is because location and energy are unrelated most of the time and become related only when the event of interest occurs . Furthermore , there are typically multiple residents in the home , making it hard to tell which of them interacted with the appliance . Our model is based on cross-modal prediction . We train a neural network that , given the home energy at a particular time , predicts the location of the home residents . Our intuition is that appliance activation events have highly predictable locations , typically the location of the appliance . In contrast , background energy events ( e.g . power cycling of the fridge ) do not lead to predictable locations . Thus , our model uses this learned predictability along with the associated location and energy representation to cluster the events in the energy stream . In addition , we use a mixture distribution to disentangle irrelevant location information of other residents in the home . Interestingly , our model not only learns when each appliance is activated but also discovers the location of that appliance in the home , all without any labeled data . We summarize the contributions of this paper as follows : • The paper introduces a new method for self-supervised event detection from weakly related data streams . The method combines neural cross-modal prediction with custom clustering based on the learned predictability and representation . We apply it to the task of detecting appliance usage events using unlabeled data from two sensors in the home : the energy meter , and a location sensor . • To evaluate our design , we have created the first dataset with concurrent streams of home energy and location data , collected from 4 homes over a period of 7 months . For each home , data was collected for 2 to 4 months . Ground truth measurements are provided via smart plugs connected directly to each appliance . • Compared to past work on unsupervised learning of appliance usage and a new baseline that leverages the two modalities , our method achieves significant improvements of 67.3 % and 51.9 % respectively for the average detection F1 score . We will release our code and dataset to encourage future work on multi-modal models for understanding appliance usage patterns and the underlying user behavior . 3 2 RELATED WORK . Energy disaggregation Our work is related to past work on energy disaggregation , which refers to the problem of separating appliance-level energy from a home ’ s total ( or aggregate ) energy signal . Past work in this domain can be broadly classified into two categories : supervised and unsupervised . Supervised methods assume that the power signatures of individual appliances are available . They use data from individual appliances to obtain models for each appliance power signature , and then use those models to detect appliance events from the aggregate energy signal . Early work learns sparse codes for different appliances ( Kolter et al. , 2010 ) or uses a Factorial HMM ( FHMM ) ( Ghahramani & Jordan , 1996 ) to model each appliance as an HMM ( Zhong et al. , 2014 ; 2015 ) . Other work uses matrix factorization approaches to estimate monthly energy breakdowns ( Batra et al. , 2017 ; 2018 ) . More recently , neural networks have been used to model appliances ( Kelly & Knottenbelt , 2015 ; Zhang et al. , 2018 ; Jia et al. , 2019 ; Bonfigli et al. , 2018 ) , where extracting appliance-level energy is formulated as a de-noising problem . However , supervised solutions typically do not generalize well 3Project website : http : //sapple.csail.mit.edu to new homes ( Kelly & Knottenbelt , 2015 ; Bonfigli et al. , 2018 ) . This is because two appliances of the same type ( e.g . coffee machine ) in different homes are often manufactured by different brands , and thus have different power signatures . Unsupervised methods do not assume prior knowledge of appliance signatures ; they attempt to learn those signatures from the aggregate energy signal . Early approaches use variants of FHMM , and learn appliance HMMs with Expectation-Maximization ( Kim et al. , 2011 ) , approximate footprint extraction procedures ( Kolter & Jaakkola , 2012 ) , or using expert knowledge to configure prior parameters ( Johnson & Willsky , 2013 ; Parson et al. , 2014 ) . Some papers propose using contextual information ( such as temperature , hour of the day , and day of the week ) ( Wytock & Kolter , 2014 ) , or use event-based signal processing methods to cluster appliances ( Zhao et al. , 2016 ) . More recently , Lange & Berges ( 2018 ) proposed using a recurrent neural network as the variational distribution in learning the FHMM . In contrast , our work leverages people ’ s location data as a self-supervising signal . We cluster appliance events through learning the relation between energy events and people ’ s locations , and also learn appliance locations as a by-product . Passive location sensing Motivated by new in-home applications and continuous health monitoring , recent years have witnessed an increasing number of indoor location sensing systems ( Adib et al. , 2015 ; Joshi et al. , 2015 ; Li et al. , 2016 ; Ghourchian et al. , 2017 ) . They infer people ’ s locations passively by analyzing how people change the surrounding radio signals ( e.g . WiFi ) and do not require people to wear any sensors . These sensors have been used for various applications including activity recognition ( Wang et al. , 2014 ; 2015 ) , sleep monitoring ( Zhao et al. , 2017 ; Hsu et al. , 2017a ) , mobility and behavioral sensing ( Hsu et al. , 2017b ; 2019 ) , and health monitoring ( Kaltiokallio et al. , 2012 ) . In our work , we leverage the availability of such sensors to introduce location data as an additional data modality for learning appliance usage patterns . Self-supervised multi-modal learning Our work is related to a growing body of work on multimodal learning . Most approaches learn to encode the multi-modal data into a shared space ( Gomez et al. , 2017 ; Harwath et al. , 2018 ; Owens & Efros , 2018 ; Zhao et al. , 2018 ; 2019 ) . In contrast , since our two modalities are mostly unrelated and become related only when an activation event happens , we learn to predict one modality conditioned on the other . Our work is also related to cross-modal prediction ( Krishna et al. , 2017 ; Owens et al. , 2016 ; Zhang et al. , 2017 ) but differs from it in an essential way . Past work on cross-modal prediction typically uses the prediction as the target outcome ( e.g . output text for video captioning ) . In contrast , our objective is to discover the hidden appliance activation events . Thus , we design our method to leverage the learned predictability and cross-modal mapping for clustering activation events . Furthermore , we introduce a mixture prediction design to disentangle unrelated information in our predicted modality ( location measurements unrelated to energy events ) . 3 PROBLEM FORMULATION . Our goal is to learn appliance activation events in an unsupervised way , using two input streams : home aggregate energy and residents ’ location data . Figure 1 shows the two data modalities . We describe each of them formally and define appliance “ events ” below . Aggregate energy signal A household ’ s total energy consumption is measured by a utility meter regularly . This measures the sum of the energy consumed by all appliances at each point in time . We denote the aggregate energy signal by y = ( y1 , y2 , . . . , yT ) , where yt ∈ R+ . Suppose there are a total of K appliances in a home , and each appliance ’ s energy signal is denoted by xk = ( x1 , k , x2 , k , . . . , xT , k ) , where xt , k ∈ R+ . Only the aggregate energy signal is observed , yt =∑K k=1 xt , k + t , where t ∼ N ( 0 , σ2 ) is the background noise . Figure 1a shows one day of an aggregate energy signal . The base power level shifts constantly throughout the day , depending on the background load ( e.g . ceiling lights ) . Added on top of the base level are the various appliance events . Figure 1b zooms in around 20:30 , and shows examples of those events . The stove was turned on around 20:28 , and its power continued to cycle between a few levels . While the stove was on , the microwave was also turned on and ran for a few minutes , and the garbage disposer was turned on shortly . Indoor location data We use a single location sensor similar to that in Hsu et al . ( 2017b ) to measure people ’ s indoor locations passively . The sensor sends out radio signals and analyzes the reflections to localize multiple people . Similarly to a regular WiFi router , the sensor has a limited coverage area of up to 40 feet . Suppose there are Pt people in the coverage area at time t. The location data is denoted by lt = ( lt,1 , lt,2 , . . . , lt , Pt ) , where lt , p ∈ R2 is the x-y location of person p at time t. We can represent the location data over multiple time frames as l1 : T = ( l1 , l2 , . . . , lT ) . Figure 1c shows one minute of location data from two people , and Figure 1d shows the data from a top-down view . Appliance activation events When an appliance is turned on , it causes a jump in energy consumption , i.e . a leading edge in the energy signal , as shown in Figure 1b . We call such a pattern an appliance activation event . On the other hand , when an appliance changes its internal state , it can also cause a change in the energy signal as shown in the same figure . We call such a pattern a background event . We are interested in discovering activation events to learn appliance usage patterns . Thus , for each jump in the aggregate signal , we take a time window ( of 25 seconds ) centered around that jump , and analyze it to detect whether it is an activation event and which appliance it corresponds to .

Summary: Authors proposed a multi-modal unsupervised algorithm to uncover the electricity usage of different appliances in a home. The detection of appliance was done by using both combined electricity consumption data and user location data from sensors. The unit of detection was set to be a 25-second window centered around any electricity usage spike. Authors used a encoder/decode set up to model two different factors of usage: type of appliance and variety within the same appliance. This part of the model was trained by predicting actual consumption. Then only the type of appliance was used to predict the location of people in the house, which was also factored into appliance related and unrelated factors. Locations are represented as images to avoid complicated modeling of multiple people.