SlowGuess/NeurIPS_2023_OCR huggingface dataset:
WYZE Rule: Federated Rule Dataset for Rule Recommendation Benchmarking
Wyze Labs, Inc.  
mkamani@wyze.com
Hanjia Lyu
University of Rochester hlyu5@ur. rochester.edu
Zhongwei Cheng
Wyze Labs, Inc.  
zcheng@wyze.com
Wyze Labs, Inc.  
lchen@wyze.com
Liangju Li
Wyze Labs, Inc.  
lli@wyze.com
Carnegie Mellon University cjoewong@andrew.cmu.edu
Jiebo Luo
University of Rochester jluo@cs.rochester.edu
In the rapidly evolving landscape of smart home automation, the potential of IoT devices is vast. In this realm, rules are the main tool utilized for this automation, which are predefined conditions or triggers that establish connections between devices, enabling seamless automation of specific processes. However, one significant challenge researchers face is the lack of comprehensive datasets to explore and advance the field of smart home rule recommendations. These datasets are essential for developing and evaluating intelligent algorithms that can effectively recommend rules for automating processes while preserving the privacy of the users, as it involves personal information about users' daily lives. To bridge this gap, we present the Wyze Rule Dataset, a large-scale dataset designed specifically for smart home rule recommendation research. Wyze Rule encompasses over 1 million rules gathered from a diverse user base of 300,000 individuals from Wyze Labs, offering an extensive and varied collection of real-world data. With a focus on federated learning, our dataset is tailored to address the unique challenges of a cross-device federated learning setting in the recommendation domain, featuring a large-scale number of clients with widely heterogeneous data. To establish a benchmark for comparison and evaluation, we have meticulously implemented multiple baselines in both centralized and federated settings. Researchers can leverage these baselines to gauge the performance and effectiveness of their rule recommendation systems, driving advancements in the domain. The Wyze Rule Dataset is publicly accessible through HuggingFace's dataset API.
Given the rapid growth of smart devices and applications, there is an increasing need to automate the functionalities of various IoT devices and applications by establishing connections between them. The true value lies in their ability to perform a wide range of tasks and functionalities, tailored to the specific needs and preferences of different users. Configuring device automation to achieve personalized and efficient outcomes becomes a complex endeavor, requiring the assistance of AI and ML techniques. To enable seamless automation, rules serve a critical role. These rules act as predefined conditions or triggers that establish connections between IoT devices, allowing for
Figure 1: Wyze Rule Dataset - A large-scale dataset for smart home rule recommendation research. The dataset includes over 1 million rules generated by 300,000 users from Wyze Labs, offering a diverse and extensive collection of real-world data. The dataset facilitates the development and evaluation of personalized rule recommendation algorithms for automating smart home processes while respecting user privacy.
customized and automated processes. For example, a rule might dictate that when a motion sensor detects movement, the lights in the room should automatically turn on.
Recommendation systems play a vital role in simplifying, enhancing efficiency, and personalizing the automation process. By recommending rules to users based on their historical interactions with devices and the rules selected by others with similar preferences, the automation process becomes easier, more efficient, and tailored to the specific needs of each user. It is important to note that rule recommendation differs from classical recommendation systems, which primarily rely on a user-item matrix completion approach. In the realm of rule recommendation, four main components are involved: trigger device, trigger state, action device, and action. The main structure of this dataset is shown in Figure 1, where each trigger-action pair is represented as a rule denoted by  $r_i$ . The triggers and actions can vary depending on the nodes being connected, making the recommendation process more challenging. Specifically, for each pair of trigger and action devices, we aim to recommend an appropriate trigger state and action combination. However, in user-item matrix completion approaches, the complete rule consisting of all four components is treated as a single item. This approach is not able to distinguish between multiple instances of an item within a household, such as different cameras, leading to the recommendation of generic rules.
The mentioned limitations highlight the need for an efficient and personalized rule recommendation system that can consider the unique characteristics of the rule recommendation task. In addition to this, privacy becomes a critical concern as rule recommendation systems require access to user data to provide tailored recommendations. By developing an effective, personalized, and private rule recommendation system, the automation process can be accelerated, leading to higher adoption rates and improved user experience. It is worth mentioning that rule recommendation extends beyond IoT devices and finds applications in diverse automation processes, including software development, workflow automation, task management, smart assistants, and healthcare routine automation.
One of the major challenges faced by researchers in the field of smart home rule recommendations is the lack of comprehensive and scalable datasets. The closest existing dataset available for rule recommendation research is the IFTTT dataset [14]. This dataset primarily focuses on rules connecting various apps, such as Gmail and Instagram, for users. It consists of rules created by a subset of users and adopted by a larger user base within the IFTTT platform. The IFTTT dataset was initially extracted from the platform in September 2015 by Ur et al. [14] and subsequently explored by Corno et al. [3]. It comprises a total of 151 users, 156 devices, 500 triggers, 235 actions, and 4,015 known rules. Although the IFTTT dataset has garnered attention from researchers in the recommendation systems domain, its small scale limits its practicality for real-world applications. To overcome these limitations and bridge the gap between research and practical applications, there is a pressing need for a more extensive and representative dataset, which the Wyze Rule Dataset aims to fulfill.
Introducing the Wyze Rule Dataset, we present a large-scale and user-centric dataset specifically designed for smart home rule recommendation research. The Wyze Rule Dataset comprises an extensive collection of over 1 million rules, created by individual users to meet their unique needs within their smart homes. This dataset is sourced from Wyze Labs and encompasses a diverse user base of 300,000 individuals, providing a rich and varied pool of real-world data. Unlike the IFTTT
dataset, the Wyze Rule Dataset reflects the specific device configurations of each user, including the possibility of multiple instances for each device type. Each user's set of available and valid rules can differ significantly from others due to the distinct combinations of devices. This inherent variability adds a layer of complexity to rule recommendation, as each pair of devices may have its own set of valid rules that differs from those of other device pairs. For example, a motion sensor and a camera may have a valid trigger-action pair of "motion detected-turn on," which may not be applicable to a contact sensor and a thermostat.
The individualized nature of the rules generated by users, based on their unique device setups, presents an excellent opportunity to explore federated learning approaches [10, 6, 4, 5]. Federated learning enables the development of personalized recommendation models without the need to transfer rules and usage data to an external server. With the wide range of device combinations within each user's household and the resulting diversity of valid rules, the distribution of data among users in the Wyze Rule Dataset exhibits a high level of heterogeneity. Consequently, the dataset offers a compelling real-world use case for developing federated learning algorithms that effectively address the challenges posed by data heterogeneity among users.
The Wyze Rule Dataset is made available under the "CC BY-NC-ND" license, specifically for non-commercial and research-related purposes. To obtain access to the dataset, interested individuals are required to complete and sign the consent form, which can be accessed through the following link: Wyze Rule Recommendation Dataset. Once the consent form is submitted, the dataset will be released to the approved individuals, enabling them to utilize it for their research endeavors.
The majority of recommendation systems primarily rely on collaborative filtering techniques, such as matrix factorization approaches [9, 2], to match users with items of interest. These systems consider user and item features, as well as historical data reflecting user preferences for a given set of items. Recently, the adoption of graph neural networks has become prevalent in recommendation systems, particularly for modeling relationships using bipartite graph structures. One notable graph-based auto-encoder framework for matrix completion problems is GCMC [1]. It utilizes graph structures to provide recommendations. R-GCNs [12], on the other hand, leverage graph convolutional networks to model relational data by considering different relations between users and items. For dealing with heterogeneous graphs containing user and item features, HetGNN [19] has been proposed. Additionally, RotatE [13] introduces a seminal approach for knowledge graph embeddings by employing relational rotation in a complex number space.
With the advancements in smart home technology, rule recommendation systems for IoT devices have gained significant practical and research potential [18, 17]. IFTTT, a mobile app offering rule execution services, provides an open dataset of users and rules for analysis [14, 11]. However, this dataset primarily consists of popular rules created by some users and utilized by others, with fixed entities. In contrast, the Wyze Rule dataset consists of rules created by individual users to cater to their specific needs. The entities in this dataset can have multiple instances, such as a user having multiple different cameras. Furthermore, the scale of the Wyze Rule dataset is much larger than that of the IFTTT dataset. For rule recommendation in IFTTT, RecRule [3] presents a recommendation approach based on a user-rule matrix and employs semantic reasoning to enrich the rules with additional semantic information. TaKG [16], on the other hand, constructs a trigger-action knowledge graph and utilizes collaborative filtering between users and rules to recommend appropriate rules. GraphRule and FedRule, recently introduced in Yao et al. [17], introduce a user-centric graph structure where each user has their own graph. In this setting, the task of rule recommendation transforms into a link prediction problem.
3 Wyze Rule Dataset
3.1 Meta Data
The dataset consists of two files, rule.csv, and device.csv, which provide information about the rules that control the behavior of Wyze smart home devices and the devices owned by users, respectively. The structure of each of these two files is as follows:
- **Rule Dataset:** This file (rule.csv) contains data related to the rules that govern the behavior of Wyze smart home devices. Each row represents a single rule and contains various attributes describing the rule. The attributes of this file are as follows:
- 'user_id' (int): A unique integer identifier for the user associated with the rule. This identifier has been anonymized and does not contain any information related to the Wyze users.
- 'trigger_device' (str): The model of the device that triggers the rule when a specific condition is met. It may be a Wyze smart home device such as a sensor or a camera.
- 'trigger_device_id' (int): A unique integer identifier for the trigger device.
- 'trigger_state' (str): The state or condition that needs to be met on the trigger device for the rule to be activated. It may represent values such as "on," "off," "motion detected," or "sensor open."
- 'action_device'(str): The model of the device that performs an action when the rule is triggered. It is a Wyze smart home device such as a light or a camera.
- 'action_device_id' (int): A unique integer identifier for the action device.
- 'action' (str): The action to be executed on the action device when the rule is triggered. It may include values like "power on," "power off," "start recording," or "change brightness."
- Device Dataset: This file contains data related to the devices owned by users. Each row represents a single device and contains information about the device model and its association with a specific user. There are a number of devices in this dataset that are not used in any rules by users, and hence, are not present in the rule dataset. The attributes of this dataset are as follows:
- 'user_id' (int): A unique integer identifier for the user associated with the device.
- 'device_id'(int): A unique integer identifier for the device.
- 'device_model' (str): The model or type of the device owned by the user. It represents various Wyze smart home devices such as a camera, a sensor, or a switch.
This dataset comprises the rules that establish connections between smart devices in various clients' households. As a result, the distribution of rules among clients is non-IID, making it suitable for federated learning settings. The dataset, referred to as the "Wyze Rule" dataset, encompasses data from over 300,000 users with more than 1 million rules. To address privacy concerns, we have thoroughly cleansed the dataset, removing all personally identifiable information (PII).
The rules in the dataset have been simplified and follow a specific structure:  $"<$  trigger entity, trigger-action pairs, action entity  $>"$ . The trigger-action pairs represent the type of connection established between devices. For instance, an example rule could be connecting a smart doorbell to a camera, where pressing the doorbell triggers the camera to power on for recording. In this case, the rule format would be "<Doorbell, Doorbell Pressed - Power On, Camera>" The dataset encompasses 16 distinct device types and 1,323 unique trigger-action pairs, resulting in 2,968 unique rules. Table 1 provides a detailed breakdown of the entity types and samples of trigger-action pairs. It is worth noting that the entity "Cloud" represents an arbitrary device where the action involves modifying settings on the same device in the app. For instance, in the rule "When the Camera detects motion, turn on the notification," the action device would be classified as "Cloud" since it changes the setting of that device in the app.
As it was mentioned, the data in this dataset is heavily non-IID among users. As evidence, in Figure 2 the distribution of the number of rules and devices among users can be seen. The distribution is a long-tailed one, indicating, the heterogeneity of data distribution among users, making this real-world dataset a perfect use case for federated learning algorithms. In the dataset, there are some rare users, with more than 200 rules, which for the sake of visualization, we did not include in the distribution in Figure 2a. Note that the Cloud only appears in the action device list.
The data heterogeneity can also be seen from the distribution of device types owned by users, as well as utilized by rules set by users. The share of device types owned by users can be seen in Figure 3a, in
Types of Entities	Types of Trigger-Action Pairs
Camera	Open, Turn On
Climate Sensor	Open, Turn Off
Contact Sensor	Open, Turn on motion detection
Irrigation	Open, Change Brightness
Leak Sensor	...
Light	Detects a person, Unmute notification
Light Strip	Detects a pet, Unmute notification
Lock	Detects a vehicle, Unmute notification
Mesh Light	...
Motion Sensor	Open, Turn on the siren
Outdoor Plug	Open, Turn off the siren
Plug	...
Robot Vacuum	Detects a person, Turn off
Switch	Detects a package, Turn on
Thermostat	Detects smoke alarm sound, Turn off
Cloud	(Doorbell) Is pressed, Turn On
Table 1: Wyze Rule Dataset: Device list and trigger-action pairs samples.
(b)
Figure 2: The distribution of the number of Rules and Devices among users in Wyze Rule Dataset.
which, Camera is the most owned device among users, followed by Plug and Light. The distribution of device types utilized by rules is fairly similar, visible in Figure 3b. As it is expected, and can be seen from Figures 3c and 3d, after Camera, sensors (contact sensor and motion sensor) are the most used devices as triggers by the users. Light and plug are the most used action devices after Camera and Cloud in the rules.
To evaluate models trained on this dataset, similar to other recommendation models, we use various offline metrics. These metrics are mainly designed to evaluate the ability of the model to predict omitted rules from the database with a higher probability. The metrics are defined as follows:
AUC The Area Under the ROC (Receiver Operating Characteristic) curve represents the overall performance of the rule recommendation model. It is a widely-used metric in machine learning evaluation. A higher AUC indicates a better ability of the model to rank rules correctly, distinguishing positive and negative recommendations effectively.
Positive Mean Rank The positive mean rank is a metric that quantifies the average position or rank of the ground truth rules within the list of recommended rules over all users' data. It specifically focuses on the rank of the positive rules, which are the rules that should ideally be included in the recommendations. A lower positive mean rank indicates better performance, as it signifies that the ground truth rules are ranked closer to the top of the recommendation list. Improving the positive
Share of devices used for rules as a trigger device
(c)
Share of devices used for rules as an action device
Figure 3: Share of different device types among users and utilized by rules either as a trigger device or action device.
mean rank metric helps enhance the overall user experience by ensuring that the most important and useful rules are presented prominently in the recommendation results.
Mean Reciprocal Rank Mean Reciprocal Rank (MRR) is a metric commonly used to evaluate the performance of a recommendation system. It measures the average of the reciprocal ranks of the ground truth rules within the recommendation list. MRR takes into account the position or rank of the first correctly recommended rule, considering a lower rank as more desirable. We average this measure over all users' data:
$$
\mathrm {M R R} = \frac {1}{M} \sum_ {m \in [ M ]} \frac {1}{| \mathcal {T} _ {m} |} \sum_ {i \in \mathcal {T} _ {m}} \frac {1}{\operatorname {r a n k} _ {i}}, \tag {1}
$$
where  $\mathcal{T}_m$  represents the set of ground truth rules that are omitted from the data of user  $m$ . By considering the reciprocal rank, MRR encourages recommendation systems to prioritize the most relevant rules and rank them higher in the recommendation list. This aligns more closely with user preferences and improves the overall user experience by increasing the chances of important rules being discovered earlier.
Weighted Mean Reciprocal Rank Weighted Mean Reciprocal Rank (WMRR) is an extension of the Mean Reciprocal Rank (MRR) metric that incorporates the notion of weights assigned to different ground truth rules. In this case, for each two nodes, the number of valid rules is different. Hence, by using this number as a weight in MRR, we can distinguish between the recommendations that have the same rank but over very different size of available valid rules.
$$
\mathrm {W M R R} = \frac {1}{M} \sum_ {m \in [ M ]} \frac {1}{\sum_ {i \in \mathcal {T} _ {m}} w _ {i}} \sum_ {i \in \mathcal {T} _ {m}} \frac {w _ {i}}{\operatorname {r a n k} _ {i}}, \tag {2}
$$
where  $w_{i}$  is the number of valid rules between two nodes of the  $i$ -th rule in the omitted ground truth rule set  $\mathcal{T}_m$ .
To demonstrate the potential of this dataset, we employ it in various training procedures aimed at real-world scenarios of rule recommendation. We utilize the defined metrics to assess the performance of these trained models on the dataset and subsequently explore the possibilities and opportunities associated with using this dataset for such tasks.
We investigate two training modes: centralized learning and federated learning, aiming to demonstrate the dataset's potential for graph representation learning in the federated learning domain. As discussed in this paper and by Yao et al. [17], we have two options: treating each rule as an individual item and applying user-item matrix completion approaches (similar to conventional recommendation systems), or constructing a graph for each user and conducting link prediction tasks. In this study, we mainly focus on graph representation learning, using the same settings as described by Yao et al. [17]. For training, we primarily utilize the Adam optimizer with a learning rate of 0.1 and train for 100 epochs. In the case of federated learning algorithms, we adopt 3 local steps during each communication round, totaling 300 local iterations over 100 communication rounds.
4.2 Centralized Benchmarks
We conduct a comparison between three graph representation learning models using the proposed GraphRule [17], a centralized optimization for graph-based link prediction of rule recommendation. It is important to note that due to the different task structures, direct comparison between graph representation learning approaches and user-item matrix completion methods is challenging. In the matrix completion approaches, each rule is treated as an individual item, irrespective of the trigger and action nodes involved. This means that two rules with the same trigger and action, but associated with different sets of motion sensors and cameras within a house, would be considered as one item in matrix completion approaches. However, in graph-based approaches, these rules would be represented as separate edges, distinguishing them based on the specific combinations of sensors and cameras.
Hit Rate @50	Hit Rate @50
(Multiple Devices with Same Type)
Matrix Factorization	0.8138	NA
GCMC	0.8445	NA
GraphRule	0.91	0.6396
Table 2: Hit Rate for different baselines. Classical user-item matrix completion approaches cannot distinguish between multiple devices with the same type in rules for a user.
As shown in Table 2, when using user-item completion (Matrix Factorization and GCMC) in the rule recommendation setting, since there is only one user-item relationship (user, select, rule item) while they cannot deal with multiple devices with the same type, which makes the performance of user-item completion method very limited. When recommending 50 rule, the hit rate of Matrix Factorization and GCMC become 0 since it only does not consider multiple devices with the same type.
For the GraphRule [17] approach we use three different models as its GNN representation model. The models are: GraphSAGE [7], Graph Attention Network (GAT) [15], and Graph Convolutional Network (GCN) [8]. For each of these models, we use two layers of neural networks with a hidden layer size of 32. The input size is 16, which is the one-hot-encoding size of the device types. The output size is 1, 323, which is the number of unique trigger-action pairs in the dataset. For the GAT model, we use 5 heads. We run the training for 100 epochs. Figure 4 shows the results of applying the GraphRule approach to the Wyze Rule Dataset. We provide a comparison between different GNN models on this approach on all different metrics we introduced in 3.3. GraphSAGE consistently outperforms the other models across various metrics, indicating its superior performance. It is worth noting that while there is a significant disparity in the MRR metric, weighted MRR offers a more accurate comparison and demonstrates their competitiveness.