SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(a) Test AUC
(d) Test MRR
Figure 4: Test results of applying GraphRule algorithms with different GNN models for node representation on Wyze Rule Dataset.
(b) Test Mean Rank
(e) Test Weighted MRR
4.3 Federated Benchmarks
To demonstrate the potential of the Wyze Rule Dataset in the realm of federated learning algorithms, we employ the proposed FedRule algorithm introduced by Yao et al. [17]. Building upon the GraphSAGE model utilized in the previous step, FedRule leverages a federated learning framework where each user independently updates the model on their local data and subsequently transmits the updated models to a central server for aggregation. The Wyze Rule Dataset presents a unique challenge for federated learning due to the high heterogeneity in the data distribution and variations in data sizes across different users. This heterogeneity poses a significant obstacle to achieving comparable performance to centralized approaches in federated learning. However, addressing this challenge is crucial as federated learning techniques offer the potential to preserve user privacy while achieving comparable performance to centralized approaches.
(a) Test AUC
Figure 5: Test results of applying FedRule algorithms with GraphSAGE models for node representation on Wyze Rule Dataset.
(b) Test Loss
(c) Test Mean Rank
Figure 5 illustrates the performance dynamics of the local models and the global model in the federated learning process. Due to the significant heterogeneity among clients' data, the local models exhibit a ripple effect with high variance. In contrast, the global model, which is obtained through the aggregation of the local models, demonstrates a smooth convergence. The comparison of Mean rank values between the FedRule approach and the centralized training provides valuable insights. After 100 rounds of communication, the Mean rank on FedRule is approximately 30, which is nearly twice the value observed in the centralized training. This substantial difference highlights the existence of a noticeable gap between the two modes of training. The observations derived from these results emphasize the importance of developing a specialized federated learning approach that can effectively address the high heterogeneity in the data distribution within the Wyze Rule dataset. Such an approach should aim to mitigate the impact of data heterogeneity and strive to achieve performance levels that are closer to those obtained in centralized training scenarios.
4.4 Evaluation against the Prior Version of the Wyze Rule Dataset
The initial introduction of the Wyze Rule dataset took place in the FedRule paper [17], where the novel GraphRule and FedRule algorithms were proposed, outperforming traditional recommendation techniques. While the GraphRule, incorporating a 2-layer GraphSage model paired with a 2-layer Neural Network predictor, exhibits excellent performance on smaller datasets, its application to our newly released, more expansive dataset poses certain challenges. It's noteworthy to mention that, despite these challenges, GraphRule continues to surpass other methodologies on this new dataset. Nevertheless, the overall efficacy of all methods seems to diminish due to the intricate nature of this new large-scale dataset.
Challenge 1: The Necessity for Enhanced Centralized Rule Recommendation Algorithms Applying the existing model to the expansive cross-device dataset results in a noticeable decrease in the Mean Rank, which is concerning given the application of recommendation for this dataset.
Test Loss	AUC	Mean Rank
GraphRule (small dataset)	0.1997	0.9768	4.349
GraphRule (large dataset)	0.1911	0.9825	10.46
There's a clear need for more refined algorithms, prompting our interest in hosting a challenge<sup>1</sup> centered around this dataset. An additional observation suggests that focusing on methods that optimize the mean rank directly, rather than merely addressing test loss, could prove beneficial.
Challenge 2: Addressing Cross-device Federated Learning with Scarce Data and High Data Heterogeneity
In our previous research [17], when employing FedGate as the training method, FedRule achieved performance equivalent to centralized training. However, using FedAvg as the training method caused FedRule not to converge. Upon deploying on the larger dataset, FedRule, regardless of whether it employs FedAvg or FedGate, can converge. Yet, it fails to match the performance of centralized training. This indicates that both FedAvg and FedGate struggle in such cross-device contexts.
Table 3: Performance comparison of the GraphRule model on small and large datasets, showcasing the variation in Test Loss, AUC, and Mean Rank values.
Test Loss	AUC	Mean Rank
FedRule (small dataset, with FedAvg)	0.3878	0.9521	6.661
FedRule (small dataset, with FedGate)	0.1892	0.9804	4.156
FedRule (large dataset, with FedAvg)	0.2854	0.9613	30.92
FedRule (large dataset, with FedGate)	0.2907	0.9486	37.99
Table 4: Performance metrics of the FedRule model using different training methods (FedAvg and FedGate) across small and large datasets, highlighting the differences in Test Loss, AUC, and Mean Rank.
In summary, the larger dataset illuminates the path for crafting superior centralized rule recommendation algorithms, formulating novel Federated Learning algorithms for cross-device scenarios, and jointly enhancing both areas.
The Wyze Rule Dataset presents exciting research opportunities in the field of rule recommendation. By leveraging the rich and diverse set of rules created by individual users, researchers can develop and evaluate novel recommendation algorithms that cater to the specific needs and preferences of smart home users. In addition, the availability of this dataset introduces a novel evaluation opportunity for
<sup>1</sup>The Wyze Rule Recommendation Challenge is taking place on the HuggingFace website from August 15, 2023, to November 10, 2023. For details, visit: https://huggingface.co/spaces/competitions/wyze-rule-recommendation.
federated learning algorithms, as it inherently exhibits non-IID characteristics. This dataset enables the exploration of advanced techniques for personalized rule recommendations, taking into account the heterogeneous device configurations and unique user contexts present in real-world smart homes. The Wyze Rule Dataset also opens up opportunities for investigating 'node recommendations', which involves suggesting suitable devices or sensors for specific automation tasks. By analyzing the relationships between devices, triggers, and actions in the dataset, researchers can develop algorithms that recommend the most appropriate nodes to users based on their specific requirements. This can help users make informed decisions when expanding their smart home ecosystems and further improve the automation capabilities of their setups.
One limitation of this dataset is that the location of each device is not indicated, hence, we do not know which devices are co-located together (i.e., in the same room or in the same house). This will make the recommendation more difficult since it might recommend valid rules that are not useful. Another limitation worth considering is the absence of direct human feedback or online comparison data within the dataset. While the Wyze Rule Dataset provides valuable information about the rules created by users, it does not include explicit feedback on the effectiveness or satisfaction of the recommended rules. Incorporating user feedback and conducting online comparisons could provide deeper insights into real-world performance and user preferences regarding rule recommendations.
[1] Rianne van den Berg, Thomas N Kipf, and Max Welling. Graph convolutional matrix completion. arXiv preprint arXiv:1706.02263, 2017.
[2] Dheeraj Bokde, Sheetal Girase, and Debajyoti Mukhopadhyay. Matrix factorization model in collaborative filtering algorithms: A survey. Procedia Computer Science, 49:136-146, 2015.
[3] Fulvio Corno, Luigi De Russis, and Alberto Monge Roffarello. Recrules: recommending if-then rules for end-user development. ACM Transactions on Intelligent Systems and Technology (TIST), 10(5):1-27, 2019.
[4] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461, 2020.
[5] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Distributionally robust federated averaging. Advances in Neural Information Processing Systems, 33:15111-15122, 2020.
[6] Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In International Conference on Artificial Intelligence and Statistics, pages 2350-2358. PMLR, 2021.
[7] William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 1025-1035, 2017.
[8] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
[9] Yehuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
[10] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
[11] Xianghang Mi, Feng Qian, Ying Zhang, and XiaoFeng Wang. An empirical characterization of ifttt: ecosystem, usage, and performance. In Proceedings of the 2017 Internet Measurement Conference, pages 398-404, 2017.
[12] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pages 593-607. Springer, 2018.
[13] Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197, 2019.
[14] Blase Ur, Melwyn Pak Yong Ho, Stephen Brawner, Jiyun Lee, Sarah Mennicken, Noah Picard, Diane Schulze, and Michael L Littman. Trigger-action programming in the wild: An analysis of 200,000 ifttt recipes. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 3227-3231, 2016.
[15] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.
[16] Qinyue Wu, Beijun Shen, and Yuting Chen. Learning to recommend trigger-action rules for end-user development. In International Conference on Software and Software Reuse, pages 190-207. Springer, 2020.
[17] Yuhang Yao, Mohammad Mahdi Kamani, Zhongwei Cheng, Lin Chen, Carlee Joe-Wong, and Tianqiang Liu. Fedrule: Federated rule recommendation system with graph neural networks. arXiv preprint arXiv:2211.06812, 2022.
[18] Tianlong Yu, Tian Li, Yuqiong Sun, Susanta Nanda, Virginia Smith, Vyas Sekar, and Srinivasan Seshan. Learning context-aware policies from multiple smart homes via federated multi-task learning. In 2020 IEEE/ACM Fifth International Conference on Internet-of-Things Design and Implementation (IoTDI), pages 104–115. IEEE, 2020.
[19] Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Heterogeneous graph neural network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 793-803, 2019.
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We provide details of the Wyze Dataset, with benchmark experiments.
(b) Did you describe the limitations of your work? [Yes] In Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In Section 3, we discuss how we anonymize the dataset so it does not contain any PII.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] The code for the main experiments, along with a sample of the dataset, can be found in the supplementary materials. However, please note that accessing the full dataset requires completing a consent form. Due to this requirement, we are unable to provide the complete dataset at this time. The consent form will be made available for anyone interested in obtaining the dataset, allowing us to release it to them.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In Section 4.1 all the details of the experiments are discussed.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In Section 4.1
(a) If your work uses existing assets, did you cite the creators? [Yes] The codes for implementing the algorithms are generally developed by our group using open-source libraries such as Torch or DGL
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] The URL for the consent form to access the dataset is provided.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We did discuss that the dataset is free of any PII due to the anonymization preprocessing of the dataset.
Dataset Papers Checklist
(a) Dataset documentation and intended uses. Recommended documentation frameworks include datasheets for datasets, dataset nutrition labels, data statements for NLP, and accountability frameworks. A: Available on the dataset page.
(b) URL to website/platform where the dataset/benchmark can be viewed and downloaded by the reviewers. A: The dataset is hosted on HuggingFace dataset API: Wyze Rule Recommendation Dataset
(c) Author statement that they bear all responsibility in case of violation of rights, etc., and confirmation of the data license. A: The consent form in the dataset webpage indicates its intended use and terms of access to the dataset.
(d) Hosting, licensing, and maintenance plan. The choice of hosting platform is yours, as long as you ensure access to the data (possibly through a curated interface) and will provide the necessary maintenance. A: The dataset will be hosted on the HuggingFace repository and will be maintained and updated from there.
(a) Links to access the dataset and its metadata. This can be hidden upon submission if the dataset is not yet publicly available but must be added in the camera-ready version. In select cases, e.g when the data can only be released at a later date, this can be added afterward. Simulation environments should link to (open source) code repositories. A: All are available on the Wyze Rule Recommendation Dataset repository on HuggingFace.
(b) The dataset itself should ideally use an open and widely used data format. Provide a detailed explanation of how the dataset can be read. For simulation environments, use existing frameworks or explain how they can be used. A: The data files are stored in a CSV format. They are easily accessible via HugginFace's dataset Python library.
(c) Long-term preservation: It must be clear that the dataset will be available for a long time, either by uploading to a data repository or by explaining how the authors themselves will ensure this. A: The dataset will be available on the HuggingFace website.
(d) Explicit license: Authors must choose a license, ideally a CC license for datasets, or an open-source license for code (e.g. RL environments). A: This dataset is licensed by cc-by-nc-nd-4.0, which prohibits commercial use, distribution, modification, and reproduction of the data without permission from the copyright holder.
(e) Add structured metadata to a dataset's meta-data page using Web standards (like schema.org and DCAT): This allows it to be discovered and organized by anyone. If you use an existing data repository, this is often done automatically. A: The meta data is available on the repository page.
(f) Highly recommended: a persistent dereferenceable identifier (e.g. a DOI minted by a data repository or a prefix on identifiers.org) for datasets, or a code repository (e.g. GitHub, GitLab,...) for code. If this is not possible or useful, please explain why. Since it can be uniquely identified by the HuggingFace dataset API it does not seem necessary at the moment.
3. For benchmarks, the supplementary materials must ensure that all results are easily reproducible. Where possible, use a reproducibility framework such as the ML reproducibility checklist, or otherwise guarantee that all results can be easily reproduced, i.e. all necessary datasets, code, and evaluation procedures must be accessible and documented. The code for the benchmarks will be available soon through the repository website.
4. For papers introducing best practices in creating or curating datasets and benchmarks, the above supplementary materials are not required.