Reviewerly/exHarmony huggingface dataset:
DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining

Communication scheduling has been shown to be effective in accelerating distributed training, which enables all-reduce communications to be overlapped with backpropagation computations. This has been commonly adopted in popular distributed deep learning frameworks. However, there exist two fundamental problems: (1) excessive startup latency proportional to the number of workers for each all-reduce operation; (2) it only achieves sub-optimal training performance due to the dependency and synchronization requirement of the feed-forward computation in the next iteration. We propose a novel scheduling algorithm, DeAR, that decouples the all-reduce primitive into two continuous operations, which overlaps with both backpropagation and feed-forward computations without extra communications. We further design a practical tensor fusion algorithm to improve the training performance. Experimental results with five popular models show that DeAR achieves up to 83% and 15% training speedup over the state-of-the-art solutions on a 64-GPU cluster with 10Gb/s Ethernet and 100Gb/s InfiniBand interconnects, respectively.
MRTOM: Mostly Reliable Totally Ordered Multicast, a Network Primitive to Offload Distributed Systems

As datacenters become the new computing platform, integrating server-centric distributed systems into modern network hardware is gaining interest under the diminishing Moore's law. Researchers want to build reusable primitives that can take advantage of modern network hardware and offload common system components of a broad range of applications. In this paper, we present Mostly Reliable Totally Ordered Multicast, a reusable network primitive that can embed reliable group communication into the network. MRTOM is a network-centric approach that handles message replication, ordering, and reliable delivery using a network fast path, freeing server CPUs for application logic. MRTOM can be implemented in the programmable switches and edge interfaces (e.g., Smart-NICs), significantly reducing network traffic compared to the existing approaches and improving job finish time amid packet loss. With MRTOM, we were able to accelerate multiple high-performance applications whose fast path can be totally offloaded into the network. For example, a Paxos application, MRTOM-Paxos, achieves > 1,100,000 transactions/secs and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$23\mu \mathrm{s}$</tex> minimum latency. A replicated key-value store, MRTOM-KV, also shows significant latency reduction with eBPF/XDP in the Linux Kernel, which is further improved by offloading into SmartNICs.
Mitigating Adversarial Attacks in Federated Learning with Trusted Execution Environments

The main premise of federated learning (FL) is that machine learning model updates are computed locally to preserve user data privacy. This approach avoids by design user data to ever leave the perimeter of their device. Once the updates aggregated, the model is broadcast to all nodes in the federation. However, without proper defenses, compromised nodes can probe the model inside their local memory in search for adversarial examples, which can lead to dangerous real-world scenarios. For instance, in image-based applications, adversarial examples consist of images slightly perturbed to the human eye getting misclassified by the local model. These adversarial images are then later presented to a victim node's counterpart model to replay the attack. Typical examples harness dissemination strategies such as altered traffic signs (patch attacks) no longer recognized by autonomous vehicles or seemingly unaltered samples that poison the local dataset of the FL scheme to undermine its robustness. PELTA is a novel shielding mechanism leveraging Trusted Execution Environments (TEEs) that reduce the ability of attackers to craft adversarial samples. PELTA masks inside the TEE the first part of the back-propagation chain rule, typically exploited by attackers to craft the malicious samples. We evaluate PELTA on state-of-the-art accurate models using three well-established datasets: CIFAR-10, CIFAR-100 and ImageNet. We show the effectiveness of PELTA in mitigating six white-box state-of-the-art adversarial attacks, such as Projected Gradient Descent, Momentum Iterative Method, Auto Projected Gradient Descent, the Carlini & Wagner attack. In particular, PELTA constitutes the first attempt at defending an ensemble model against the Self-Attention Gradient attack to the best of our knowledge. Our code is available to the research community at https://github.com/queyrusi/Pelta
bbTopk: Bandwidth-Aware Sparse Allreduce with Blocked Sparsification for Efficient Distributed Training

Communication overhead is one of the major bottlenecks for large-scale distributed model training. Sparse gradient has been proposed to reduce the communication volume dramatically without affecting the model accuracy. However, high performance implementation of sparse gradient is still hindered by the overheads of gradient sparsification and inefficient implementation of sparse allreduce. For a sparse allreduce operation, the density of intermediate results may dynamically increase as summations progress. Meanwhile, high-performance sparse allreduce is further complicated with heterogeneous bandwidths. To tackle these challenges, we propose bbTopk for efficient sparse gradient training, which includes a new blocked top <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$k$</tex> sparsification technique and a novel bandwidth-aware sparse allreduce algorithm. In particular, to alleviate the sparsification overhead, we design a blocked top- <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$k$</tex> method, which can reduce the overhead without sacrificing model accuracy. We then build a heterogeneity-aware communication model combined with the dynamic workload feature of the sparse allreduce. Guided by that, a new sparse allreduce algorithm is proposed that can take advantage of the network resources by improving bandwidth utilization, reducing cross-node hops, and adjusting the round order. Experiments are conducted on a variety of typical neural networks and distributed environments. Results show that bbTopk can substantially outperform the previous state-of-the-art work in most test cases with up to 2.57x speedup while achieving similar accuracy with the dense model empirically.
GeckoDAG: Towards a Lightweight DAG-Based Blockchain via Reducing Data Redundancy

To overcome the scaling and performance limitations, the Directed Acyclic Graph (DAG) is utilized as the underlying storage model of blockchain systems, which enables concurrent transaction processing and confirmation. However, accompanied by high performance, DAG-based blockchains still suffer from the severe challenge of constrained storage scalability, i.e., expensive storage overhead. Based on an in-depth analysis of the data, we discover that the root cause of storage overhead stems from the considerable data redundancy in the DAG-based blockchains. In this paper, we propose GeckoDAG, a lightweight DAG-based blockchain, whose design consists of two steps. First, we abstract a storage model named Basic from the existing DAG-based blockchain systems, which offers both high performance and security. On top of Basic, we then devise GeckoDAG, which merges previous transactions into Transaction Union (TU) and reduces the data redundancy in TU, thus lowering the storage overhead. To evaluate our design, we implement a prototype of GeckoDAG and conduct various experiments on it. The experimental results demonstrate that GeckoDAG can offer storage scalability while maintaining the security and efficiency of DAG-based blockchains.
BlockPAT: A Blockchain-Enabled Second-Hand Physical Asset Tokenization Management System

In this work, we develop BlockPAT, a blockchain-enabled management system for the tokenization of second-hand physical assets, e.g., laptops. With this system, the information gap between buyers and sellers in the second-hand market will be eliminated, and with the help of a price oracle, the liquidity of the second-hand market can be greatly improved. Furthermore, our system is built upon the latest ZK-rollups solution; thus, the overall transaction cost and time delay will be limited to an affordable value.
Demo: SGX-MR-Prot: Efficient and Developer-Friendly Access-Pattern Protection in Trusted Execution Environments

Trusted Execution Environments, such as Intel SGX, use hardware supports to ensure the confidentiality and integrity of applications against a compromised cloud system. However, side channels like access patterns remain for adversaries to exploit and obtain sensitive information. Common approaches use oblivious programs or primitives, such as ORAM, to make access patterns oblivious to input data, which are challenging to develop. This demonstration shows a prototype SGX-MR-Prot for efficiently protecting access patterns of SGX-based data-intensive applications and minimizing developers' efforts. SGX-MR-Prot uses the MapReduce framework to regulate application dataflows to reduce the cost of access-pattern protection and hide the data oblivious details from SGX developers. This demonstration will allow users to intuitively understand the unique contributions of the framework-based protection approach via interactive exploration and visualization.
Multi-Site Clinical Federated Learning Using Recursive and Attentive Models and NVFlare

The prodigious growth of digital health data has precipitated a mounting interest in harnessing machine learning methodologies, such as natural language processing (NLP), to scrutinize medical records, clinical notes, and other text-based health information. Although NLP techniques have exhibited substantial potential in augmenting patient care and informing clinical decision-making, data privacy and adherence to regulations persist as critical concerns. Federated learning (FL) emerges as a viable solution, empowering multiple organizations to train machine learning models collaboratively without disseminating raw data. This paper proffers a pragmatic approach to medical NLP by amalgamating FL, NLP models, and the NVFlare framework, developed by NVIDIA. We introduce two exemplary NLP models, the Long-Short Term Memory (LSTM)-based model and Bidirectional Encoder Representations from Transformers (BERT), which have demonstrated exceptional performance in comprehending context and semantics within medical data. This paper encompasses the development of an integrated framework that addresses data privacy and regulatory compliance challenges while maintaining elevated accuracy and performance, incorporating BERT pretraining, and comprehensively substantiating the efficacy of the proposed approach.
Fair DNN Model Selection in Edge AI via A Cooperative Game Approach

Edge intelligence is an emerging paradigm that leverages edge computing to pave the last-mile delivery of artificial intelligence (AI). To adapt to the resource restriction, model selection which adaptively selects DNN model variants is widely applied to shape the resource demand of edge AI inference tasks. Unfortunately, in current edge AI serving systems, applications are suffering unfairness since the DNN model selection is performed in a best-effort manner to maximize the system-wide inference accuracy. To achieve a predictable inference accuracy for the applications, edge AI serving systems should guarantee the minimum inference accuracy in a fair fashion at the application level. At the same time, edge resources should be efficiently utilized to minimize operational costs. In this paper, we model the edge DNN model selection problem as a Nash Bargaining Game (NBG), and propose the model selection principles by guaranteeing a base accuracy for each application. Based on the rigorous cooperative game-theoretic approach, we design an approximate algorithm to achieve computationally-efficient and fair model selection, corresponding to the Nash Bargaining Solution (NBS). With extensive trace-driven simulations, we show that our strategy can meet two desirable requirements towards the predictable inference accuracy for applications as well as low operational costs for the system.
Distributed Pruning Towards Tiny Neural Networks in Federated Learning

Neural network pruning is an essential technique for reducing the size and complexity of deep neural networks, enabling large-scale models on devices with limited resources. However, existing pruning approaches heavily rely on training data for guiding the pruning strategies, making them ineffective for federated learning over distributed and confidential datasets. Additionally, the memory- and computation-intensive pruning process becomes infeasible for recourse-constrained devices in federated learning. To address these challenges, we propose FedTiny, a distributed pruning framework for federated learning that generates specialized tiny models for memory-and computing-constrained devices. We introduce two key modules in FedTiny to adaptively search coarse- and finer-pruned specialized models to fit deployment scenarios with sparse and cheap local computation. First, an adaptive batch normalization selection module is designed to mitigate biases in pruning caused by the heterogeneity of local data. Second, a lightweight progressive pruning module aims to finer prune the models under strict memory and computational budgets, allowing the pruning policy for each layer to be gradually determined rather than evaluating the overall model structure. The experimental results demonstrate the effectiveness of FedTiny, which outperforms state-of-the-art approaches, particularly when compressing deep models to extremely sparse tiny models. FedTiny achieves an accuracy improvement of 2.61% while significantly reducing the computational cost by 95.91% and the memory footprint by 94.01% compared to state-of-the-art methods.
Hierarchical Privacy-Preserved Knowledge Graph

The knowledge graphs have found widespread use in numerous areas. However, as its applications expand, privacy concerns have been raised due to its ability to reveal links between entities in the real world. This brief work introduces an innovative approach to address the concern by considering the hierarchical structure of the knowledge graph to safeguard the privacy of information presented in knowledge graphs.
Stash: A Comprehensive Stall-Centric Characterization of Public Cloud VMs for Distributed Deep Learning

Deep neural networks (DNNs) are increasingly popular owing to their ability to solve complex problems such as image recognition, autonomous driving, and natural language processing. Their growing complexity coupled with the use of larger volumes of training data (to achieve acceptable accuracy) has warranted the use of GPUs and other accelerators. Such accelerators are typically expensive, with users having to pay a high upfront cost to acquire them. For infrequent use, users can, instead, leverage the public cloud to mitigate the high acquisition cost. However, with the wide diversity of hardware instances (particularly GPU instances) available in public cloud, it becomes challenging for a user to make an appropriate choice from a cost/performance standpoint. In this work, we try to address this problem by (i) introducing a comprehensive distributed deep learning (DDL) profiler Stash, which determines the various execution stalls that DDL suffers from, and (ii) using Stash to extensively characterize various public cloud GPU instances by running popular DNN models on them. Specifically, it estimates two types of communication stalls, namely, interconnect and network stalls, that play a dominant role in DDL execution time. Stash is implemented on top of prior work, DS-analyzer, that computes only the CPU and disk stalls. Using our detailed stall characterization, we list the advantages and shortcomings of public cloud GPU instances for users to help them make an informed decision(s). Our characterization results indicate that the more expensive GPU instances may not be the most performant for all DNN models and that AWS can sometimes sub-optimally allocate hardware interconnect resources. Specifically, the intra-machine interconnect can introduce communication overheads of up to 90% of DNN training time and the network-connected instances can suffer from up to 5× slowdown compared to training on a single instance. Furthermore, (iii) we also model the impact of DNN macroscopic features such as the number of layers and the number of gradients on communication stalls, and finally, (iv) we briefly discuss a cost comparison with existing work.
ScanFed: Scalable Behavior-Based Backdoor Detection in Federated Learning

Federated Learning (FL) has been adopted in practical network applications and plays a critical role. As FL allows participants to contribute to the global model by training locally with private data, it is known particularly vulnerable to neural backdoor attacks. This paper proposes a new defense, ScanFed, against neural backdoor attacks to FL systems. It leverages the synchronous nature of FL to effectively single out malicious neuron candidates and further validate if they indeed hijack the model's behaviors. Compared to existing neural backdoor defenses, ScanFed has the following distinct properties. First, it is extremely computation-friendly that is six orders of magnitude faster than state-of-the-art behavior-based backdoor defenses, rendering it highly suitable for large-scale FL systems. Second, it inherits the precise nature of behavior-based backdoor detection, making it significantly more effective than similarity-based defenses against advanced attacks. Third, it is robust to biased models uploaded by clients with non-IID (Independent and Identically Distributed) data, which is very common in practical FL systems. In addition, it is a plug-n-play scheme that can be seamlessly integrated into existing FL systems. To the best of our knowledge, this is the first behavior-based defense that enables scalable, efficient and accurate neural backdoor detection of FL systems in non-IID scenarios. This work delivers a ScanFed prototype and fully tests it in various settings of datasets, neural architectures, and backdoor attacks. The experiments demonstrate ScanFed achieves competitive accuracy and minimal detection time.
RangeQC: A Query Control Framework for Range Query Leakage Quantification and Mitigation

Encrypted range schemes enable the user to perform expressive range queries over encrypted databases by only revealing the necessary information for the search. In comparison to the extensively studied keyword schemes, range schemes inherently carry a more detrimental leakage profile as they contain extra structural information that can be exploited by attackers. This is evidenced by the escalation of attacks and defenses targeting range schemes in recent years, showing an inadequate understanding of range schemes' leakage implications. To this end, we identify and investigate two major aspects that have been largely overlooked in prior leakage analysis: 1) lack of a comprehensive approach for coordinating access, volume, and search patterns simultaneously, resulting in separate resolutions for each type of leakage; 2) absence of granular leakage quantification aimed at individual queries, leading to the gradual accumulation of imperceptible leakage over time. Based on this understanding, we present the query control framework RangeQC that provides a novel uniformed viewpoint of all three primary patterns at a per-query level. At the core of our techniques is a set of customized entropy analyzes tailored for the representative structures extracting various types of patterns. We further develop feasible countermeasures to suppress pattern leakage with tunable query control rules. Extensive evaluation results on real-world datasets demonstrate the superiority of RangeQC for leakage quantification and mitigation effectiveness.
Get More for Less in Decentralized Learning Systems

Decentralized learning (DL) systems have been gaining popularity because they avoid raw data sharing by communicating only model parameters, hence preserving data confidentiality. However, the large size of deep neural networks poses a significant challenge for decentralized training, since each node needs to exchange gigabytes of data, overloading the network. In this paper, we address this challenge with Jwins, a communication-efficient and fully decentralized learning system that shares only a subset of parameters through sparsification. Jwins uses wavelet transform to limit the information loss due to sparsification and a randomized communication cut-off that reduces communication usage without damaging the performance of trained models. We demonstrate empirically with 96 DL nodes on non-IID datasets that Jwins can achieve similar accuracies to full-sharing DL while sending up to 64% fewer bytes. Additionally, on low communication budgets, Jwins outperforms the state-of-the-art communication-efficient DL algorithm Choco-SGD by up to 4x in terms of network savings and time.
FedDM: Data and Model Heterogeneity-Aware Federated Learning via Dynamic Weight Sharing

Federated Learning (FL) plays an indispensable role in edge computing systems. Prevalent FL methods mainly address challenges involved in heterogeneous data distribution across devices. Model heterogeneity, however, has seldom been put under scrutiny. In practice, different devices (e.g., PCs and smartphones) generally have disparate computation and communication resources, necessitating neural network models with varying parameter sizes. Therefore, we propose FedDM, a novel data and model heterogeneity-aware FL system, which improves the FL system's accuracy while reducing edge devices' computation and communication costs for heterogeneous model training. FedDM features: 1) dynamic weight sharing scheme that handles model heterogeneity by dynamically selecting parts of the large model to share with smaller ones; 2) tree-structured layer-wise client cooperation scheme that handles data heterogeneity by allowing clients with similar data distribution to share some network layers. We implement FedDM and evaluate it using five public datasets with different tasks.
Distributed Agent-Based Control in Dynamic Environments

We present our research project on distributed, agent-based control in dynamic environments. We motivate the usage of agents and their difficulties in dynamic environments like the Web, and relate this to the fields of distributed and agent-based systems. We describe our idea of lightweight strategies for agents to adapt behavior on disturbances without relying on centralized mechanisms, and outline objectives and challenges of our project. Finally, we explain our simulation experiments as research method, and discuss our initial results to design distributed, lightweight, scalable agents for dynamic environments.
CAUSEC: Cache-Based Secure Key Computation with (Mostly) Deprivileged Execution

As cold boot attacks become a realistic threat to cryptographic systems, several defense solutions have been proposed in the past decade to protect cryptographic systems against such attacks. Interestingly, most of these defense solutions are implemented at the kernel level. Yet running them at the kernel level is risky. Given the complexity of these defense solutions, they inevitably introduce vulnerabilities that could be exploited by attackers and then lead to the compromise of the entire operating system. In this paper, we present CAUSEC which avoids storing crypto keys and other sensitive information in the memory and performs key computation in the cache. CAUSEC protects cryptographic systems against cold boot attacks, but is mostly deprivileged to the user mode. Our experimental results demonstrate that CAUSEC secures key computation and incurs reasonable performance overhead: 11.99% in decryption rate and 7.1% in decryption/signing requests processing when incorporated with the Apache web server.