OpenRaiser/Intern-Atlas huggingface dataset:
Privacy-Preserving Graph Convolutional Networks for Text Classification
Venue: arXiv cs.SI 2022
Year: 2022

Graph convolutional networks (GCNs) are a powerful architecture for representation learning on documents that naturally occur as graphs, e.g., citation or social networks. However, sensitive personal information, such as documents with people's profiles or relationships as edges, are prone to privacy leaks, as the trained model might reveal the original input. Although differential privacy (DP) offers a well-founded privacy-preserving framework, GCNs pose theoretical and practical challenges due to their training specifics. We address these challenges by adapting differentially-private gradient-based training to GCNs and conduct experiments using two optimizers on five NLP datasets in two languages. We propose a simple yet efficient method based on random graph splits that not only improves the baseline privacy bounds by a factor of 2.7 while retaining competitive F1 scores, but also provides strong privacy guarantees of epsilon = 1.0. We show that, under certain modeling choices, privacy-preserving GCNs perform up to 90% of their non-private variants, while formally guaranteeing strong privacy measures.
Effective Cache Apportioning for Performance Isolation Under Compiler Guidance
Venue: arXiv cs.PL 2022
Year: 2022

With a growing number of cores in modern high-performance servers, effective sharing of the last level cache (LLC) is more critical than ever. The primary agenda of such systems is to maximize performance by efficiently supporting multi-tenancy of diverse workloads. However, this could be particularly challenging to achieve in practice, because modern workloads exhibit dynamic phase behaviour, which causes their cache requirements & sensitivities to vary at finer granularities during execution. Unfortunately, existing systems are oblivious to the application phase behavior, and are unable to detect and react quickly enough to these rapidly changing cache requirements, often incurring significant performance degradation. In this paper, we propose Com-CAS, a new apportioning system that provides dynamic cache allocations for co-executing applications. Com-CAS differs from the existing cache partitioning systems by adapting to the dynamic cache requirements of applications just-in-time, as opposed to reacting, without any hardware modifications. The front-end of Com-CAS consists of compiler-analysis equipped with machine learning mechanisms to predict cache requirements, while the back-end consists of proactive scheduler that dynamically apportions LLC amongst co-executing applications leveraging Intel Cache Allocation Technology (CAT). Com-CAS's partitioning scheme utilizes the compiler-generated information across finer granularities to predict the rapidly changing dynamic application behaviors, while simultaneously maintaining data locality. Our experiments show that Com-CAS improves average weighted throughput by 15% over unpartitioned cache system, and outperforms state-of-the-art partitioning system KPart by 20%, while maintaining the worst individual application completion time degradation to meet various Service-Level Agreement (SLA) requirements.
Resource Management for 5G NR Integrated Access and Backhaul: a Semi-centralized Approach
Venue: arXiv cs.NI 2022
Year: 2022

The next generations of mobile networks will be deployed as ultra-dense networks, to match the demand for increased capacity and the challenges that communications in the higher portion of the spectrum (i.e., the mmWave band) introduce. Ultra-dense networks, however, require pervasive, high-capacity backhaul solutions, and deploying fiber optic to all base stations is generally considered to be too expensive for network operators. The 3rd Generation Partnership Project (3GPP) has thus introduced Integrated Access and Backhaul (IAB), a wireless backhaul solution in which the access and backhaul links share the same hardware, protocol stack, and also spectrum. The multiplexing of different links in the same frequency bands, however, introduces interference and capacity sharing issues, thus calling for the introduction of advanced scheduling and coordination schemes. This paper proposes a semi-centralized resource allocation scheme for IAB networks, designed to be flexible, with low complexity, and compliant with the 3GPP IAB specifications. We develop a version of the Maximum Weighted Matching (MWM) problem that can be applied on a spanning tree that represents the IAB network and whose complexity is linear in the number of IAB-nodes. The proposed solution is compared with state-of-the-art distributed approaches through end-to-end, full-stack system-level simulations with a 3GPP-compliant channel model, protocol stack, and a diverse set of user applications. The results show how that our scheme can increase the throughput of cell-edge users up to 5 times, while decreasing the overall network congestion with an end-to-end delay reduction of up to 25 times.
Exploring Factors and Metrics to Select Open Source Software Components for Integration: An Empirical Study
Venue: arXiv cs.SE 2022
Year: 2022

[Context] Open Source Software (OSS) is nowadays used and integrated in most of the commercial products. However, the selection of OSS projects for integration is not a simple process, mainly due to a of lack of clear selection models and lack of information from the OSS portals. [Objective] We investigate the factors and metrics that practitioners currently consider when selecting OSS. We also investigate the source of information and portals that can be used to assess the factors, as well as the possibility to automatically extract such information with APIs. [Method] We elicited the factors and the metrics adopted to assess and compare OSS performing a survey among 23 experienced developers who often integrate OSS in the software they develop. Moreover, we investigated the APIs of the portals adopted to assess OSS extracting information for the most starred 100K projects in GitHub. [Result] We identified a set consisting of 8 main factors and 74 sub-factors, together with 170 related metrics that companies can use to select OSS to be integrated in their software projects. Unexpectedly, only a small part of the factors can be evaluated automatically, and out of 170 metrics, only 40 are available, of which only 22 returned information for all the 100K projects. Therefore, we recommend project maintainers and project repositories to pay attention to provide information for the project they are hosting, so as to increase the likelihood of being adopted [Conclusion] OSS selection can be partially automated, by extracting the information needed for the selection from portal APIs. OSS producers can benefit from our results by checking if they are providing all the information commonly required by potential adopters...
An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks
Venue: arXiv cs.LG 2022
Year: 2022

Edge TPUs are a domain of accelerators for low-power, edge devices and are widely used in various Google products such as Coral and Pixel devices. In this paper, we first discuss the major microarchitectural details of Edge TPUs. Then, we extensively evaluate three classes of Edge TPUs, covering different computing ecosystems, that are either currently deployed in Google products or are the product pipeline, across 423K unique convolutional neural networks. Building upon this extensive study, we discuss critical and interpretable microarchitectural insights about the studied classes of Edge TPUs. Mainly, we discuss how Edge TPU accelerators perform across convolutional neural networks with different structures. Finally, we present our ongoing efforts in developing high-accuracy learned machine learning models to estimate the major performance metrics of accelerators such as latency and energy consumption. These learned models enable significantly faster (in the order of milliseconds) evaluations of accelerators as an alternative to time-consuming cycle-accurate simulators and establish an exciting opportunity for rapid hard-ware/software co-design.
Resilience of Bayesian Layer-Wise Explanations under Adversarial Attacks
Venue: arXiv cs.LG 2022
Year: 2022

We consider the problem of the stability of saliency-based explanations of Neural Network predictions under adversarial attacks in a classification task. Saliency interpretations of deterministic Neural Networks are remarkably brittle even when the attacks fail, i.e. for attacks that do not change the classification label. We empirically show that interpretations provided by Bayesian Neural Networks are considerably more stable under adversarial perturbations of the inputs and even under direct attacks to the explanations. By leveraging recent results, we also provide a theoretical explanation of this result in terms of the geometry of the data manifold. Additionally, we discuss the stability of the interpretations of high level representations of the inputs in the internal layers of a Network. Our results demonstrate that Bayesian methods, in addition to being more robust to adversarial attacks, have the potential to provide more stable and interpretable assessments of Neural Network predictions.
SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning
Venue: arXiv cs.LG 2022
Year: 2022

In this paper, we consider risk-sensitive sequential decision-making in Reinforcement Learning (RL). Our contributions are two-fold. First, we introduce a novel and coherent quantification of risk, namely composite risk, which quantifies the joint effect of aleatory and epistemic risk during the learning process. Existing works considered either aleatory or epistemic risk individually, or as an additive combination. We prove that the additive formulation is a particular case of the composite risk when the epistemic risk measure is replaced with expectation. Thus, the composite risk is more sensitive to both aleatory and epistemic uncertainty than the individual and additive formulations. We also propose an algorithm, SENTINEL-K, based on ensemble bootstrapping and distributional RL for representing epistemic and aleatory uncertainty respectively. The ensemble of K learners uses Follow The Regularised Leader (FTRL) to aggregate the return distributions and obtain the composite risk. We experimentally verify that SENTINEL-K estimates the return distribution better, and while used with composite risk estimates, demonstrates higher risk-sensitive performance than state-of-the-art risk-sensitive and distributional RL algorithms.
Recent Advances in Fully Dynamic Graph Algorithms
Venue: arXiv cs.DS 2022
Year: 2022

In recent years, significant advances have been made in the design and analysis of fully dynamic algorithms. However, these theoretical results have received very little attention from the practical perspective. Few of the algorithms are implemented and tested on real datasets, and their practical potential is far from understood. Here, we present a quick reference guide to recent engineering and theory results in the area of fully dynamic graph algorithms.
Uncertainty Maximization in Partially Observable Domains: A Cognitive Perspective
Venue: arXiv cs.AI 2022
Year: 2022

Faced with an ever-increasing complexity of their domains of application, artificial learning agents are now able to scale up in their ability to process an overwhelming amount of information coming from their interaction with an environment. However, this process of scaling does come with a cost of encoding and processing an increasing amount of redundant information that is not necessarily beneficial to the learning process itself. This work exploits the properties of the learning systems defined over partially observable domains by selectively focusing on the specific type of information that is more likely to express the causal interaction among the transitioning states of the environment. Adaptive masking of the observation space based on the temporal difference displacement criterion enabled a significant improvement in convergence of temporal difference algorithms defined over a partially observable Markov process.
A theory of Automated Market Makers in DeFi
Venue: arXiv cs.FL 2022
Year: 2022

Automated market makers (AMMs) are one of the most prominent decentralized finance (DeFi) applications. AMMs allow users to trade different types of crypto-tokens, without the need to find a counter-party. There are several implementations and models for AMMs, featuring a variety of sophisticated economic mechanisms. We present a theory of AMMs. The core of our theory is an abstract operational model of the interactions between users and AMMs, which can be concretised by instantiating the economic mechanisms. We exploit our theory to formally prove a set of fundamental properties of AMMs, characterizing both structural and economic aspects. We do this by abstracting from the actual economic mechanisms used in implementations, and identifying sufficient conditions which ensure the relevant properties. Notably, we devise a general solution to the arbitrage problem, the main game-theoretic foundation behind the economic mechanisms of AMMs.
A Graphical Framework to Study the Correlation between Geometric Design and Simulation
Venue: arXiv cs.CG 2022
Year: 2022

Partial differential equations can be solved on general polygonal and polyhedral meshes, through Polytopal Element Methods (PEMs). Unfortunately, the relation between geometry and analysis is still unknown and subject to ongoing research in order to identify weaker shape-regularity criteria under which PEMs can reliably work. We propose PEMesh, a graphical framework to support the analysis of the relation between the geometric properties of polygonal meshes and the numerical performances of PEM solvers. PEMesh allows the design of polygonal meshes that increasingly stress some geometric properties, by exploiting any external PEM solver, and supports the study of the correlation between the performances of such a solver and geometric properties of the input mesh. Furthermore, it is highly modular, customisable, easy to use, and provides the possibility to export analysis results both as numerical values and graphical plots. PEMesh has a potential practical impact on ongoing and future research activities related to PEM methods, polygonal mesh generation and processing.
Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others
Venue: arXiv cs.AI 2022
Year: 2022

To achieve human-like common sense about everyday life, machine learning systems must understand and reason about the goals, preferences, and actions of other agents in the environment. By the end of their first year of life, human infants intuitively achieve such common sense, and these cognitive achievements lay the foundation for humans' rich and complex understanding of the mental states of others. Can machines achieve generalizable, commonsense reasoning about other agents like human infants? The Baby Intuitions Benchmark (BIB) challenges machines to predict the plausibility of an agent's behavior based on the underlying causes of its actions. Because BIB's content and paradigm are adopted from developmental cognitive science, BIB allows for direct comparison between human and machine performance. Nevertheless, recently proposed, deep-learning-based agency reasoning models fail to show infant-like reasoning, leaving BIB an open challenge.
Pre-Training on Dynamic Graph Neural Networks
Venue: arXiv cs.LG 2022
Year: 2022

The pre-training on the graph neural network model can learn the general features of large-scale networks or networks of the same type by self-supervised methods, which allows the model to work even when node labels are missing. However, the existing pre-training methods do not take network evolution into consideration. This paper proposes a pre-training method on dynamic graph neural networks (PT-DGNN), which uses dynamic attributed graph generation tasks to simultaneously learn the structure, semantics, and evolution features of the graph. The method includes two steps: 1) dynamic sub-graph sampling, and 2) pre-training with dynamic attributed graph generation task. Comparative experiments on three realistic dynamic network datasets show that the proposed method achieves the best results on the link prediction fine-tuning task.
High-Capacity Framework for Reversible Data Hiding in Encrypted Image Using Pixel Predictions and Entropy Encoding
Venue: arXiv cs.MM 2022
Year: 2022

While the existing vacating room before encryption (VRBE) based schemes can achieve decent embedding rate, the payloads of the existing vacating room after encryption (VRAE) based schemes are relatively low. To address this issue, this paper proposes a generalized framework for high-capacity RDHEI for both VRBE and VRAE cases. First, an efficient embedding room generation algorithm (ERGA) is designed to produce large embedding room by using pixel prediction and entropy encoding. Then, we propose two RDHEI schemes, one for VRBE, another for VRAE. In the VRBE scenario, the image owner generates the embedding room with ERGA and encrypts the preprocessed image by using the stream cipher with two encryption keys. Then, the data hider locates the embedding room and embeds the encrypted additional data. In the VRAE scenario, the cover image is encrypted by an improved block modulation and permutation encryption algorithm, where the spatial redundancy in the plain-text image is largely preserved. Then, the data hider applies ERGA on the encrypted image to generate the embedding room and conducts data embedding. For both schemes, the receivers with different authentication keys can respectively conduct error-free data extraction and/or error-free image recovery. The experimental results show that the two proposed schemes outperform many state-of-the-art RDHEI arts. Besides, the schemes can ensure high security level, where the original image can be hardly discovered from the encrypted version before and after data hiding by the unauthorized user.
The Magic of Superposition: A Survey on Simultaneous Transmission Based Wireless Systems
Venue: arXiv cs.IT 2022
Year: 2022

In conventional communication systems, any interference between two communicating points is regarded as unwanted noise since it distorts the received signals. On the other hand, allowing simultaneous transmission and intentionally accepting the interference of signals and even benefiting from it have been considered for a range of wireless applications. As prominent examples, non-orthogonal multiple access (NOMA), joint source-channel coding, and the computation codes are designed to exploit this scenario. They also inspired many other fundamental works from network coding to consensus algorithms. Especially, federated learning is an emerging technology that can be applied to distributed machine learning networks by allowing simultaneous transmission. Although various simultaneous transmission applications exist independently in the literature, their main contributions are all based on the same principle; the superposition property. In this survey, we aim to emphasize the connections between these studies and provide a guide for the readers on the wireless communication techniques that benefit from the superposition of signals. We classify the existing literature depending on their purpose and application area and present their contributions. The survey shows that simultaneous transmission can bring scalability, security, low-latency, low-complexity and energy efficiency for certain distributed wireless scenarios which are inevitable with the emerging Internet of things (IoT) applications.
PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation
Venue: arXiv cs.LG 2022
Year: 2022

One of the most efficient methods for model compression is hint distillation, where the student model is injected with information (hints) from several different layers of the teacher model. Although the selection of hint points can drastically alter the compression performance, conventional distillation approaches overlook this fact and use the same hint points as in the early studies. Therefore, we propose a clustering based hint selection methodology, where the layers of teacher model are clustered with respect to several metrics and the cluster centers are used as the hint points. Our method is applicable for any student network, once it is applied on a chosen teacher network. The proposed approach is validated in CIFAR-100 and ImageNet datasets, using various teacher-student pairs and numerous hint distillation methods. Our results show that hint points selected by our algorithm results in superior compression performance compared to state-of-the-art knowledge distillation algorithms on the same student models and datasets.
Image-Based Trajectory Tracking through Unknown Environments without Absolute Positioning
Venue: arXiv cs.RO 2022
Year: 2022

This paper describes a stereo image-based visual servoing system for trajectory tracking by a non-holonomic robot without externally derived pose information nor a known visual map of the environment. It is called trajectory servoing. The critical component is a feature-based, indirect Simultaneous Localization And Mapping (SLAM) method to provide a pool of available features with estimated depth, so that they may be propagated forward in time to generate image feature trajectories for visual servoing. Short and long distance experiments show the benefits of trajectory servoing for navigating unknown areas without absolute positioning. Empirically, trajectory servoing has better trajectory tracking performance than pose-based feedback when both rely on the same underlying SLAM system.
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
Venue: arXiv cs.LG 2022
Year: 2022

We empirically demonstrate that full-batch gradient descent on neural network training objectives typically operates in a regime we call the Edge of Stability. In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value $2 / \text{(step size)}$, and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales. Since this behavior is inconsistent with several widespread presumptions in the field of optimization, our findings raise questions as to whether these presumptions are relevant to neural network training. We hope that our findings will inspire future efforts aimed at rigorously understanding optimization at the Edge of Stability. Code is available at https://github.com/locuslab/edge-of-stability.