OpenRaiser/Intern-Atlas huggingface dataset:
Infusing Finetuning with Semantic Dependencies
Venue: arXiv cs.CL 2021
Year: 2021

For natural language processing systems, two kinds of evidence support the use of text representations from neural language models "pretrained" on large unannotated corpora: performance on application-inspired benchmarks (Peters et al., 2018, inter alia), and the emergence of syntactic abstractions in those representations (Tenney et al., 2019, inter alia). On the other hand, the lack of grounded supervision calls into question how well these representations can ever capture meaning (Bender and Koller, 2020). We apply novel probes to recent language models -- specifically focusing on predicate-argument structure as operationalized by semantic dependencies (Ivanova et al., 2012) -- and find that, unlike syntax, semantics is not brought to the surface by today's pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning, yielding benefits to natural language understanding (NLU) tasks in the GLUE benchmark. This approach demonstrates the potential for general-purpose (rather than task-specific) linguistic supervision, above and beyond conventional pretraining and finetuning. Several diagnostics help to localize the benefits of our approach.
Segmenting Natural Language Sentences via Lexical Unit Analysis
Venue: arXiv cs.CL 2021
Year: 2021

In this work, we present Lexical Unit Analysis (LUA), a framework for general sequence segmentation tasks. Given a natural language sentence, LUA scores all the valid segmentation candidates and utilizes dynamic programming (DP) to extract the maximum scoring one. LUA enjoys a number of appealing properties such as inherently guaranteeing the predicted segmentation to be valid and facilitating globally optimal training and inference. Besides, the practical time complexity of LUA can be reduced to linear time, which is very efficient. We have conducted extensive experiments on 5 tasks, including syntactic chunking, named entity recognition (NER), slot filling, Chinese word segmentation, and Chinese part-of-speech (POS) tagging, across 15 datasets. Our models have achieved the state-of-the-art performances on 13 of them. The results also show that the F1 score of identifying long-length segments is notably improved.
Communication-Computation Efficient Secure Aggregation for Federated Learning
Venue: arXiv cs.LG 2021
Year: 2021

Federated learning has been spotlighted as a way to train neural networks using distributed data with no need for individual nodes to share data. Unfortunately, it has also been shown that adversaries may be able to extract local data contents off model parameters transmitted during federated learning. A recent solution based on the secure aggregation primitive enabled privacy-preserving federated learning, but at the expense of significant extra communication/computational resources. In this paper, we propose a low-complexity scheme that provides data privacy using substantially reduced communication/computational resources relative to the existing secure solution. The key idea behind the suggested scheme is to design the topology of secret-sharing nodes as a sparse random graph instead of the complete graph corresponding to the existing solution. We first obtain the necessary and sufficient condition on the graph to guarantee both reliability and privacy. We then suggest using the Erdős-Rényi graph in particular and provide theoretical guarantees on the reliability/privacy of the proposed scheme. Through extensive real-world experiments, we demonstrate that our scheme, using only $20 \sim 30\%$ of the resources required in the conventional scheme, maintains virtually the same levels of reliability and data privacy in practical federated learning systems.
An Overview of 5G System Accessibility Differentiation and Control
Venue: arXiv cs.NI 2021
Year: 2021

5G system is characterized by its capability to support a wide range of use cases and services. Supporting accessibility differentiation becomes therefore essential to preserve a stable network condition during high traffic load, while ensuring connection and service quality to prioritized devices and services. In this article, we describe some use cases and requirements that impact the 3GPP design of the 5G accessibility differentiation framework. We then provide an overview of the supported mechanisms for accessibility differentiation and control in 5G Stand Alone (SA) system, including cell barring and reservation, unified access control, paging control, random access control and admission control. We discuss how these functionalities can be used to maintain the service quality and selectively limit the incoming traffic to the network at high load situations, leveraging different connection-type indicators and connection-priority identifiers.
Geometric Adversarial Attacks and Defenses on 3D Point Clouds
Venue: arXiv cs.CV 2021
Year: 2021

Deep neural networks are prone to adversarial examples that maliciously alter the network's outcome. Due to the increasing popularity of 3D sensors in safety-critical systems and the vast deployment of deep learning models for 3D point sets, there is a growing interest in adversarial attacks and defenses for such models. So far, the research has focused on the semantic level, namely, deep point cloud classifiers. However, point clouds are also widely used in a geometric-related form that includes encoding and reconstructing the geometry. In this work, we are the first to consider the problem of adversarial examples at a geometric level. In this setting, the question is how to craft a small change to a clean source point cloud that leads, after passing through an autoencoder model, to the reconstruction of a different target shape. Our attack is in sharp contrast to existing semantic attacks on 3D point clouds. While such works aim to modify the predicted label by a classifier, we alter the entire reconstructed geometry. Additionally, we demonstrate the robustness of our attack in the case of defense, where we show that remnant characteristics of the target shape are still present at the output after applying the defense to the adversarial input. Our code is publicly available at https://github.com/itailang/geometric_adv.
On the Secrecy Capacity of MIMO Wiretap Channels: Convex Reformulation and Efficient Numerical Methods
Venue: arXiv cs.IT 2021
Year: 2021

This paper presents novel numerical approaches to finding the secrecy capacity of the multiple-input multiple-output (MIMO) wiretap channel subject to multiple linear transmit covariance constraints, including sum power constraint, per antenna power constraints and interference power constraint. An analytical solution to this problem is not known and existing numerical solutions suffer from slow convergence rate and/or high per-iteration complexity. Deriving computationally efficient solutions to the secrecy capacity problem is challenging since the secrecy rate is expressed as a difference of convex functions (DC) of the transmit covariance matrix, for which its convexity is only known for some special cases. In this paper we propose two low-complexity methods to compute the secrecy capacity along with a convex reformulation for degraded channels. In the first method we capitalize on the accelerated DC algorithm which requires solving a sequence of convex subproblems, for which we propose an efficient iterative algorithm where each iteration admits a closed-form solution. In the second method, we rely on the concave-convex equivalent reformulation of the secrecy capacity problem which allows us to derive the so-called partial best response algorithm to obtain an optimal solution. Notably, each iteration of the second method can also be done in closed form. The simulation results demonstrate a faster convergence rate of our methods compared to other known solutions. We carry out extensive numerical experiments to evaluate the impact of various parameters on the achieved secrecy capacity.
Using Differentiable Programming for Flexible Statistical Modeling
Venue: arXiv cs.LG 2021
Year: 2021

Differentiable programming has recently received much interest as a paradigm that facilitates taking gradients of computer programs. While the corresponding flexible gradient-based optimization approaches so far have been used predominantly for deep learning or enriching the latter with modeling components, we want to demonstrate that they can also be useful for statistical modeling per se, e.g., for quick prototyping when classical maximum likelihood approaches are challenging or not feasible. In an application from a COVID-19 setting, we utilize differentiable programming to quickly build and optimize a flexible prediction model adapted to the data quality challenges at hand. Specifically, we develop a regression model, inspired by delay differential equations, that can bridge temporal gaps of observations in the central German registry of COVID-19 intensive care cases for predicting future demand. With this exemplary modeling challenge, we illustrate how differentiable programming can enable simple gradient-based optimization of the model by automatic differentiation. This allowed us to quickly prototype a model under time pressure that outperforms simpler benchmark models. We thus exemplify the potential of differentiable programming also outside deep learning applications, to provide more options for flexible applied statistical modeling.
Deep Argumentative Explanations
Venue: arXiv cs.AI 2021
Year: 2021

Despite the recent, widespread focus on eXplainable AI (XAI), explanations computed by XAI methods tend to provide little insight into the functioning of Neural Networks (NNs). We propose a novel framework for obtaining (local) explanations from NNs while providing transparency about their inner workings, and show how to deploy it for various neural architectures and tasks. We refer to our novel explanations collectively as Deep Argumentative eXplanations (DAXs in short), given that they reflect the deep structure of the underlying NNs and that they are defined in terms of notions from computational argumentation, a form of symbolic AI offering useful reasoning abstractions for explanation. We evaluate DAXs empirically showing that they exhibit deep fidelity and low computational cost. We also conduct human experiments indicating that DAXs are comprehensible to humans and align with their judgement, while also being competitive, in terms of user acceptance, with some existing approaches to XAI that also have an argumentative spirit.
Virtual Classrooms and Real Harms: Remote Learning at U.S. Universities
Venue: arXiv cs.CR 2021
Year: 2021

Universities have been forced to rely on remote educational technology to facilitate the rapid shift to online learning. In doing so, they acquire new risks of security vulnerabilities and privacy violations. To help universities navigate this landscape, we develop a model that describes the actors, incentives, and risks, informed by surveying 49 educators and 14 administrators at U.S. universities. Next, we develop a methodology for administrators to assess security and privacy risks of these products. We then conduct a privacy and security analysis of 23 popular platforms using a combination of sociological analyses of privacy policies and 129 state laws, alongside a technical assessment of platform software. Based on our findings, we develop recommendations for universities to mitigate the risks to their stakeholders.
Making Cross-Domain Recommendations by Associating Disjoint Users and Items Through the Affective Aware Pseudo Association Method
Venue: arXiv cs.IR 2021
Year: 2021

This paper utilizes an ingenious text-based affective aware pseudo association method (AAPAM) to link disjoint users and items across different information domains and leverage them to make cross-domain content-based and collaborative filtering recommendations. This paper demonstrates that the AAPAM method could seamlessly join different information domain datasets to act as one without any additional cross-domain information retrieval protocols. Besides making cross-domain recommendations, the benefit of joining datasets from different information domains through AAPAM is that it eradicates cold start issues while making serendipitous recommendations.
Fairness-Oriented Multiple RISs-Aided MmWave Transmission: Stochastic Optimization Methods
Venue: arXiv cs.IT 2021
Year: 2021

In millimeter wave (mmWave) systems, it is challenging to ensure the reliable connectivity of communications due to its sensitivity to the presence of blockages. In order to improve the robustness of the mmWave system under the presence of the random blockages, multiple reconfigurable intelligent surfaces (RISs) are deployed to enhance the spatial diversity gain, and robust beamforming is then designed based on a stochastic optimization for minimizing the maximum outage probability among multiple users to ensure the fairness. Under the stochastic optimization framework, we adopt the stochastic majorization--minimization (SMM) method and the stochastic successive convex approximation (SSCA) method to construct deterministic surrogate problems at each iteration for new channel realizations, and obtain the closed-form solutions of the precoding matrix at the base station (BS) and the passive beamforming vectors at the RISs. Both stochastic optimization methods have been proved to converge to the set of stationary points of the original stochastic problems. Finally, simulation results show that the proposed robust beamforming in the RIS-aided system can effectively compensate for the performance loss caused by the presence of the random blockages, especially at high blockage probability, compared with the benchmark solutions.
Acoustic Leak Detection in Water Networks
Venue: arXiv cs.LG 2021
Year: 2021

In this work, we present a general procedure for acoustic leak detection in water networks that satisfies multiple real-world constraints such as energy efficiency and ease of deployment. Based on recordings from seven contact microphones attached to the water supply network of a municipal suburb, we trained several shallow and deep anomaly detection models. Inspired by how human experts detect leaks using electronic sounding-sticks, we use these models to repeatedly listen for leaks over a predefined decision horizon. This way we avoid constant monitoring of the system. While we found the detection of leaks in close proximity to be a trivial task for almost all models, neural network based approaches achieve better results at the detection of distant leaks.
Inferring urban social networks from publicly available data
Venue: arXiv cs.SI 2021
Year: 2021

The emergence of social networks and the definition of suitable generative models for synthetic yet realistic social graphs are widely studied problems in the literature. By not being tied to any real data, random graph models cannot capture all the subtleties of real networks and are inadequate for many practical contexts -- including areas of research, such as computational epidemiology, which are recently high on the agenda. At the same time, the so-called contact networks describe interactions, rather than relationships, and are strongly dependent on the application and on the size and quality of the sample data used to infer them. To fill the gap between these two approaches, we present a data-driven model for urban social networks, implemented and released as open source software. Given a territory of interest, and only based on widely available aggregated demographic and social-mixing data, we construct an age-stratified and geo-referenced synthetic population whose individuals are connected by "strong ties" of two types: intra-household (e.g., kinship) or friendship. While household links are entirely data-driven, we propose a parametric probabilistic model for friendship, based on the assumption that distances and age differences play a role, and that not all individuals are equally sociable. The demographic and geographic factors governing the structure of the obtained network, under different configurations, are thoroughly studied through extensive simulations focused on three Italian cities of different size.
Convolutional LSTM Neural Networks for Modeling Wildland Fire Dynamics
Venue: arXiv cs.LG 2021
Year: 2021

As the climate changes, the severity of wildland fires is expected to worsen. Models that accurately capture fire propagation dynamics greatly help efforts for understanding, responding to and mitigating the damages caused by these fires. Machine learning techniques provide a potential approach for developing such models. The objective of this study is to evaluate the feasibility of using a Convolutional Long Short-Term Memory (ConvLSTM) recurrent neural network to model the dynamics of wildland fire propagation. The machine learning model is trained on simulated wildfire data generated by a mathematical analogue model. Three simulated datasets are analyzed, each with increasing degrees of complexity. The simplest dataset includes a constant wind direction as a single confounding factor, whereas the most complex dataset includes dynamic wind, complex terrain, spatially varying moisture content and heterogenous vegetation density distributions. We examine how effective the ConvLSTM can learn the fire-spread dynamics over consecutive time steps. It is shown that ConvLSTMs can capture local fire transmission events, as well as the overall fire dynamics, such as the rate at which the fire spreads. Finally, we demonstrate that ConvLSTMs outperform other network architectures that have previously been used to model similar wildland fire dynamics.
Communication-Efficient Federated Learning with Compensated Overlap-FedAvg
Venue: arXiv cs.LG 2021
Year: 2021

Petabytes of data are generated each day by emerging Internet of Things (IoT), but only few of them can be finally collected and used for Machine Learning (ML) purposes due to the apprehension of data & privacy leakage, which seriously retarding ML's growth. To alleviate this problem, Federated learning is proposed to perform model training by multiple clients' combined data without the dataset sharing within the cluster. Nevertheless, federated learning introduces massive communication overhead as the synchronized data in each epoch is of the same size as the model, and thereby leading to a low communication efficiency. Consequently, variant methods mainly focusing on the communication rounds reduction and data compression are proposed to reduce the communication overhead of federated learning. In this paper, we propose Overlap-FedAvg, a framework that parallels the model training phase with model uploading & downloading phase, so that the latter phase can be totally covered by the former phase. Compared to vanilla FedAvg, Overlap-FedAvg is further developed with a hierarchical computing strategy, a data compensation mechanism and a nesterov accelerated gradients~(NAG) algorithm. Besides, Overlap-FedAvg is orthogonal to many other compression methods so that they can be applied together to maximize the utilization of the cluster. Furthermore, the theoretical analysis is provided to prove the convergence of the proposed Overlap-FedAvg framework. Extensive experiments on both conventional and recurrent tasks with multiple models and datasets also demonstrate that the proposed Overlap-FedAvg framework substantially boosts the federated learning process.
Internet of Underwater Things and Big Marine Data Analytics -- A Comprehensive Survey
Venue: arXiv cs.NI 2021
Year: 2021

The Internet of Underwater Things (IoUT) is an emerging communication ecosystem developed for connecting underwater objects in maritime and underwater environments. The IoUT technology is intricately linked with intelligent boats and ships, smart shores and oceans, automatic marine transportations, positioning and navigation, underwater exploration, disaster prediction and prevention, as well as with intelligent monitoring and security. The IoUT has an influence at various scales ranging from a small scientific observatory, to a midsized harbor, and to covering global oceanic trade. The network architecture of IoUT is intrinsically heterogeneous and should be sufficiently resilient to operate in harsh environments. This creates major challenges in terms of underwater communications, whilst relying on limited energy resources. Additionally, the volume, velocity, and variety of data produced by sensors, hydrophones, and cameras in IoUT is enormous, giving rise to the concept of Big Marine Data (BMD), which has its own processing challenges. Hence, conventional data processing techniques will falter, and bespoke Machine Learning (ML) solutions have to be employed for automatically learning the specific BMD behavior and features facilitating knowledge extraction and decision support. The motivation of this paper is to comprehensively survey the IoUT, BMD, and their synthesis. It also aims for exploring the nexus of BMD with ML. We set out from underwater data collection and then discuss the family of IoUT data communication techniques with an emphasis on the state-of-the-art research challenges. We then review the suite of ML solutions suitable for BMD handling and analytics. We treat the subject deductively from an educational perspective, critically appraising the material surveyed.
CrypTag: Thwarting Physical and Logical Memory Vulnerabilities using Cryptographically Colored Memory
Venue: arXiv cs.CR 2021
Year: 2021

Memory vulnerabilities are a major threat to many computing systems. To effectively thwart spatial and temporal memory vulnerabilities, full logical memory safety is required. However, current mitigation techniques for memory safety are either too expensive or trade security against efficiency. One promising attempt to detect memory safety vulnerabilities in hardware is memory coloring, a security policy deployed on top of tagged memory architectures. However, due to the memory storage and bandwidth overhead of large tags, commodity tagged memory architectures usually only provide small tag sizes, thus limiting their use for security applications. Irrespective of logical memory safety, physical memory safety is a necessity in hostile environments prevalent for modern cloud computing and IoT devices. Architectures from Intel and AMD already implement transparent memory encryption to maintain confidentiality and integrity of all off-chip data. Surprisingly, the combination of both, logical and physical memory safety, has not yet been extensively studied in previous research, and a naive combination of both security strategies would accumulate both overheads. In this paper, we propose CrypTag, an efficient hardware/software co-design mitigating a large class of logical memory safety issues and providing full physical memory safety. At its core, CrypTag utilizes a transparent memory encryption engine not only for physical memory safety, but also for memory coloring at hardly any additional costs. The design avoids any overhead for tag storage by embedding memory colors in the upper bits of a pointer and using these bits as an additional input for the memory encryption. A custom compiler extension automatically leverages CrypTag to detect logical memory safety issues for commodity programs and is fully backward compatible.