OpenRaiser/Intern-Atlas huggingface dataset:
The DEWCAD Project: Pushing Back the Doubly Exponential Wall of Cylindrical Algebraic Decomposition
Venue: arXiv cs.SC 2021
Year: 2021

This abstract seeks to introduce the ISSAC community to the DEWCAD project, which is based at Coventry University and the University of Bath, in the United Kingdom. The project seeks to push back the Doubly Exponential Wall of Cylindrical Algebraic Decomposition, through the integration of SAT/SMT technology, the extension of Lazard projection theory, and the development of new algorithms based on CAD technology but without producing CADs themselves. The project also seeks to develop applications of CAD and will focus on applications in the domains of economics and bio-network analysis.
Towards Optimally Weighted Physics-Informed Neural Networks in Ocean Modelling
Venue: arXiv cs.LG 2021
Year: 2021

The carbon pump of the world's ocean plays a vital role in the biosphere and climate of the earth, urging improved understanding of the functions and influences of the ocean for climate change analyses. State-of-the-art techniques are required to develop models that can capture the complexity of ocean currents and temperature flows. This work explores the benefits of using physics-informed neural networks (PINNs) for solving partial differential equations related to ocean modeling; such as the Burgers, wave, and advection-diffusion equations. We explore the trade-offs of using data vs. physical models in PINNs for solving partial differential equations. PINNs account for the deviation from physical laws in order to improve learning and generalization. We observed how the relative weight between the data and physical model in the loss function influence training results, where small data sets benefit more from the added physics information.
OpenSSLNTRU: Faster post-quantum TLS key exchange
Venue: arXiv cs.CR 2021
Year: 2021

Google's CECPQ1 experiment in 2016 integrated a post-quantum key-exchange algorithm, newhope1024, into TLS 1.2. The Google-Cloudflare CECPQ2 experiment in 2019 integrated a more efficient key-exchange algorithm, ntruhrss701, into TLS 1.3. This paper revisits the choices made in CECPQ2, and shows how to achieve higher performance for post-quantum key exchange in TLS 1.3 using a higher-security algorithm, sntrup761. Previous work had indicated that ntruhrss701 key generation was much faster than sntrup761 key generation, but this paper makes sntrup761 key generation much faster by generating a batch of keys at once. Batch key generation is invisible at the TLS protocol layer, but raises software-engineering questions regarding the difficulty of integrating batch key exchange into existing TLS libraries and applications. This paper shows that careful choices of software layers make it easy to integrate fast post-quantum software, including batch key exchange, into TLS with minor changes to TLS libraries and no changes to applications. As a demonstration of feasibility, this paper reports successful integration of its fast sntrup761 library, via a lightly patched OpenSSL, into an unmodified web browser and an unmodified TLS terminator. This paper also reports TLS 1.3 handshake benchmarks, achieving more TLS 1.3 handshakes per second than any software included in OpenSSL.
Unsupervised Person Re-identification via Multi-Label Prediction and Classification based on Graph-Structural Insight
Venue: arXiv cs.CV 2021
Year: 2021

This paper addresses unsupervised person re-identification (Re-ID) using multi-label prediction and classification based on graph-structural insight. Our method extracts features from person images and produces a graph that consists of the features and a pairwise similarity of them as nodes and edges, respectively. Based on the graph, the proposed graph structure based multi-label prediction (GSMLP) method predicts multi-labels by considering the pairwise similarity and the adjacency node distribution of each node. The multi-labels created by GSMLP are applied to the proposed selective multi-label classification (SMLC) loss. SMLC integrates a hard-sample mining scheme and a multi-label classification. The proposed GSMLP and SMLC boost the performance of unsupervised person Re-ID without any pre-labelled dataset. Experimental results justify the superiority of the proposed method in unsupervised person Re-ID by producing state-of-the-art performance. The source code for this paper is publicly available on 'https://github.com/uknownpioneer/GSMLP-SMLC.git'.
Federated Learning over Energy Harvesting Wireless Networks
Venue: arXiv cs.IT 2021
Year: 2021

In this paper, the deployment of federated learning (FL) is investigated in an energy harvesting wireless network in which the base station (BS) employs massive multiple-input multiple-output (MIMO) to serve a set of users powered by independent energy harvesting sources. Since a certain number of users may not be able to participate in FL due to the interference and energy constraints, a joint energy management and user scheduling problem in FL over wireless systems is formulated. This problem is formulated as an optimization problem whose goal is to minimize the FL training loss via optimizing user scheduling. To find how the factors such as transmit power and number of scheduled users affect the training loss, the convergence rate of the FL algorithm is first analyzed. Given this analytical result, the user scheduling and energy management optimization problem can be decomposed, simplified, and solved. Further, the system model is extended by considering multiple BSs. Hence, a joint user association and scheduling problem in FL over wireless systems is studied. The optimal user association problem is solved using the branch-and-bound technique. Simulation results show that the proposed user scheduling and user association algorithm can reduce training loss compared to a standard FL algorithm.
Opportunistic Relay in Multicast Channels with Generalized Shadowed Fading Effects: A Physical Layer Security Perspective
Venue: arXiv cs.IT 2021
Year: 2021

Through ordinary transmissions over wireless multicast networks are greatly hampered due to the simultaneous presence of fading and shadowing of wireless channels, secure transmissions can be enhanced by properly exploiting random attributes of the propagation medium. This study focuses on the utilization of those attributes to enhance the physical layer security (PLS) performance of a dual-hop wireless multicast network over kappa-mu shadow-fading channel under the wiretapping attempts of multiple eavesdroppers. In order to improve the secrecy level, the best relay selection strategy among multiple relays is employed. Performance analysis is carried out based on the mathematical modeling in terms of analytical expressions of non-zero secrecy capacity probability, secure outage probability, and ergodic secrecy capacity over multicast relay networks. Capitalizing on those expressions, the effects of system parameters, i.e., fading, shadowing, the number of antennas, destination receivers, eavesdroppers, and relays, on the secrecy performance are investigated. Numerical results show that the detrimental impacts caused by fading and shadowing can be remarkably mitigated using the well-known opportunistic relaying technique. Moreover, the proposed model unifies secrecy analysis of several classical models, thereby exhibiting enormous versatility than the existing works.
Costs and Benefits of Fair Regression
Venue: arXiv cs.LG 2021
Year: 2021

Real-world applications of machine learning tools in high-stakes domains are often regulated to be fair, in the sense that the predicted target should satisfy some quantitative notion of parity with respect to a protected attribute. However, the exact tradeoff between fairness and accuracy with a real-valued target is not entirely clear. In this paper, we characterize the inherent tradeoff between statistical parity and accuracy in the regression setting by providing a lower bound on the error of any fair regressor. Our lower bound is sharp, algorithm-independent, and admits a simple interpretation: when the moments of the target differ between groups, any fair algorithm has to make an error on at least one of the groups. We further extend this result to give a lower bound on the joint error of any (approximately) fair algorithm, using the Wasserstein distance to measure the quality of the approximation. With our novel lower bound, we also show that the price paid by a fair regressor that does not take the protected attribute as input is less than that of a fair regressor with explicit access to the protected attribute. On the upside, we establish the first connection between individual fairness, accuracy parity, and the Wasserstein distance by showing that if a regressor is individually fair, it also approximately verifies the accuracy parity, where the gap is given by the Wasserstein distance between the two groups. Inspired by our theoretical results, we develop a practical algorithm for fair regression through the lens of representation learning, and conduct experiments on a real-world dataset to corroborate our findings.
Concatenated Reed-Solomon and Polarization-Adjusted Convolutional (PAC) Codes
Venue: arXiv cs.IT 2021
Year: 2021

Two concatenated coding schemes incorporating algebraic Reed-Solomon (RS) codes and polarization-adjusted convolutional (PAC) codes are proposed. Simulation results show that at a bit error rate of $10^{-5}$, a concatenated scheme using RS and PAC codes has more than $0.25$ dB coding gain over the NASA standard concatenation scheme, which uses RS and convolutional codes.
Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation
Venue: arXiv cs.LG 2021
Year: 2021

State-of-the-art transformer models use pairwise dot-product based self-attention, which comes at a computational cost quadratic in the input sequence length. In this paper, we investigate the global structure of attention scores computed using this dot product mechanism on a typical distribution of inputs, and study the principal components of their variation. Through eigen analysis of full attention score matrices, as well as of their individual rows, we find that most of the variation among attention scores lie in a low-dimensional eigenspace. Moreover, we find significant overlap between these eigenspaces for different layers and even different transformer models. Based on this, we propose to compute scores only for a partial subset of token pairs, and use them to estimate scores for the remaining pairs. Beyond investigating the accuracy of reconstructing attention scores themselves, we investigate training transformer models that employ these approximations, and analyze the effect on overall accuracy. Our analysis and the proposed method provide insights into how to balance the benefits of exact pair-wise attention and its significant computational expense.
Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems
Venue: arXiv cs.CL 2021
Year: 2021

Dialogue policy optimisation via reinforcement learning requires a large number of training interactions, which makes learning with real users time consuming and expensive. Many set-ups therefore rely on a user simulator instead of humans. These user simulators have their own problems. While hand-coded, rule-based user simulators have been shown to be sufficient in small, simple domains, for complex domains the number of rules quickly becomes intractable. State-of-the-art data-driven user simulators, on the other hand, are still domain-dependent. This means that adaptation to each new domain requires redesigning and retraining. In this work, we propose a domain-independent transformer-based user simulator (TUS). The structure of our TUS is not tied to a specific domain, enabling domain generalisation and learning of cross-domain user behaviour from data. We compare TUS with the state of the art using automatic as well as human evaluations. TUS can compete with rule-based user simulators on pre-defined domains and is able to generalise to unseen domains in a zero-shot fashion.
Multilinear Dirichlet Processes
Venue: arXiv cs.LG 2021
Year: 2021

Dependent Dirichlet processes (DDP) have been widely applied to model data from distributions over collections of measures which are correlated in some way. On the other hand, in recent years, increasing research efforts in machine learning and data mining have been dedicated to dealing with data involving interactions from two or more factors. However, few researchers have addressed the heterogeneous relationship in data brought by modulation of multiple factors using techniques of DDP. In this paper, we propose a novel technique, MultiLinear Dirichlet Processes (MLDP), to constructing DDPs by combining DP with a state-of-the-art factor analysis technique, multilinear factor analyzers (MLFA). We have evaluated MLDP on real-word data sets for different applications and have achieved state-of-the-art performance.
Nonparametric Empirical Bayes Estimation and Testing for Sparse and Heteroscedastic Signals
Venue: arXiv cs.LG 2021
Year: 2021

Large-scale modern data often involves estimation and testing for high-dimensional unknown parameters. It is desirable to identify the sparse signals, ``the needles in the haystack'', with accuracy and false discovery control. However, the unprecedented complexity and heterogeneity in modern data structure require new machine learning tools to effectively exploit commonalities and to robustly adjust for both sparsity and heterogeneity. In addition, estimates for high-dimensional parameters often lack uncertainty quantification. In this paper, we propose a novel Spike-and-Nonparametric mixture prior (SNP) -- a spike to promote the sparsity and a nonparametric structure to capture signals. In contrast to the state-of-the-art methods, the proposed methods solve the estimation and testing problem at once with several merits: 1) an accurate sparsity estimation; 2) point estimates with shrinkage/soft-thresholding property; 3) credible intervals for uncertainty quantification; 4) an optimal multiple testing procedure that controls false discovery rate. Our method exhibits promising empirical performance on both simulated data and a gene expression case study.
Detecting chaos in lineage-trees: A deep learning approach
Venue: arXiv cs.LG 2021
Year: 2021

Many complex phenomena, from weather systems to heartbeat rhythm patterns, are effectively modeled as low-dimensional dynamical systems. Such systems may behave chaotically under certain conditions, and so the ability to detect chaos based on empirical measurement is an important step in characterizing and predicting these processes. Classifying a system as chaotic usually requires estimating its largest Lyapunov exponent, which quantifies the average rate of convergence or divergence of initially close trajectories in state space, and for which a positive value is generally accepted as an operational definition of chaos. Estimating the largest Lyapunov exponent from observations of a process is especially challenging in systems affected by dynamical noise, which is the case for many models of real-world processes, in particular models of biological systems. We describe a novel method for estimating the largest Lyapunov exponent from data, based on training Deep Learning models on synthetically generated trajectories, and demonstrate that this method yields accurate and noise-robust predictions given relatively short inputs and across a range of different dynamical systems. Our method is unique in that it can analyze tree-shaped data, a ubiquitous topology in biological settings, and specifically in dynamics over lineages of cells or organisms. We also characterize the types of input information extracted by our models for their predictions, allowing for a deeper understanding into the different ways by which chaos can be analyzed in different topologies.
Herd Behaviors in Epidemics: A Dynamics-Coupled Evolutionary Games Approach
Venue: arXiv cs.GT 2021
Year: 2021

The recent COVID-19 pandemic has led to an increasing interest in the modeling and analysis of infectious diseases. The pandemic has made a significant impact on the way we behave and interact in our daily life. The past year has witnessed a strong interplay between human behaviors and epidemic spreading. In this paper, we propose an evolutionary game-theoretic framework to study the coupled evolutions of herd behaviors and epidemics. Our framework extends the classical degree-based mean-field epidemic model over complex networks by coupling it with the evolutionary game dynamics. The statistically equivalent individuals in a population choose their social activity intensities based on the fitness or the payoffs that depend on the state of the epidemics. Meanwhile, the spreading of the infectious disease over the complex network is reciprocally influenced by the players' social activities. We analyze the coupled dynamics by studying the stationary properties of the epidemic for a given herd behavior and the structural properties of the game for a given epidemic process. The decisions of the herd turn out to be strategic substitutes. We formulate an equivalent finite-player game and an equivalent network to represent the interactions among the finite populations. We develop structure-preserving approximation techniques to study time-dependent properties of the joint evolution of the behavioral and epidemic dynamics. The resemblance between the simulated coupled dynamics and the real COVID-19 statistics in the numerical experiments indicates the predictive power of our framework.
"I have no idea what they're trying to accomplish:" Enthusiastic and Casual Signal Users' Understanding of Signal PINs
Venue: arXiv cs.CR 2021
Year: 2021

We conducted an online study with $n = 235$ Signal users on their understanding and usage of PINs in Signal. In our study, we observe a split in PIN management and composition strategies between users who can explain the purpose of the Signal PINs (56%; enthusiasts) and users who cannot (44%; casual users). Encouraging adoption of PINs by Signal appears quite successful: only 14% opted-out of setting a PIN entirely. Among those who did set a PIN, most enthusiasts had long, complex alphanumeric PINs generated by and saved in a password manager. Meanwhile more casual Signal users mostly relied on short numeric-only PINs. Our results suggest that better communication about the purpose of the Signal PIN could help more casual users understand the features PINs enable (such as that it is not simply a personal identification number). This communication could encourage a stronger security posture.
Cascading Modular Network (CAM-Net) for Multimodal Image Synthesis
Venue: arXiv cs.CV 2021
Year: 2021

Deep generative models such as GANs have driven impressive advances in conditional image synthesis in recent years. A persistent challenge has been to generate diverse versions of output images from the same input image, due to the problem of mode collapse: because only one ground truth output image is given per input image, only one mode of the conditional distribution is modelled. In this paper, we focus on this problem of multimodal conditional image synthesis and build on the recently proposed technique of Implicit Maximum Likelihood Estimation (IMLE). Prior IMLE-based methods required different architectures for different tasks, which limit their applicability, and were lacking in fine details in the generated images. We propose CAM-Net, a unified architecture that can be applied to a broad range of tasks. Additionally, it is capable of generating convincing high frequency details, achieving a reduction of the Frechet Inception Distance (FID) by up to 45.3% compared to the baseline.
Deformation Driven Seq2Seq Longitudinal Tumor and Organs-at-Risk Prediction for Radiotherapy
Venue: arXiv cs.CV 2021
Year: 2021

Purpose: Radiotherapy presents unique challenges and clinical requirements for longitudinal tumor and organ-at-risk (OAR) prediction during treatment. The challenges include tumor inflammation/edema and radiation-induced changes in organ geometry, whereas the clinical requirements demand flexibility in input/output sequence timepoints to update the predictions on rolling basis and the grounding of all predictions in relationship to the pre-treatment imaging information for response and toxicity assessment in adaptive radiotherapy. Methods: To deal with the aforementioned challenges and to comply with the clinical requirements, we present a novel 3D sequence-to-sequence model based on Convolution Long Short Term Memory (ConvLSTM) that makes use of series of deformation vector fields (DVF) between individual timepoints and reference pre-treatment/planning CTs to predict future anatomical deformations and changes in gross tumor volume as well as critical OARs. High-quality DVF training data is created by employing hyper-parameter optimization on the subset of the training data with DICE coefficient and mutual information metric. We validated our model on two radiotherapy datasets: a publicly available head-and-neck dataset (28 patients with manually contoured pre-, mid-, and post-treatment CTs), and an internal non-small cell lung cancer dataset (63 patients with manually contoured planning CT and 6 weekly CBCTs). Results: The use of DVF representation and skip connections overcomes the blurring issue of ConvLSTM prediction with the traditional image representation. The mean and standard deviation of DICE for predictions of lung GTV at week 4, 5, and 6 were 0.83$\pm$0.09, 0.82$\pm$0.08, and 0.81$\pm$0.10, respectively, and for post-treatment ipsilateral and contralateral parotids, were 0.81$\pm$0.06 and 0.85$\pm$0.02.
Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
Venue: arXiv cs.LG 2021
Year: 2021

Offline Reinforcement Learning (RL) aims to extract near-optimal policies from imperfect offline data without additional environment interactions. Extracting policies from diverse offline datasets has the potential to expand the range of applicability of RL by making the training process safer, faster, and more streamlined. We investigate how to improve the performance of offline RL algorithms, its robustness to the quality of offline data, as well as its generalization capabilities. To this end, we introduce Offline Model-based RL with Adaptive Behavioral Priors (MABE). Our algorithm is based on the finding that dynamics models, which support within-domain generalization, and behavioral priors, which support cross-domain generalization, are complementary. When combined together, they substantially improve the performance and generalization of offline RL policies. In the widely studied D4RL offline RL benchmark, we find that MABE achieves higher average performance compared to prior model-free and model-based algorithms. In experiments that require cross-domain generalization, we find that MABE outperforms prior methods. Our website is available at https://sites.google.com/berkeley.edu/mabe .