Miracle-dz/UniFAR-FTUs huggingface dataset:
UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost

Transformer architecture achieves great success in abundant natural language processing tasks. The over-parameterization of the Transformer model has motivated plenty of works to alleviate its overfitting for superior performances. With some explorations, we find simple techniques such as dropout, can greatly boost model performance with a careful design. Therefore, in this paper, we integrate different dropout techniques into the training of Transformer models. Specifically, we propose an approach named UniDrop to unites three different dropout techniques from fine-grain to coarse-grain, i.e., feature dropout, structure dropout, and data dropout. Theoretically, we demonstrate that these three dropouts play different roles from regularization perspectives. Empirically, we conduct experiments on both neural machine translation and text classification benchmark datasets. Extensive results indicate that Transformer with UniDrop can achieve around 1.5 BLEU improvement on IWSLT14 translation tasks, and better accuracy for the classification even using strong pre-trained RoBERTa as backbone.
Backtranslation Feedback Improves User Confidence in MT, Not Quality

Translating text into a language unknown to the text’s author, dubbed outbound translation, is a modern need for which the user experience has significant room for improvement, beyond the basic machine translation facility. We demonstrate this by showing three ways in which user confidence in the outbound translation, as well as its overall final quality, can be affected: backward translation, quality estimation (with alignment) and source paraphrasing. In this paper, we describe an experiment on outbound translation from English to Czech and Estonian. We examine the effects of each proposed feedback module and further focus on how the quality of machine translation systems influence these findings and the user perception of success. We show that backward translation feedback has a mixed effect on the whole process: it increases user confidence in the produced translation, but not the objective quality.
Stochastic Stability of Discrete-time Phase-coupled Oscillators over Uncertain and Random Networks

This article studies stochastic relative phase stability, i.e., stochastic phase-cohesiveness, of discrete-time phase-coupled oscillators. Stochastic phase-cohesiveness in two types of networks is studied. First, we consider oscillators coupled with 2{\pi}-periodic odd functions over underlying undirected graphs subject to both multiplicative and additive stochastic uncertainties. We prove stochastic phase-cohesiveness of the network with respect to two specific, namely in-phase and anti-phase, sets by deriving sufficient coupling conditions. We show the dependency of these conditions on the size of the mean values of additive and multiplicative uncertainties, as well as the sign of the mean values of multiplicative uncertainties. Furthermore, we discuss the results under a relaxation of the odd property of the coupling function. Second, we study an uncertain network in which the multiplicative uncertainties are governed by the Bernoulli process representing the well-known Erdos-Renyi network. We assume constant exogenous frequencies and derive sufficient conditions for achieving both stochastic phase-cohesive and phase-locked solutions, i.e., stochastic phase-cohesiveness with respect to the origin. For the latter case, where identical exogenous frequencies are assumed, we prove that any positive probability of connectivity leads to phase-locking. Thorough analyses are provided, and insights obtained from stochastic analysis are discussed, along with numerical simulations to validate the analytical results.
Demand-pull and technology-push: What drives the direction of technological change? An empirical network-based approach

Demand-pull and technology-push are drivers of technological change and policy-makers need to understand how both interact and differ by impact. I introduce two concepts of demand-pull and technology-push measured by a two-layer network of input-output (market) and patent citation (innovation) links between 307 NAICS 6-digit US manufacturing industries in 1977-2012: (1) Demand-pull and technology-push are cross-layer spillovers when demand shocks in the market pull innovation and innovation pushes growth in the market. (2) Demand-pull may also arise from downstream links within the same layer when output users trigger upstream growth. Push effects, in contrast, spill over from upto downstream industries. The results show that innovation is a driver of market growth which is factor-biased in favor of capital against labor. I also find support for demandpull within the market: industries with a strong customer network grow faster, invest more, and grow by labor productivity. Upstream centrality exhibits the opposite effect which may indicate input constraints in the market. Innovation evolves differently: knowledge spillovers give rise to increasing returns as driver of concentration and clustering. Despite limitations related to data and classifications, the results enable a differentiated view on the drivers of technological change and its consequences which is essential for policy to shape the technological evolution.
ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning

While relation extraction is an essential task in knowledge acquisition and representation, and new-generated relations are common in the real world, less effort is made to predict unseen relations that cannot be observed at the training stage. In this paper, we formulate the zero-shot relation extraction problem by incorporating the text description of seen and unseen relations. We propose a novel multi-task learning model, Zero-Shot BERT (ZS-BERT), to directly predict unseen relations without hand-crafted attribute labeling and multiple pairwise classifications. Given training instances consisting of input sentences and the descriptions of their seen relations, ZS-BERT learns two functions that project sentences and relations into an embedding space by jointly minimizing the distances between them and classifying seen relations. By generating the embeddings of unseen relations and new-coming sentences based on such two functions, we use nearest neighbor search to obtain the prediction of unseen relations. Experiments conducted on two well-known datasets exhibit that ZS-BERT can outperform existing methods by at least 13.54% improvement on F1 score.
GPflux: A Library for Deep Gaussian Processes

We introduce GPflux, a Python library for Bayesian deep learning with a strong emphasis on deep Gaussian processes (DGPs). Implementing DGPs is a challenging endeavour due to the various mathematical subtleties that arise when dealing with multivariate Gaussian distributions and the complex bookkeeping of indices. To date, there are no actively maintained, open-sourced and extendable libraries available that support research activities in this area. GPflux aims to fill this gap by providing a library with state-of-the-art DGP algorithms, as well as building blocks for implementing novel Bayesian and GP-based hierarchical models and inference schemes. GPflux is compatible with and built on top of the Keras deep learning eco-system. This enables practitioners to leverage tools from the deep learning community for building and training customised Bayesian models, and create hierarchical models that consist of Bayesian and standard neural network layers in a single coherent framework. GPflux relies on GPflow for most of its GP objects and operations, which makes it an efficient, modular and extensible library, while having a lean codebase.
Priority Lists for Power System Investments: Locating Phasor Measurement Units

Power systems incrementally and continuously upgrade their components, such as transmission lines, reactive capacitors, or generating units. Decision-making tools often support the selection of the best set of components to upgrade. Optimization models are often used to support decision making at a given point in time. After certain time intervals, re-optimization is performed to find new components to add. In this paper, we propose a decision-making framework for incrementally updating power system components. This is an alternative approach to the classical sequential re-optimization decision making for an investment problem with modeled budget constraints. Our approach provides a priority list as a solution with a list of new components to upgrade. We show that i) our framework is consistent with the evolution of power system upgrades, and ii) in particular circumstances, both frameworks provide the same solution if the problem satisfies submodularity property.We have selected the problem of phasor measurement unit localization and compared the solution with the classical sequential re-optimization framework. For this particular problem, we show that the two approaches provide close results, while only our proposed algorithm is applicable in practice. The cases of 14 and 118 IEEE buses are used to illustrate the proposed methodology.
Noether: The More Things Change, the More Stay the Same

Symmetries have proven to be important ingredients in the analysis of neural networks. So far their use has mostly been implicit or seemingly coincidental. We undertake a systematic study of the role that symmetry plays. In particular, we clarify how symmetry interacts with the learning algorithm. The key ingredient in our study is played by Noether's celebrated theorem which, informally speaking, states that symmetry leads to conserved quantities (e.g., conservation of energy or conservation of momentum). In the realm of neural networks under gradient descent, model symmetries imply restrictions on the gradient path. E.g., we show that symmetry of activation functions leads to boundedness of weight matrices, for the specific case of linear activations it leads to balance equations of consecutive layers, data augmentation leads to gradient paths that have"momentum"-type restrictions, and time symmetry leads to a version of the Neural Tangent Kernel. Symmetry alone does not specify the optimization path, but the more symmetries are contained in the model the more restrictions are imposed on the path. Since symmetry also implies over-parametrization, this in effect implies that some part of this over-parametrization is cancelled out by the existence of the conserved quantities. Symmetry can therefore be thought of as one further important tool in understanding the performance of neural networks under gradient descent.
Practical Two-party Privacy-preserving Neural Network Based on Secret Sharing

Neural networks, with the capability to provide efficient predictive models, have been widely used in medical, financial, and other fields, bringing great convenience to our lives. However, the high accuracy of the model requires a large amount of data from multiple parties, raising public concerns about privacy. Privacy-preserving neural network based on multi-party computation is one of the current methods used to provide model training and inference under the premise of solving data privacy. In this study, we propose a new two-party privacy-preserving neural network training and inference framework in which privacy data is distributed to two non-colluding servers. We construct a preprocessing protocol for mask generation, support and realize secret sharing comparison on 2PC, and propose a new method to further reduce the communication rounds. Based on the comparison protocol, we construct building blocks such as division and exponential, and realize the process of training and inference that no longer needs to convert between different types of secret sharings and is entirely based on arithmetic secret sharing. Compared with the previous works, our work obtains higher accuracy, which is very close to that of plaintext training. While the accuracy has been improved, the runtime is reduced, considering the online phase, our work is 5x faster than SecureML, 4.32-5.75x faster than SecureNN, and is very close to the current optimal 3PC implementation, FALCON. For secure inference, as far as known knowledge is concerned, we should be the current optimal 2PC implementation, which is 4-358x faster than other works.
What Kind of Person Wins the Turing Award?

Computer science has grown rapidly since its inception in the 1950s and the pioneers in the field are celebrated annually by the A.M. Turing Award. In this paper, we attempt to shed light on the path to influential computer scientists by examining the characteristics of the 72 Turing Award laureates. To achieve this goal, we build a comprehensive dataset of the Turing Award laureates and analyze their characteristics, including their personal information, family background, academic background, and industry experience. The FP-Growth algorithm is used for frequent feature mining. Logistic regression plot, pie chart, word cloud and map are generated accordingly for each of the interesting features to uncover insights regarding personal factors that drive influential work in the field of computer science. In particular, we show that the Turing Award laureates are most commonly white, male, married, United States citizen, and received a PhD degree. Our results also show that the age at which the laureate won the award increases over the years; most of the Turing Award laureates did not major in computer science; birth order is strongly related to the winners' success; and the number of citations is not as important as one would expect.
Complex Spectral Mapping With Attention Based Convolution Recurrent Neural Network for Speech Enhancement

Speech enhancement has benefited from the success of deep learning in terms of intelligibility and perceptual quality. Conventional time-frequency (TF) domain methods focus on predicting TF-masks or speech spectrum,via a naive convolution neural network or recurrent neural network.Some recent studies were based on Complex spectral Mapping convolution recurrent neural network (CRN) . These models skiped directly from encoder layers' output and decoder layers' input ,which maybe thoughtless. We proposed an attention mechanism based skip connection between encoder and decoder layers,namely Complex Spectral Mapping With Attention Based Convolution Recurrent Neural Network (CARN). Compared with CRN model,the proposed CARN model improved more than 10% relatively at several metrics such as PESQ,CBAK,COVL,CSIG and son,and outperformed the place 1st model in both real time and non-real time track of the DNS Challenge 2020 at these metrics.
Higher-order percolation processes on multiplex hypergraphs

Higher-order interactions are increasingly recognized as a fundamental aspect of complex systems ranging from the brain to social contact networks. Hypergraphs as well as simplicial complexes capture the higher-order interactions of complex systems and allow us to investigate the relation between their higher-order structure and their function. Here we establish a general framework for assessing hypergraph robustness and we characterize the critical properties of simple and higher-order percolation processes. This general framework builds on the formulation of the random multiplex hypergraph ensemble where each layer is characterized by hyperedges of given cardinality. We observe that in presence of the structural cutoff the ensemble of multiplex hypergraphs can be mapped to an ensemble of multiplex bipartite networks. We reveal the relation between higher-order percolation processes in random multiplex hypergraphs, interdependent percolation of multiplex networks, and K-core percolation. The structural correlations of the random multiplex hypergraphs are shown to have a significant effect on their percolation properties. The wide range of critical behaviors observed for higher-order percolation processes on multiplex hypergraphs elucidates the mechanisms responsible for the emergence of discontinuous transition and uncovers interesting critical properties which can be applied to the study of epidemic spreading and contagion processes on higher-order networks.
A Preliminary Model for the Design of Music Visualizations

Music Visualization is basically the transformation of data from the aural to the visual space. There are a variety of music visualizations, across applications, present on the web. Models of Visualization include conceptual frameworks helpful for designing, understanding and making sense of visualizations. In this paper, we propose a preliminary model for Music Visualization. We build the model by using two conceptual pivots, Visualization Stimulus and Data Property. To demonstrate the utility of the model we deconstruct and design visualizations with toy examples using the model and finally conclude by proposing further applications of and future work on our proposed model.
Spiked eigenvalues of noncentral Fisher matrix with applications

In this paper, we investigate the asymptotic behavior of spiked eigenvalues of the noncentral Fisher matrix defined by ${\mathbf F}_p={\mathbf C}_n(\mathbf S_N)^{-1}$, where ${\mathbf C}_n$ is a noncentral sample covariance matrix defined by $(\mathbf \Xi+\mathbf X)(\mathbf \Xi+\mathbf X)^*/n$ and $\mathbf S_N={\mathbf Y}{\mathbf Y}^*/N$. The matrices $\mathbf X$ and $\mathbf Y$ are two independent {Gaussian} arrays, with respective $p\times n$ and $p\times N$ and the Gaussian entries of them are \textit {independent and identically distributed} (i.i.d.) with mean $0$ and variance $1$. When $p$, $n$, and $N$ grow to infinity proportionally, we establish a phase transition of the spiked eigenvalues of $\mathbf F_p$. Furthermore, we derive the \textit{central limiting theorem} (CLT) for the spiked eigenvalues of $\mathbf F_p$. As an accessory to the proof of the above results, the fluctuations of the spiked eigenvalues of ${\mathbf C}_n$ are studied, which should have its own interests. Besides, we develop the limits and CLT for the sample canonical correlation coefficients by the results of the spiked noncentral Fisher matrix and give three consistent estimators, including the population spiked eigenvalues and the population canonical correlation coefficients.
Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog

Semantic parsing using sequence-to-sequence models allows parsing of deeper representations compared to traditional word tagging based models. In spite of these advantages, widespread adoption of these models for real-time conversational use cases has been stymied by higher compute requirements and thus higher latency. In this work, we propose a non-autoregressive approach to predict semantic parse trees with an efficient seq2seq model architecture. By combining non-autoregressive prediction with convolutional neural networks, we achieve significant latency gains and parameter size reduction compared to traditional RNN models. Our novel architecture achieves up to an 81% reduction in latency on TOP dataset and retains competitive performance to non-pretrained models on three different semantic parsing datasets.
Learning from Censored and Dependent Data: The case of Linear Dynamics

Observations from dynamical systems often exhibit irregularities, such as censoring, where values are recorded only if they fall within a certain range. Censoring is ubiquitous in practice, due to saturating sensors, limit-of-detection effects, and image-frame effects. In light of recent developments on learning linear dynamical systems (LDSs), and on censored statistics with independent data, we revisit the decades-old problem of learning an LDS, from censored observations (Lee and Maddala (1985); Zeger and Brookmeyer (1986)). Here, the learner observes the state $x_t \in \mathbb{R}^d$ if and only if $x_t$ belongs to some set $S_t \subseteq \mathbb{R}^d$. We develop the first computationally and statistically efficient algorithm for learning the system, assuming only oracle access to the sets $S_t$. Our algorithm, Stochastic Online Newton with Switching Gradients, is a novel second-order method that builds on the Online Newton Step (ONS) of Hazan et al. (2007). Our Switching-Gradient scheme does not always use (stochastic) gradients of the function we want to optimize, which we call"censor-aware"function. Instead, in each iteration, it performs a simple test to decide whether to use the censor-aware, or another"censor-oblivious"function, for getting a stochastic gradient. In our analysis, we consider a"generic"Online Newton method, which uses arbitrary vectors instead of gradients, and we prove an error-bound for it. This can be used to appropriately design these vectors, leading to our Switching-Gradient scheme. This framework significantly deviates from the recent long line of works on censored statistics (e.g., Daskalakis et al. (2018); Kontonis et al. (2019); Daskalakis et al. (2019)), which apply Stochastic Gradient Descent (SGD), and their analysis reduces to establishing conditions for off-the-shelf SGD-bounds.
Impact of Electric Vehicle Routing With Stochastic Demand on Grid Operation

Given the rise of electric vehicle (EV) adoption, supported by government policies and dropping technology prices, new challenges arise in the modeling and operation of electric transportation. In this paper, we present a model for solving the EV routing problem while accounting for real-life stochastic demand behavior. We present a mathematical formulation that minimizes travel time and energy costs of a EV fleet. The EV is represented by a battery energy consumption model. To adapt our formulation to real-life scenarios, customer pick-ups and drop-offs were modeled as stochastic parameters. A chance-constrained optimization model is proposed for addressing pick-ups and drop-offs uncertainties. Computational validation of the model is provided based on representative transportation scenarios. Results obtained showed a quick convergence of our model with verifiable solutions. Finally, the impact of electric vehicles charging is validated in Downtown Manhattan, New York by assessing the effect on the distribution grid.
Some Lessons Learned Running Virtual Reality Experiments Out of the Laboratory

In the past twelve months, our team has had to move rapidly from conducting most of our user experiments in a laboratory setting, to running experiments in the wild away from the laboratory and without direct synchronous oversight from an experimenter. This has challenged us to think about what types of experiment we can run, and to improve our tools and methods to allow us to reliably capture the necessary data. It has also offered us an opportunity to engage with a more diverse population than we would normally engage with in the laboratory. In this position paper we elaborate on the challenges and opportunities, and give some lessons learned from our own experience.
The Atari Data Scraper

Reinforcement learning has made great strides in recent years due to the success of methods using deep neural networks. However, such neural networks act as a black box, obscuring the inner workings. While reinforcement learning has the potential to solve unique problems, a lack of trust and understanding of reinforcement learning algorithms could prevent their widespread adoption. Here, we present a library that attaches a “data scraper” to deep reinforcement learning agents, acting as an observer, and then show how the data collected by the Atari Data Scraper can be used to understand and interpret deep reinforcement learning agents. The code for the Atari Data Scraper can be found here: https: // github. com/ IRLL/ Atari-Data-Scraper .