Reviewerly/exHarmony huggingface dataset:
Barron Spaces and the Compositional Function Spaces for Neural Network Models.

One of the key issues in the analysis of machine learning models is to identify the appropriate function space for the model. This is the space of functions that the particular machine learning model can approximate with good accuracy, endowed with a natural norm associated with the approximation process. In this paper, we address this issue for two representative neural network models: the two-layer networks and the residual neural networks. We define Barron space and show that it is the right space for two-layer neural network models in the sense that optimal direct and inverse approximation theorems hold for functions in the Barron space. For residual neural network models, we construct the so-called compositional function space, and prove direct and inverse approximation theorems for this space. In addition, we show that the Rademacher complexity has the optimal upper bounds for these spaces.
Integer Discrete Flows and Lossless Compression

Lossless compression methods shorten the expected representation size of data without loss of information, using a statistical model. Flow-based models are attractive in this setting because they admit exact likelihood optimization, which is equivalent to minimizing the expected number of bits per message. However, conventional flows assume continuous data, which may lead to reconstruction errors when quantized for compression. For that reason, we introduce a flow-based generative model for ordinal discrete data called Integer Discrete Flow (IDF): a bijective integer map that can learn rich transformations on high-dimensional data. As building blocks for IDFs, we introduce a flexible transformation layer called integer discrete coupling. Our experiments show that IDFs are competitive with other flow-based generative models. Furthermore, we demonstrate that IDF based compression achieves state-of-the-art lossless compression rates on CIFAR10, ImageNet32, and ImageNet64. To the best of our knowledge, this is the first lossless compression method that uses invertible neural networks.
Privacy-Preserving Deep Learning on Machine Learning as a Service—a Comprehensive Survey

The exponential growth of big data and deep learning has increased the data exchange traffic in society. Machine Learning as a Service, (MLaaS) which leverages deep learning techniques for predictive analytics to enhance decision-making, has become a hot commodity. However, the adoption of MLaaS introduces data privacy challenges for data owners and security challenges for deep learning model owners. Data owners are concerned about the safety and privacy of their data on MLaaS platforms, while MLaaS platform owners worry that their models could be stolen by adversaries who pose as clients. Consequently, Privacy-Preserving Deep Learning (PPDL) arises as a possible solution to this problem. Recently, several papers about PPDL for MLaaS have been published. However, to the best of our knowledge, no previous paper has summarized the existing literature on PPDL and its specific applicability to the MLaaS environment. In this paper, we present a comprehensive survey of privacy-preserving techniques, starting from classical privacy-preserving techniques to well-known deep learning techniques. Additionally, we present a detailed description of PPDL and address the issue of using PPDL for MLaaS. Furthermore, we undertake detailed comparisons between state-of-the-art PPDL methods. Subsequently, we classify an adversarial model on PPDL by highlighting possible PPDL attacks and their potential solutions. Ultimately, our paper serves as a single point of reference for detailed knowledge on PPDL and its applicability to MLaaS environments for both new and experienced researchers.
Understanding tables with intermediate pre-training

Table entailment, the binary classification task of finding if a sentence is supported or refuted by the content of a table, requires parsing language and table structure as well as numerical and discrete reasoning.While there is extensive work on textual entailment, table entailment is less well studied.We adapt TAPAS (Herzig et al., 2020), a table-based BERT model, to recognize entailment.Motivated by the benefits of data augmentation, we create a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to finetuning.This new data is not only useful for table entailment, but also for SQA (Iyyer et al., 2017), a sequential table QA task.To be able to use long examples as input of BERT models, we evaluate table pruning techniques as a pre-processing step to drastically improve the training and prediction efficiency at a moderate drop in accuracy.The different methods set the new state-of-the-art on the TAB-FACT (Chen et al., 2020) and SQA datasets.
Multiview Learning With Generalized Eigenvalue Proximal Support Vector Machines

Generalized eigenvalue proximal support vector machines (GEPSVMs) are a simple and effective binary classification method in which each hyperplane is closest to one of the two classes and as far as possible from the other class. They solve a pair of generalized eigenvalue problems to obtain two nonparallel hyperplanes. Multiview learning considers learning with multiple feature sets to improve the learning performance. In this paper, we propose multiview GEPSVMs (MvGSVMs) which effectively combine two views by introducing a multiview co-regularization term to maximize the consensus on distinct views, and skillfully transform a complicated optimization problem to a simple generalized eigenvalue problem. We also propose multiview improved GEPSVMs (MvIGSVMs), which use the minus instead of ratio in MvGSVMs to measure the differences of the distances between the two classes and the hyperplane and lead to a simpler eigenvalue problem. Linear MvGSVMs and MvIGSVMs are generalized to the nonlinear case by the kernel trick. Experimental results on multiple data sets show the effectiveness of our proposed approaches.
Hybrid Regularized Echo State Network for Multivariate Chaotic Time Series Prediction

Multivariate chaotic time series prediction is a hot research topic, the goal of which is to predict the future of the time series based on past observations. Echo state networks (ESNs) have recently been widely used in time series prediction, but there may be an ill-posed problem for a large number of unknown output weights. To solve this problem, we propose a hybrid regularized ESN, which employs a sparse regression with the L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/2</sub> regularization and the L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> regularization to compute the output weights. The L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/2</sub> penalty shows many attractive properties, such as unbiasedness and sparsity. The L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> penalty presents appealing ability on shrinking the amplitude of the output weights. After the output weights are calculated, the input weights, internal weights, and output weights are fine-tuning by a Hessian-free optimization method-conjugate gradient backpropagation algorithm. The fine-tuning helps to bubble up the input information toward the output layer. Besides, the largest Lyapunov exponent is used to calculate the predictable horizon of a chaotic time series. Experimental results on benchmark and real-world datasets show that our proposed method is superior to other ESN-based models, as sparser, smaller-absolute-value, and more informative output weights are obtained. All of the predictions within the predictable horizon of the proposed model are accurate.
Convolution-Based Neural Attention With Applications to Sentiment Classification

Neural attention mechanism has achieved many successes in various tasks in natural language processing. However, existing neural attention models based on a densely connected network are loosely related to the attention mechanism found in psychology and neuroscience. Motivated by the finding in neuroscience that human possesses the template-searching attention mechanism, we propose to use convolution operation to simulate attentions and give a mathematical explanation of our neural attention model. We then introduce a new network architecture, which combines a recurrent neural network with our convolution-based attention model and further stacks an attention-based neural model to build a hierarchical sentiment classification model. The experimental results show that our proposed models can capture salient parts of the text to improve the performance of sentiment classification at both the sentence level and the document level.
Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernels

The performance of multi-task learning in Convolutional Neural Networks (CNNs) hinges on the design of feature sharing between tasks within the architecture. The number of possible sharing patterns are combinatorial in the depth of the network and the number of tasks, and thus hand-crafting an architecture, purely based on the human intuitions of task relationships can be time-consuming and suboptimal. In this paper, we present a probabilistic approach to learning task-specific and shared representations in CNNs for multi-task learning. Specifically, we propose "stochastic filter groups" (SFG), a mechanism to assign convolution kernels in each layer to "specialist" and "generalist" groups, which are specific to and shared across different tasks, respectively. The SFG modules determine the connectivity between layers and the structures of task-specific and shared representations in the network. We employ variational inference to learn the posterior distribution over the possible grouping of kernels and network parameters. Experiments demonstrate the proposed method generalises across multiple tasks and shows improved performance over baseline methods.
SAR2SAR: A Semi-Supervised Despeckling Algorithm for SAR Images

Speckle reduction is a key step in many remote sensing applications. By strongly affecting synthetic aperture radar (SAR) images, it makes them difficult to analyse. Due to the difficulty to model the spatial correlation of speckle, a deep learning algorithm with self-supervision is proposed in this paper: SAR2SAR. Multi-temporal time series are leveraged and the neural network learns to restore SAR images by only looking at noisy acquisitions. To this purpose, the recently proposed noise2noise framework has been employed. The strategy to adapt it to SAR despeckling is presented, based on a compensation of temporal changes and a loss function adapted to the statistics of speckle. A study with synthetic speckle noise is presented to compare the performances of the proposed method with other state-of-the-art filters. Then, results on real images are discussed, to show the potential of the proposed algorithm. The code is made available to allow testing and reproducible research in this field.
Improved Geometric Path Enumeration for Verifying ReLU Neural Networks

Neural networks provide quick approximations to complex functions, and have been increasingly used in perception as well as control tasks. For use in mission-critical and safety-critical applications, however, it is important to be able to analyze what a neural network can and cannot do. For feed-forward neural networks with ReLU activation functions, although exact analysis is NP-complete, recently-proposed verification methods can sometimes succeed. The main practical problem with neural network verification is excessive analysis runtime. Even on small networks, tools that are theoretically complete can sometimes run for days without producing a result. In this paper, we work to address the runtime problem by improving upon a recently-proposed geometric path enumeration method. Through a series of optimizations, several of which are new algorithmic improvements, we demonstrate significant speed improvement of exact analysis on the well-studied ACAS Xu benchmarks, sometimes hundreds of times faster than the original implementation. On more difficult benchmark instances, our optimized approach is often the fastest, even outperforming inexact methods that leverage overapproximation and refinement.
Emoticon Style: Interpreting Differences in Emoticons Across Cultures

Emoticons are a key aspect of text-based communication, and are the equivalent of nonverbal cues to the medium of online chat, forums, and social media like Twitter. As emoticons become more widespread in computer mediated communication, a vocabulary of different symbols with subtle emotional distinctions emerges especially across different cultures. In this paper, we investigate the semantic, cultural, and social aspects of emoticon usage on Twitter and show that emoticons are not limited to conveying a specific emotion or used as jokes, but rather are socio-cultural norms, whose meaning can vary depending on the identity of the speaker. We also demonstrate how these norms propagate through the Twitter @-reply network. We confirm our results on a large-scale dataset of over one billion Tweets from different time periods and countries.
On classification trees and random forests in corpus linguistics: Some words of caution and suggestions for improvement

Abstract This paper is a discussion of methodological problems that (can) arise in the analysis of multifactorial data analyzed with tree-based or forest-based classifiers in (corpus) linguistics. I showcase a data set that highlights where such methods can fail at providing optimal results and then discuss solutions to this problem as well as the interpretation of random forests more generally.
RecSys-DAN: Discriminative Adversarial Networks for Cross-Domain Recommender Systems

Data sparsity and data imbalance are practical and challenging issues in cross-domain recommender systems (RSs). This paper addresses those problems by leveraging the concepts which derive from representation learning, adversarial learning, and transfer learning (particularly, domain adaptation). Although various transfer learning methods have shown promising performance in this context, our proposed novel method RecSys-DAN focuses on alleviating the cross-domain and within-domain data sparsity and data imbalance and learns transferable latent representations for users, items, and their interactions. Different from the existing approaches, the proposed method transfers the latent representations from a source domain to a target domain in an adversarial way. The mapping functions in the target domain are learned by playing a min-max game with an adversarial loss, aiming to generate domain indistinguishable representations for a discriminator. Four neural architectural instances of ResSys-DAN are proposed and explored. Empirical results on real-world Amazon data show that, even without using labeled data (i.e., ratings) in the target domain, RecSys-DAN achieves competitive performance as compared to the state-of-the-art supervised methods. More importantly, RecSys-DAN is highly flexible to both unimodal and multimodal scenarios, and thus it is more robust to the cold-start recommendation which is difficult for the previous methods.
Cybersickness in Head-Mounted Displays Is Caused by Differences in the User's Virtual and Physical Head Pose

Sensory conflict, eye-movement, and postural instability theories each have difficulty accounting for the motion sickness experienced during head-mounted display based virtual reality (HMD VR). In this paper we review the limitations of existing theories in explaining cybersickness and propose a practical alternative approach. We start by providing a clear operational definition of provocative motion stimulation during active HMD VR. In this situation, whenever the user makes a head movement, his/her virtual head will tend to trail its true position and orientation due to the display lag (or motion to photon latency). Importantly, these differences in virtual and physical head pose (DVP) will vary over time. Based on our own research findings, we propose that cybersickness in HMD VR is triggered by large magnitude, time-varying patterns of DVP. We then show how this hypothesis can be tested by: 1) systematically manipulating display lag magnitudes and head movement speeds across HMD VR conditions; and 2) comparing the user’s estimates of DVP and cybersickness produced in each of these conditions. We believe that this approach will allow researchers to precisely predict which situations will (and will not) be provocative for cybersickness in HMD VR..
Projection-based runtime assertions for testing and debugging Quantum programs

In this paper, we propose Proq, a runtime assertion scheme for testing and debugging quantum programs on a quantum computer. The predicates in Proq are represented by projections (or equivalently, closed subspaces of the state space), following Birkhoff-von Neumann quantum logic. The satisfaction of a projection by a quantum state can be directly checked upon a small number of projective measurements rather than a large number of repeated executions. On the theory side, we rigorously prove that checking projection-based assertions can help locate bugs or statistically assure that the semantic function of the tested program is close to what we expect, for both exact and approximate quantum programs. On the practice side, we consider hardware constraints and introduce several techniques to transform the assertions, making them directly executable on the measurement-restricted quantum computers. We also propose to achieve simplified assertion implementation using local projection technique with soundness guaranteed. We compare Proq with existing quantum program assertions and demonstrate the effectiveness and efficiency of Proq by its applications to assert two sophisticated quantum algorithms, the Harrow-Hassidim-Lloyd algorithm and Shor’s algorithm.
Optimal User-Edge Assignment in Hierarchical Federated Learning Based on Statistical Properties and Network Topology Constraints

Distributed learning algorithms aim to leverage distributed and diverse data stored at users' devices to learn a global phenomena by performing training amongst participating devices and periodically aggregating their local models' parameters into a global model. Federated learning is a promising paradigm that allows for extending local training among the participant devices before aggregating the parameters, offering better communication efficiency. However, in the cases where the participants' data are strongly skewed (i.e., non-IID), the local models can overfit local data, leading to low performing global model. In this paper, we first show that a major cause of the performance drop is the weighted distance between the distribution over classes on users' devices and the global distribution. Then, to face this challenge, we leverage the edge computing paradigm to design a hierarchical learning system that performs Federated Gradient Descent on the user-edge layer and Federated Averaging on the edge-cloud layer. In this hierarchical architecture, we formalize and optimize this user-edge assignment problem such that edge-level data distributions turn to be similar (i.e., close to IID), which enhances the Federated Averaging performance. Our experiments on multiple real-world datasets show that the proposed optimized assignment is tractable and leads to faster convergence of models towards a better accuracy value.
Financial literacy or investment experience: which is more influential in cryptocurrency investment?

Purpose The purpose of this study was to investigate how financial literacy and investment experience impact cryptocurrency investment behavior and explore which factor is more influential in cryptocurrency investment. Design/methodology/approach Using a sample of US individual investors from the 2018 National Financial Capability Study Investor Survey, a three-step hierarchical logistic regression was conducted following a model-comparison approach. In addition, a mediation analysis was conducted using the Karlson−Holm−Breen (KHB) method to further explore the mediating effect of investment experience between financial literacy and cryptocurrency investment. Findings This study found that while both financial literacy and investment experience were positively associated with investing in cryptocurrencies, investment experience was more influential in cryptocurrency investment. The findings also demonstrated that investment experience, especially risky asset holding, had a significant mediating effect between subjective financial knowledge and cryptocurrency investment behavior. Practical implications The findings of this study offer insight to researchers by providing a deeper understanding of the determinants of cryptocurrency investment in the United States. This study also provides detailed implications for financial institutions, financial professionals and policymakers to guide rational cryptocurrency investment behavior. Originality/value This study is one of the initial attempts to explore the determinant factors in cryptocurrency investment, an area that has rarely been studied in the literature.
Entanglement transport and a nanophotonic interface for atoms in optical tweezers

The realization of an efficient quantum optical interface for multi-qubit systems is an outstanding challenge in science and engineering. Using two atoms in individually controlled optical tweezers coupled to a nanofabricated photonic crystal cavity, we demonstrate entanglement generation, fast nondestructive readout, and full quantum control of atomic qubits. The entangled state is verified in free space after being transported away from the cavity by encoding the qubits into long-lived states and using dynamical decoupling. Our approach bridges quantum operations at an optical link and in free space with a coherent one-way transport, potentially enabling an integrated optical interface for atomic quantum processors.
Temporally Identity-Aware SSD With Attentional LSTM

Temporal object detection has attracted significant attention, but most popular detection methods cannot leverage rich temporal information in videos. Very recently, many algorithms have been developed for video detection task, yet very few approaches can achieve real-time online object detection in videos. In this paper, based on the attention mechanism and convolutional long short-term memory (ConvLSTM), we propose a temporal single-shot detector (TSSD) for real-world detection. Distinct from the previous methods, we take aim at temporally integrating pyramidal feature hierarchy using ConvLSTM, and design a novel structure, including a low-level temporal unit as well as a high-level one for multiscale feature maps. Moreover, we develop a creative temporal analysis unit, namely, attentional ConvLSTM, in which a temporal attention mechanism is specially tailored for background suppression and scale suppression, while a ConvLSTM integrates attention-aware features across time. An association loss and a multistep training are designed for temporal coherence. Besides, an online tubelet analysis (OTA) is exploited for identification. Our framework is evaluated on ImageNet VID dataset and 2DMOT15 dataset. Extensive comparisons on the detection and tracking capability validate the superiority of the proposed approach. Consequently, the developed TSSD-OTA achieves a fast speed and an overall competitive performance in terms of detection and tracking. Finally, a real-world maneuver is conducted for underwater object grasping.