OpenRaiser/Intern-Atlas huggingface dataset:
Knowledge Graph Embeddings in Geometric Algebras
Venue: arXiv cs.LG 2021
Year: 2021

Knowledge graph (KG) embedding aims at embedding entities and relations in a KG into a lowdimensional latent representation space. Existing KG embedding approaches model entities andrelations in a KG by utilizing real-valued , complex-valued, or hypercomplex-valued (Quaternionor Octonion) representations, all of which are subsumed into a geometric algebra. In this work,we introduce a novel geometric algebra-based KG embedding framework, GeomE, which uti-lizes multivector representations and the geometric product to model entities and relations. Ourframework subsumes several state-of-the-art KG embedding approaches and is advantageouswith its ability of modeling various key relation patterns, including (anti-)symmetry, inversionand composition, rich expressiveness with higher degree of freedom as well as good general-ization capacity. Experimental results on multiple benchmark knowledge graphs show that theproposed approach outperforms existing state-of-the-art models for link prediction.
On the linearity of large non-linear models: when and why the tangent kernel is constant
Venue: arXiv cs.LG 2021
Year: 2021

The goal of this work is to shed light on the remarkable phenomenon of transition to linearity of certain neural networks as their width approaches infinity. We show that the transition to linearity of the model and, equivalently, constancy of the (neural) tangent kernel (NTK) result from the scaling properties of the norm of the Hessian matrix of the network as a function of the network width. We present a general framework for understanding the constancy of the tangent kernel via Hessian scaling applicable to the standard classes of neural networks. Our analysis provides a new perspective on the phenomenon of constant tangent kernel, which is different from the widely accepted "lazy training". Furthermore, we show that the transition to linearity is not a general property of wide neural networks and does not hold when the last layer of the network is non-linear. It is also not necessary for successful optimization by gradient descent.
Large System Analysis of Box-Relaxation in Correlated Massive MIMO Systems Under Imperfect CSI (Extended Version)
Venue: arXiv cs.IT 2021
Year: 2021

In this paper, we study the mean square error (MSE) and the bit error rate (BER) performance of the box-relaxation decoder in massive multiple-input-multiple-output (MIMO) systems under the assumptions of imperfect channel state information (CSI) and receive-side channel correlation. Our analysis assumes that the number of transmit and receive antennas ($n$,and $m$) grow simultaneously large while their ratio remains fixed. For simplicity of the analysis, we consider binary phase shift keying (BPSK) modulated signals. The asymptotic approximations of the MSE and BER enable us to derive the optimal power allocation scheme under MSE/BER minimization. Numerical simulations suggest that the asymptotic approximations are accurate even for small $n$ and $m$. They also show the important role of the box constraint in mitigating the so called double descent phenomenon.
Multi-microphone Complex Spectral Mapping for Utterance-wise and Continuous Speech Separation
Venue: arXiv cs.SD 2021
Year: 2021

We propose multi-microphone complex spectral mapping, a simple way of applying deep learning for time-varying non-linear beamforming, for speaker separation in reverberant conditions. We aim at both speaker separation and dereverberation. Our study first investigates offline utterance-wise speaker separation and then extends to block-online continuous speech separation (CSS). Assuming a fixed array geometry between training and testing, we train deep neural networks (DNN) to predict the real and imaginary (RI) components of target speech at a reference microphone from the RI components of multiple microphones. We then integrate multi-microphone complex spectral mapping with minimum variance distortionless response (MVDR) beamforming and post-filtering to further improve separation, and combine it with frame-level speaker counting for block-online CSS. Although our system is trained on simulated room impulse responses (RIR) based on a fixed number of microphones arranged in a given geometry, it generalizes well to a real array with the same geometry. State-of-the-art separation performance is obtained on the simulated two-talker SMS-WSJ corpus and the real-recorded LibriCSS dataset.
Learning Manifold Implicitly via Explicit Heat-Kernel Learning
Venue: arXiv cs.LG 2021
Year: 2021

Manifold learning is a fundamental problem in machine learning with numerous applications. Most of the existing methods directly learn the low-dimensional embedding of the data in some high-dimensional space, and usually lack the flexibility of being directly applicable to down-stream applications. In this paper, we propose the concept of implicit manifold learning, where manifold information is implicitly obtained by learning the associated heat kernel. A heat kernel is the solution of the corresponding heat equation, which describes how "heat" transfers on the manifold, thus containing ample geometric information of the manifold. We provide both practical algorithm and theoretical analysis of our framework. The learned heat kernel can be applied to various kernel-based machine learning models, including deep generative models (DGM) for data generation and Stein Variational Gradient Descent for Bayesian inference. Extensive experiments show that our framework can achieve state-of-the-art results compared to existing methods for the two tasks.
Substitution discrete plane tilings with $2n$-fold rotational symmetry for odd n
Venue: arXiv cs.DM 2021
Year: 2021

We study substitution tilings that are also discrete plane tilings, that is, satisfy a relaxed version of cut-and-projection. We prove that the Sub Rosa substitution tilings with a 2n-fold rotational symmetry for odd n greater than 5 defined by Kari and Rissanen are not discrete planes, and therefore not cut-and-project tilings either. We then define new Planar Rosa substitution tilings with a 2n-fold rotational symmetry for any odd n, and show that these satisfy the discrete plane condition. The tilings we consider are edge-to-edge rhombus tilings. We give an explicit construction for the 10-fold case, and provide a construction method for the general case of any odd n.
Deliberative Acting, Online Planning and Learning with Hierarchical Operational Models
Venue: arXiv cs.AI 2021
Year: 2021

In AI research, synthesizing a plan of action has typically used descriptive models of the actions that abstractly specify what might happen as a result of an action, and are tailored for efficiently computing state transitions. However, executing the planned actions has needed operational models, in which rich computational control structures and closed-loop online decision-making are used to specify how to perform an action in a nondeterministic execution context, react to events and adapt to an unfolding situation. Deliberative actors, which integrate acting and planning, have typically needed to use both of these models together -- which causes problems when attempting to develop the different models, verify their consistency, and smoothly interleave acting and planning. As an alternative, we define and implement an integrated acting and planning system in which both planning and acting use the same operational models. These rely on hierarchical task-oriented refinement methods offering rich control structures. The acting component, called Reactive Acting Engine (RAE), is inspired by the well-known PRS system. At each decision step, RAE can get advice from a planner for a near-optimal choice with respect to a utility function. The anytime planner uses a UCT-like Monte Carlo Tree Search procedure, called UPOM, whose rollouts are simulations of the actor's operational models. We also present learning strategies for use with RAE and UPOM that acquire, from online acting experiences and/or simulated planning results, a mapping from decision contexts to method instances as well as a heuristic function to guide UPOM. We demonstrate the asymptotic convergence of UPOM towards optimal methods in static domains, and show experimentally that UPOM and the learning strategies significantly improve the acting efficiency and robustness.
Deep Reinforcement Learning for Electric Vehicle Routing Problem with Time Windows
Venue: arXiv cs.LG 2021
Year: 2021

The past decade has seen a rapid penetration of electric vehicles (EV) in the market, more and more logistics and transportation companies start to deploy EVs for service provision. In order to model the operations of a commercial EV fleet, we utilize the EV routing problem with time windows (EVRPTW). In this research, we propose an end-to-end deep reinforcement learning framework to solve the EVRPTW. In particular, we develop an attention model incorporating the pointer network and a graph embedding technique to parameterize a stochastic policy for solving the EVRPTW. The model is then trained using policy gradient with rollout baseline. Our numerical studies show that the proposed model is able to efficiently solve EVRPTW instances of large sizes that are not solvable with any existing approaches.
Invertible DenseNets
Venue: arXiv cs.LG 2021
Year: 2021

We introduce Invertible Dense Networks (i-DenseNets), a more parameter efficient alternative to Residual Flows. The method relies on an analysis of the Lipschitz continuity of the concatenation in DenseNets, where we enforce the invertibility of the network by satisfying the Lipschitz constraint. Additionally, we extend this method by proposing a learnable concatenation, which not only improves the model performance but also indicates the importance of the concatenated representation. We demonstrate the performance of i-DenseNets and Residual Flows on toy, MNIST, and CIFAR10 data. Both i-DenseNets outperform Residual Flows evaluated in negative log-likelihood, on all considered datasets under an equal parameter budget.
Interdependent Diffusion: The social contagion of interacting beliefs
Venue: arXiv cs.SI 2021
Year: 2021

Social contagion is the process in which people adopt a belief, idea, or practice from a neighbor and pass it along to someone else. For over 100 years, scholars of social contagion have almost exclusively made the same implicit assumption: that only one belief, idea, or practice spreads through the population at a time. It is a default assumption that we don't bother to state, let alone justify. The assumption is so ingrained that our literature doesn't even have a word for "whatever is to be diffused", because we have never needed to discuss more than one of them. But this assumption is obviously false. Millions of beliefs, ideas, and practices (let's call them "diffusants") spread through social contagion every day. To assume that diffusants spread one at a time - or more generously, that they spread independently of one another - is to assume that interactions between diffusants have no influence on adoption patterns. This could be true, or it could be wildly off the mark. We've never stopped to find out. This paper makes a direct comparison between the spread of independent and interdependent beliefs using simulations, observational data, and a 2400-subject laboratory experiment. I find that in assuming independence between diffusants, scholars have overlooked social processes that fundamentally change the outcomes of social contagion. Interdependence between beliefs generates polarization, irrespective of social network structure, homophily, demographics, politics, or any other commonly cited cause. It also coordinates structures of beliefs that can have both internal justification and social support without any grounding in external truth.
Temporal Difference Uncertainties as a Signal for Exploration
Venue: arXiv cs.AI 2021
Year: 2021

An effective approach to exploration in reinforcement learning is to rely on an agent's uncertainty over the optimal policy, which can yield near-optimal exploration strategies in tabular settings. However, in non-tabular settings that involve function approximators, obtaining accurate uncertainty estimates is almost as challenging a problem. In this paper, we highlight that value estimates are easily biased and temporally inconsistent. In light of this, we propose a novel method for estimating uncertainty over the value function that relies on inducing a distribution over temporal difference errors. This exploration signal controls for state-action transitions so as to isolate uncertainty in value that is due to uncertainty over the agent's parameters. Because our measure of uncertainty conditions on state-action transitions, we cannot act on this measure directly. Instead, we incorporate it as an intrinsic reward and treat exploration as a separate learning problem, induced by the agent's temporal difference uncertainties. We introduce a distinct exploration policy that learns to collect data with high estimated uncertainty, which gives rise to a curriculum that smoothly changes throughout learning and vanishes in the limit of perfect value estimates. We evaluate our method on hard exploration tasks, including Deep Sea and Atari 2600 environments and find that our proposed form of exploration facilitates both diverse and deep exploration.
How to send a real number using a single bit (and some shared randomness)
Venue: arXiv cs.DS 2021
Year: 2021

We consider the fundamental problem of communicating an estimate of a real number $x\in[0,1]$ using a single bit. A sender that knows $x$ chooses a value $X\in\set{0,1}$ to transmit. In turn, a receiver estimates $x$ based on the value of $X$. We consider both the biased and unbiased estimation problems and aim to minimize the cost. For the biased case, the cost is the worst-case (over the choice of $x$) expected squared error, which coincides with the variance if the algorithm is required to be unbiased. We first overview common biased and unbiased estimation approaches and prove their optimality when no shared randomness is allowed. We then show how a small amount of shared randomness, which can be as low as a single bit, reduces the cost in both cases. Specifically, we derive lower bounds on the cost attainable by any algorithm with unrestricted use of shared randomness and propose near-optimal solutions that use a small number of shared random bits. Finally, we discuss open problems and future directions.
A Parallel Batch-Dynamic Data Structure for the Closest Pair Problem
Venue: arXiv cs.DS 2021
Year: 2021

We propose a theoretically-efficient and practical parallel batch-dynamic data structure for the closest pair problem. Our solution is based on a serial dynamic closest pair data structure by Golin et al., and supports batches of insertions and deletions in parallel. For a data set of size $n$, our data structure supports a batch of insertions or deletions of size $m$ in $O(m(1+\log ((n+m)/m)))$ expected work and $O(\log (n+m)\log^*(n+m))$ depth with high probability, and takes linear space. The key techniques for achieving these bounds are a new work-efficient parallel batch-dynamic binary heap, and careful management of the computation across sets of points to minimize work and depth. We provide an optimized multicore implementation of our data structure using dynamic hash tables, parallel heaps, and dynamic $k$-d trees. Our experiments on a variety of synthetic and real-world data sets show that it achieves a parallel speedup of up to 38.57x (15.10x on average) on 48 cores with hyper-threading. In addition, we also implement and compare four parallel algorithms for static closest pair problem, for which we are not aware of any existing practical implementations. On 48 cores with hyper-threading, the static algorithms achieve up to 51.45x (29.42x on average) speedup, and Rabin's algorithm performs the best on average. Comparing our dynamic algorithm to the fastest static algorithm, we find that it is advantageous to use the dynamic algorithm for batch sizes of up to 20\% of the data set. As far as we know, our work is the first to experimentally evaluate parallel closest pair algorithms, in both the static and the dynamic settings.
Determination for minimum symbol-pair and RT weights via torsional degrees of repeated-root cyclic codes
Venue: arXiv cs.IT 2021
Year: 2021

There are various metrics for researching error-correcting codes. Especially, high-density data storage system gives the existence of inconsistency for the reading and writing process. The symbol-pair metric is motivated for outputs that have overlapping pairs of symbols in a certain channel. The Rosenbloom-Tsfasman (RT) metric is introduced since there exists a problem that is related to transmission over several parallel communication channels with some channels not available for the transmission. In this paper, we determine the minimum symbol-pair weight and RT weight of repeated-root cyclic codes over $\mathfrak R=\Bbb F_{p^m}[u]/\langle u^4\rangle$ of length $n=p^k$. For the determination, we explicitly present third torsional degree for all different types of cyclic codes over $\mathfrak R$ of length $n$.
If beam search is the answer, what was the question?
Venue: arXiv cs.CL 2021
Year: 2021

Quite surprisingly, exact maximum a posteriori (MAP) decoding of neural language generators frequently leads to low-quality results. Rather, most state-of-the-art results on language generation tasks are attained using beam search despite its overwhelmingly high search error rate. This implies that the MAP objective alone does not express the properties we desire in text, which merits the question: if beam search is the answer, what was the question? We frame beam search as the exact solution to a different decoding objective in order to gain insights into why high probability under a model alone may not indicate adequacy. We find that beam search enforces uniform information density in text, a property motivated by cognitive science. We suggest a set of decoding objectives that explicitly enforce this property and find that exact decoding with these objectives alleviates the problems encountered when decoding poorly calibrated language generation models. Additionally, we analyze the text produced using various decoding strategies and see that, in our neural machine translation experiments, the extent to which this property is adhered to strongly correlates with BLEU.
Characterization of surface motion patterns in highly deformable soft tissue organs from dynamic MRI: An application to assess 4D bladder motion
Venue: arXiv cs.CV 2021
Year: 2021

Dynamic MRI may capture temporal anatomical changes in soft tissue organs with high contrast but the obtained sequences usually suffer from limited volume coverage which makes the high resolution reconstruction of organ shape trajectories a major challenge in temporal studies. Because of the variability of abdominal organ shapes across time and subjects, the objective of this study is to go towards 3D dense velocity measurements to fully cover the entire surface and to extract meaningful features characterizing the observed organ deformations and enabling clinical action or decision. We present a pipeline for characterization of bladder surface dynamics during deep respiratory movements. For a compact shape representation, the reconstructed temporal volumes were first used to establish subject-specific dynamical 4D mesh sequences using the LDDMM framework. Then, we performed a statistical characterization of organ dynamics from mechanical parameters such as mesh elongations and distortions. Since we refer to organs as non flat surfaces, we have also used the mean curvature changes as metric to quantify surface evolution. However, the numerical computation of curvature is strongly dependant on the surface parameterization. To cope with this dependency, we employed a new method for surface deformation analysis. Independent of parameterization and minimizing the length of the geodesic curves, it stretches smoothly the surface curves towards a sphere by minimizing a Dirichlet energy. An Eulerian PDE approach is used to derive a shape descriptor from the curve-shortening flow. Intercorrelations between individual motion patterns are computed using the Laplace Beltrami operator eigenfunctions for spherical mapping. Application to extracting characterization correlation curves for locally controlled simulated shape trajectories demonstrates the stability of the proposed shape descriptor.
Learning a mixture of two subspaces over finite fields
Venue: arXiv cs.DS 2021
Year: 2021

We study the problem of learning a mixture of two subspaces over $\mathbb{F}_2^n$. The goal is to recover the individual subspaces, given samples from a (weighted) mixture of samples drawn uniformly from the two subspaces $A_0$ and $A_1$. This problem is computationally challenging, as it captures the notorious problem of "learning parities with noise" in the degenerate setting when $A_1 \subseteq A_0$. This is in contrast to the analogous problem over the reals that can be solved in polynomial time (Vidal'03). This leads to the following natural question: is Learning Parities with Noise the only computational barrier in obtaining efficient algorithms for learning mixtures of subspaces over $\mathbb{F}_2^n$? The main result of this paper is an affirmative answer to the above question. Namely, we show the following results: 1. When the subspaces $A_0$ and $A_1$ are incomparable, i.e., $A_0$ and $A_1$ are not contained inside each other, then there is a polynomial time algorithm to recover the subspaces $A_0$ and $A_1$. 2. In the case when $A_1$ is a subspace of $A_0$ with a significant gap in the dimension i.e., $dim(A_1) \le αdim(A_0)$ for $α<1$, there is a $n^{O(1/(1-α))}$ time algorithm to recover the subspaces $A_0$ and $A_1$. Thus, our algorithms imply computational tractability of the problem of learning mixtures of two subspaces, except in the degenerate setting captured by learning parities with noise.
Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
Venue: arXiv cs.LG 2021
Year: 2021

Uncertainty quantification for complex deep learning models is increasingly important as these techniques see growing use in high-stakes, real-world settings. Currently, the quality of a model's uncertainty is evaluated using point-prediction metrics such as negative log-likelihood or the Brier score on heldout data. In this study, we provide the first large scale evaluation of the empirical frequentist coverage properties of well known uncertainty quantification techniques on a suite of regression and classification tasks. We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that coverage is not maintained on out-of-distribution data. Our results demonstrate the failings of current uncertainty quantification techniques as dataset shift increases and establish coverage as an important metric in developing models for real-world applications.