OpenRaiser/Intern-Atlas huggingface dataset:
Personalization for Web-based Services using Offline Reinforcement Learning
Venue: arXiv cs.LG 2021
Year: 2021

Large-scale Web-based services present opportunities for improving UI policies based on observed user interactions. We address challenges of learning such policies through model-free offline Reinforcement Learning (RL) with off-policy training. Deployed in a production system for user authentication in a major social network, it significantly improves long-term objectives. We articulate practical challenges, compare several ML techniques, provide insights on training and evaluation of RL models, and discuss generalizations.
Voice Cloning: a Multi-Speaker Text-to-Speech Synthesis Approach based on Transfer Learning
Venue: arXiv cs.SD 2021
Year: 2021

Deep learning models are becoming predominant in many fields of machine learning. Text-to-Speech (TTS), the process of synthesizing artificial speech from text, is no exception. To this end, a deep neural network is usually trained using a corpus of several hours of recorded speech from a single speaker. Trying to produce the voice of a speaker other than the one learned is expensive and requires large effort since it is necessary to record a new dataset and retrain the model. This is the main reason why the TTS models are usually single speaker. The proposed approach has the goal to overcome these limitations trying to obtain a system which is able to model a multi-speaker acoustic space. This allows the generation of speech audio similar to the voice of different target speakers, even if they were not observed during the training phase.
Hitting Sets and Reconstruction for Dense Orbits in $\text{VP}_e$ and $ΣΠΣ$ Circuits
Venue: arXiv cs.CC 2021
Year: 2021

In this paper we study polynomials in $\text{VP}_e$ (polynomial-sized formulas) and in $ΣΠΣ$ (polynomial-size depth-$3$ circuits) whose orbits, under the action of the affine group $\text{GL}_n^{\text{aff}}(\mathbb{F})$, are $\mathit{dense}$ in their ambient class. We construct hitting sets and interpolating sets for these orbits as well as give reconstruction algorithms. As $\text{VP}=\text{VNC}^2$, our results for $\text{VP}_e$ translate immediately to $\text{VP}$ with a quasipolynomial blow up in parameters. If any of our hitting or interpolating sets could be made $\mathit{robust}$ then this would immediately yield a hitting set for the superclass in which the relevant class is dense, and as a consequence also a lower bound for the superclass. Unfortunately, we also prove that the kind of constructions that we have found (which are defined in terms of $k$-independent polynomial maps) do not necessarily yield robust hitting sets.
Generating Synthetic Text Data to Evaluate Causal Inference Methods
Venue: arXiv cs.CL 2021
Year: 2021

Drawing causal conclusions from observational data requires making assumptions about the true data-generating process. Causal inference research typically considers low-dimensional data, such as categorical or numerical fields in structured medical records. High-dimensional and unstructured data such as natural language complicates the evaluation of causal inference methods; such evaluations rely on synthetic datasets with known causal effects. Models for natural language generation have been widely studied and perform well empirically. However, existing methods not immediately applicable to producing synthetic datasets for causal evaluations, as they do not allow for quantifying a causal effect on the text itself. In this work, we develop a framework for adapting existing generation models to produce synthetic text datasets with known causal effects. We use this framework to perform an empirical comparison of four recently-proposed methods for estimating causal effects from text data. We release our code and synthetic datasets.
Automated Video Labelling: Identifying Faces by Corroborative Evidence
Venue: arXiv cs.CV 2021
Year: 2021

We present a method for automatically labelling all faces in video archives, such as TV broadcasts, by combining multiple evidence sources and multiple modalities (visual and audio). We target the problem of ever-growing online video archives, where an effective, scalable indexing solution cannot require a user to provide manual annotation or supervision. To this end, we make three key contributions: (1) We provide a novel, simple, method for determining if a person is famous or not using image-search engines. In turn this enables a face-identity model to be built reliably and robustly, and used for high precision automatic labelling; (2) We show that even for less-famous people, image-search engines can then be used for corroborative evidence to accurately label faces that are named in the scene or the speech; (3) Finally, we quantitatively demonstrate the benefits of our approach on different video domains and test settings, such as TV shows and news broadcasts. Our method works across three disparate datasets without any explicit domain adaptation, and sets new state-of-the-art results on all the public benchmarks.
Resilient Architectures for Free Space Optical Wireless Interconnection Systems
Venue: arXiv cs.NI 2021
Year: 2021

In this paper, we propose the use of two Passive Optical Network (PON)-based network architectures to connect free-space Optical Wireless Communication (OWC) Access Points (APs) within a room with multiple users. We optimize through a Mixed Linear Integer Programming (MILP) model the assignment of mobile OWC users to more than one AP to improve the resilience of the fronthaul network, i.e the OWC system and the wired network linked to APs, and study the impact of users distribution and channel characteristics.
Differentiable Generative Phonology
Venue: arXiv cs.CL 2021
Year: 2021

The goal of generative phonology, as formulated by Chomsky and Halle (1968), is to specify a formal system that explains the set of attested phonological strings in a language. Traditionally, a collection of rules (or constraints, in the case of optimality theory) and underlying forms (UF) are posited to work in tandem to generate phonological strings. However, the degree of abstraction of UFs with respect to their concrete realizations is contentious. As the main contribution of our work, we implement the phonological generative system as a neural model differentiable end-to-end, rather than as a set of rules or constraints. Contrary to traditional phonology, in our model, UFs are continuous vectors in $\mathbb{R}^d$, rather than discrete strings. As a consequence, UFs are discovered automatically rather than posited by linguists, and the model can scale to the size of a realistic vocabulary. Moreover, we compare several modes of the generative process, contemplating: i) the presence or absence of an underlying representation in between morphemes and surface forms (SFs); and ii) the conditional dependence or independence of UFs with respect to SFs. We evaluate the ability of each mode to predict attested phonological strings on 2 datasets covering 5 and 28 languages, respectively. The results corroborate two tenets of generative phonology, viz. the necessity for UFs and their independence from SFs. In general, our neural model of generative phonology learns both UFs and SFs automatically and on a large-scale.
An Affective Aware Pseudo Association Method to Connect Disjoint Users Across Multiple Datasets -- An Enhanced Validation Method for Text-based Emotion Aware Recommender
Venue: arXiv cs.IR 2021
Year: 2021

We derive a method to enhance the evaluation for a text-based Emotion Aware Recommender that we have developed. However, we did not implement a suitable way to assess the top-N recommendations subjectively. In this study, we introduce an emotion-aware Pseudo Association Method to interconnect disjointed users across different datasets so data files can be combined to form a more extensive data file. Users with the same user IDs found in separate data files in the same dataset are often the same users. However, users with the same user ID may not be the same user across different datasets. We advocate an emotion aware Pseudo Association Method to associate users across different datasets. The approach interconnects users with different user IDs across different datasets through the most similar users' emotion vectors (UVECs). We found the method improved the evaluation process of assessing the top-N recommendations objectively.
Interpreting a concurrent $λ$-calculus in differential proof nets (extended version)
Venue: arXiv cs.LO 2021
Year: 2021

In this paper, we show how to interpret a language featuring concurrency, references and replication into proof nets, which correspond to a fragment of differential linear logic. We prove a simulation and adequacy theorem. A key element in our translation are routing areas, a family of nets used to implement communication primitives which we define and study in detail.
A better lower bound for Lower-Left Anchored Rectangle Packing
Venue: arXiv cs.CG 2021
Year: 2021

Given any set of points $S$ in the unit square that contains the origin, does a set of axis aligned rectangles, one for each point in $S$, exist, such that each of them has a point in $S$ as its lower-left corner, they are pairwise interior disjoint, and the total area that they cover is at least 1/2? This question is also known as Freedman's conjecture (conjecturing that such a set of rectangles does exist) and has been open since Allen Freedman posed it in 1969. In this paper, we improve the best known lower bound on the total area that can be covered from 0.09121 to 0.1039. Although this step is small, we introduce new insights that push the limits of this analysis. Our lower bound uses a greedy algorithm with a particular order of the points in $S$. Therefore, it also implies that this greedy algorithm achieves an approximation ratio of 0.1039. We complement the result with an upper bound of 3/4 on the approximation ratio for a natural class of greedy algorithms that includes the one that achieves the lower bound.
Customizing Contextualized Language Models forLegal Document Reviews
Venue: arXiv cs.CL 2021
Year: 2021

Inspired by the inductive transfer learning on computer vision, many efforts have been made to train contextualized language models that boost the performance of natural language processing tasks. These models are mostly trained on large general-domain corpora such as news, books, or Wikipedia.Although these pre-trained generic language models well perceive the semantic and syntactic essence of a language structure, exploiting them in a real-world domain-specific scenario still needs some practical considerations to be taken into account such as token distribution shifts, inference time, memory, and their simultaneous proficiency in multiple tasks. In this paper, we focus on the legal domain and present how different language model strained on general-domain corpora can be best customized for multiple legal document reviewing tasks. We compare their efficiencies with respect to task performances and present practical considerations.
Fisher Information and Mutual Information Constraints
Venue: arXiv cs.IT 2021
Year: 2021

We consider the processing of statistical samples $X\sim P_θ$ by a channel $p(y|x)$, and characterize how the statistical information from the samples for estimating the parameter $θ\in\mathbb{R}^d$ can scale with the mutual information or capacity of the channel. We show that if the statistical model has a sub-Gaussian score function, then the trace of the Fisher information matrix for estimating $θ$ from $Y$ can scale at most linearly with the mutual information between $X$ and $Y$. We apply this result to obtain minimax lower bounds in distributed statistical estimation problems, and obtain a tight preconstant for Gaussian mean estimation. We then show how our Fisher information bound can also imply mutual information or Jensen-Shannon divergence based distributed strong data processing inequalities.
Translation Invariant Fréchet Distance Queries
Venue: arXiv cs.CG 2021
Year: 2021

The Fréchet distance is a popular similarity measure between curves. For some applications, it is desirable to match the curves under translation before computing the Fréchet distance between them. This variant is called the Translation Invariant Fréchet distance, and algorithms to compute it are well studied. The query version, finding an optimal placement in the plane for a query segment where the Fréchet distance becomes minimized, is much less well understood. We study Translation Invariant Fréchet distance queries in a restricted setting of horizontal query segments. More specifically, we preprocess a trajectory in $\mathcal O(n^2 \log^2 n) $ time and $\mathcal O(n^{3/2})$ space, such that for any subtrajectory and any horizontal query segment we can compute their Translation Invariant Fréchet distance in $\mathcal O(\text{polylog } n)$ time. We hope this will be a step towards answering Translation Invariant Fréchet queries between arbitrary trajectories.
Deep Reinforcement Learning for Combinatorial Optimization: Covering Salesman Problems
Venue: arXiv cs.NE 2021
Year: 2021

This paper introduces a new deep learning approach to approximately solve the Covering Salesman Problem (CSP). In this approach, given the city locations of a CSP as input, a deep neural network model is designed to directly output the solution. It is trained using the deep reinforcement learning without supervision. Specifically, in the model, we apply the Multi-head Attention to capture the structural patterns, and design a dynamic embedding to handle the dynamic patterns of the problem. Once the model is trained, it can generalize to various types of CSP tasks (different sizes and topologies) with no need of re-training. Through controlled experiments, the proposed approach shows desirable time complexity: it runs more than 20 times faster than the traditional heuristic solvers with a tiny gap of optimality. Moreover, it significantly outperforms the current state-of-the-art deep learning approaches for combinatorial optimization in the aspect of both training and inference. In comparison with traditional solvers, this approach is highly desirable for most of the challenging tasks in practice that are usually large-scale and require quick decisions.
Privacy-Preserving Self-Taught Federated Learning for Heterogeneous Data
Venue: arXiv cs.LG 2021
Year: 2021

Many application scenarios call for training a machine learning model among multiple participants. Federated learning (FL) was proposed to enable joint training of a deep learning model using the local data in each party without revealing the data to others. Among various types of FL methods, vertical FL is a category to handle data sources with the same ID space and different feature spaces. However, existing vertical FL methods suffer from limitations such as restrictive neural network structure, slow training speed, and often lack the ability to take advantage of data with unmatched IDs. In this work, we propose an FL method called self-taught federated learning to address the aforementioned issues, which uses unsupervised feature extraction techniques for distributed supervised deep learning tasks. In this method, only latent variables are transmitted to other parties for model training, while privacy is preserved by storing the data and parameters of activations, weights, and biases locally. Extensive experiments are performed to evaluate and demonstrate the validity and efficiency of the proposed method.
Brain Modelling as a Service: The Virtual Brain on EBRAINS
Venue: arXiv cs.CE 2021
Year: 2021

The Virtual Brain (TVB) is now available as open-source cloud ecosystem on EBRAINS, a shared digital research platform for brain science. It offers services for constructing, simulating and analysing brain network models (BNMs) including the TVB network simulator; magnetic resonance imaging (MRI) processing pipelines to extract structural and functional connectomes; multiscale co-simulation of spiking and large-scale networks; a domain specific language for automatic high-performance code generation from user-specified models; simulation-ready BNMs of patients and healthy volunteers; Bayesian inference of epilepsy spread; data and code for mouse brain simulation; and extensive educational material. TVB cloud services facilitate reproducible online collaboration and discovery of data assets, models, and software embedded in scalable and secure workflows, a precondition for research on large cohort data sets, better generalizability and clinical translation.
A formal proof of modal completeness for provability logic
Venue: arXiv cs.LO 2021
Year: 2021

This work presents a formalized proof of modal completeness for Gödel-Löb provability logic (GL) in the HOL Light theorem prover. We describe the code we developed, and discuss some details of our implementation, focusing on our choices in structuring proofs which make essential use of the tools of HOL Light and which differ in part from the standard strategies found in main textbooks covering the topic in an informal setting. Moreover, we propose a reflection on our own experience in using this specific theorem prover for this formalization task, with an analysis of pros and cons of reasoning within and about the formal system for GL we implemented in our code.
VIODE: A Simulated Dataset to Address the Challenges of Visual-Inertial Odometry in Dynamic Environments
Venue: arXiv cs.RO 2021
Year: 2021

Dynamic environments such as urban areas are still challenging for popular visual-inertial odometry (VIO) algorithms. Existing datasets typically fail to capture the dynamic nature of these environments, therefore making it difficult to quantitatively evaluate the robustness of existing VIO methods. To address this issue, we propose three contributions: firstly, we provide the VIODE benchmark, a novel dataset recorded from a simulated UAV that navigates in challenging dynamic environments. The unique feature of the VIODE dataset is the systematic introduction of moving objects into the scenes. It includes three environments, each of which is available in four dynamic levels that progressively add moving objects. The dataset contains synchronized stereo images and IMU data, as well as ground-truth trajectories and instance segmentation masks. Secondly, we compare state-of-the-art VIO algorithms on the VIODE dataset and show that they display substantial performance degradation in highly dynamic scenes. Thirdly, we propose a simple extension for visual localization algorithms that relies on semantic information. Our results show that scene semantics are an effective way to mitigate the adverse effects of dynamic objects on VIO algorithms. Finally, we make the VIODE dataset publicly available at https://github.com/kminoda/VIODE.
L-SNet: from Region Localization to Scale Invariant Medical Image Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

Coarse-to-fine models and cascade segmentation architectures are widely adopted to solve the problem of large scale variations in medical image segmentation. However, those methods have two primary limitations: the first-stage segmentation becomes a performance bottleneck; the lack of overall differentiability makes the training process of two stages asynchronous and inconsistent. In this paper, we propose a differentiable two-stage network architecture to tackle these problems. In the first stage, a localization network (L-Net) locates Regions of Interest (RoIs) in a detection fashion; in the second stage, a segmentation network (S-Net) performs fine segmentation on the recalibrated RoIs; a RoI recalibration module between L-Net and S-Net eliminating the inconsistencies. Experimental results on the public dataset show that our method outperforms state-of-the-art coarse-to-fine models with negligible computation overheads.
Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank
Venue: arXiv cs.LG 2021
Year: 2021

Existing work in counterfactual Learning to Rank (LTR) has focussed on optimizing feature-based models that predict the optimal ranking based on document features. LTR methods based on bandit algorithms often optimize tabular models that memorize the optimal ranking per query. These types of model have their own advantages and disadvantages. Feature-based models provide very robust performance across many queries, including those previously unseen, however, the available features often limit the rankings the model can predict. In contrast, tabular models can converge on any possible ranking through memorization. However, memorization is extremely prone to noise, which makes tabular models reliable only when large numbers of user interactions are available. Can we develop a robust counterfactual LTR method that pursues memorization-based optimization whenever it is safe to do? We introduce the Generalization and Specialization (GENSPEC) algorithm, a robust feature-based counterfactual LTR method that pursues per-query memorization when it is safe to do so. GENSPEC optimizes a single feature-based model for generalization: robust performance across all queries, and many tabular models for specialization: each optimized for high performance on a single query. GENSPEC uses novel relative high-confidence bounds to choose which model to deploy per query. By doing so, GENSPEC enjoys the high performance of successfully specialized tabular models with the robustness of a generalized feature-based model. Our results show that GENSPEC leads to optimal performance on queries with sufficient click data, while having robust behavior on queries with little or noisy data.
Full waveform inversion using extended and simultaneous sources
Venue: arXiv cs.CE 2021
Year: 2021

PDE-constrained optimization problems are often treated using the reduced formulation where the PDE constraints are eliminated. This approach is known to be more computationally feasible than other alternatives at large scales. However, the elimination of the constraints forces the optimization process to fulfill the constraints at all times. In some problems this may lead to a highly non-linear objective, which is hard to solve. An example to such a problem, which we focus on in this work, is Full Waveform Inversion (FWI), which appears in seismic exploration of oil and gas reservoirs, and medical imaging. In an attempt to relieve the non-linearity of FWI, several approaches suggested to expand the optimization search space and relax the PDE constraints. This comes, however, with severe memory and computational costs, which we aim to reduce. In this work we adopt the expanded search space approach, and suggest a new formulation of FWI using extended source functions. To make the source-extended problem more feasible in memory and computations, we couple the source extensions in the form of a low-rank matrix. This way, we have a large-but-manageable additional parameter space, which has a rather low memory footprint, and is much more suitable for solving large scale instances of the problem than the full rank additional space. In addition, we show how our source-extended approach is applied together with the popular simultaneous sources technique -- a stochastic optimization technique that significantly reduces the computations needed for FWI inversions. We demonstrate our approaches for solving FWI problems using 2D and 3D models with high frequency data only.
A Continuized View on Nesterov Acceleration
Venue: arXiv cs.DC 2021
Year: 2021

We introduce the "continuized" Nesterov acceleration, a close variant of Nesterov acceleration whose variables are indexed by a continuous time parameter. The two variables continuously mix following a linear ordinary differential equation and take gradient steps at random times. This continuized variant benefits from the best of the continuous and the discrete frameworks: as a continuous process, one can use differential calculus to analyze convergence and obtain analytical expressions for the parameters; but a discretization of the continuized process can be computed exactly with convergence rates similar to those of Nesterov original acceleration. We show that the discretization has the same structure as Nesterov acceleration, but with random parameters.