OpenRaiser/Intern-Atlas huggingface dataset:
Visual Speech Enhancement Without A Real Visual Stream
Venue: arXiv cs.CV 2020
Year: 2020

In this work, we re-think the task of speech enhancement in unconstrained real-world environments. Current state-of-the-art methods use only the audio stream and are limited in their performance in a wide range of real-world noises. Recent works using lip movements as additional cues improve the quality of generated speech over "audio-only" methods. But, these methods cannot be used for several applications where the visual stream is unreliable or completely absent. We propose a new paradigm for speech enhancement by exploiting recent breakthroughs in speech-driven lip synthesis. Using one such model as a teacher network, we train a robust student network to produce accurate lip movements that mask away the noise, thus acting as a "visual noise filter". The intelligibility of the speech enhanced by our pseudo-lip approach is comparable (< 3% difference) to the case of using real lips. This implies that we can exploit the advantages of using lip movements even in the absence of a real video stream. We rigorously evaluate our model using quantitative metrics as well as human evaluations. Additional ablation studies and a demo video on our website containing qualitative comparisons and results clearly illustrate the effectiveness of our approach. We provide a demo video which clearly illustrates the effectiveness of our proposed approach on our website: \url{http://cvit.iiit.ac.in/research/projects/cvit-projects/visual-speech-enhancement-without-a-real-visual-stream}. The code and models are also released for future research: \url{https://github.com/Sindhu-Hegde/pseudo-visual-speech-denoising}.
Structure-Constrained Process Graphs for the Process Semantics of Regular Expressions
Venue: arXiv cs.LO 2020
Year: 2020

Milner (1984) introduced a process semantics for regular expressions as process graphs. Unlike for the language semantics, where every regular (that is, DFA-accepted) language is the interpretation of some regular expression, there are finite process graphs that are not bisimilar to the process interpretation of any regular expression. For reasoning about graphs that are expressible by regular expressions modulo bisimilarity it is desirable to have structural representations of process graphs in the image of the interpretation. For `1-free' regular expressions, their process interpretations satisfy the structural property LEE (loop existence and elimination). But this is not in general the case for all regular expressions, as we show by examples. Yet as a remedy, we describe the possibility to recover the property LEE for a close variant of the process interpretation. For this purpose we refine the process semantics of regular expressions to yield process graphs with 1-transitions, similar to silent moves for finite-state automata. This report accompanies the paper with the same title in the post-proceedings of the workshop TERMGRAPH 2020. Here we give the proofs of not only one but of both of the two central theorems.
Learning to Localize Using a LiDAR Intensity Map
Venue: arXiv cs.CV 2020
Year: 2020

In this paper we propose a real-time, calibration-agnostic and effective localization system for self-driving cars. Our method learns to embed the online LiDAR sweeps and intensity map into a joint deep embedding space. Localization is then conducted through an efficient convolutional matching between the embeddings. Our full system can operate in real-time at 15Hz while achieving centimeter level accuracy across different LiDAR sensors and environments. Our experiments illustrate the performance of the proposed approach over a large-scale dataset consisting of over 4000km of driving.
Learning Halfspaces With Membership Queries
Venue: arXiv cs.LG 2020
Year: 2020

Active learning is a subfield of machine learning, in which the learning algorithm is allowed to choose the data from which it learns. In some cases, it has been shown that active learning can yield an exponential gain in the number of samples the algorithm needs to see, in order to reach generalization error $\leq ε$. In this work we study the problem of learning halfspaces with membership queries. In the membership query scenario, we allow the learning algorithm to ask for the label of every sample in the input space. We suggest a new algorithm for this problem, and prove it achieves a near optimal label complexity in some cases. We also show that the algorithm works well in practice, and significantly outperforms uncertainty sampling.
Bayesian Optimization of Area-based Road Pricing
Venue: arXiv cs.MA 2020
Year: 2020

This study presents a Bayesian Optimization framework for area- and distance-based time-of-day pricing (TODP) for urban networks. The road pricing optimization problem can reach high level of complexity depending on the pricing scheme considered, its associated detailed network properties and the affected heterogeneous demand features. We consider heterogeneous travellers with individual-specific trip attributes and departure-time choice parameters together with a Macroscopic Fundamental Diagram (MFD) model for the urban network. Its mathematical formulation is presented and an agent-based simulation framework is constructed as evaluation function for the TODP optimization problem. The latter becomes highly nonlinear and relying on an expensive-to-evaluate objective function. We then present and test a Bayesian Optimization approach to compute different time-of-day pricing schemes by maximizing social welfare. Our proposed method learns the relationship between the prices and welfare within a few iterations and is able to find good solutions even in scenarios with high dimensionality in the decision variables space, setting a path for complexity reduction in more realistic road pricing optimization problems. Furthermore and as expected, the simulation results show that TODP improves the social welfare against the no-pricing case.
Fairness, Welfare, and Equity in Personalized Pricing
Venue: arXiv cs.LG 2020
Year: 2020

We study the interplay of fairness, welfare, and equity considerations in personalized pricing based on customer features. Sellers are increasingly able to conduct price personalization based on predictive modeling of demand conditional on covariates: setting customized interest rates, targeted discounts of consumer goods, and personalized subsidies of scarce resources with positive externalities like vaccines and bed nets. These different application areas may lead to different concerns around fairness, welfare, and equity on different objectives: price burdens on consumers, price envy, firm revenue, access to a good, equal access, and distributional consequences when the good in question further impacts downstream outcomes of interest. We conduct a comprehensive literature review in order to disentangle these different normative considerations and propose a taxonomy of different objectives with mathematical definitions. We focus on observational metrics that do not assume access to an underlying valuation distribution which is either unobserved due to binary feedback or ill-defined due to overriding behavioral concerns regarding interpreting revealed preferences. In the setting of personalized pricing for the provision of goods with positive benefits, we discuss how price optimization may provide unambiguous benefit by achieving a "triple bottom line": personalized pricing enables expanding access, which in turn may lead to gains in welfare due to heterogeneous utility, and improve revenue or budget utilization. We empirically demonstrate the potential benefits of personalized pricing in two settings: pricing subsidies for an elective vaccine, and the effects of personalized interest rates on downstream outcomes in microcredit.
A Note on Graph-Based Nearest Neighbor Search
Venue: arXiv cs.LG 2020
Year: 2020

Nearest neighbor search has found numerous applications in machine learning, data mining and massive data processing systems. The past few years have witnessed the popularity of the graph-based nearest neighbor search paradigm because of its superiority over the space-partitioning algorithms. While a lot of empirical studies demonstrate the efficiency of graph-based algorithms, not much attention has been paid to a more fundamental question: why graph-based algorithms work so well in practice? And which data property affects the efficiency and how? In this paper, we try to answer these questions. Our insight is that "the probability that the neighbors of a point o tends to be neighbors in the KNN graph" is a crucial data property for query efficiency. For a given dataset, such a property can be qualitatively measured by clustering coefficient of the KNN graph. To show how clustering coefficient affects the performance, we identify that, instead of the global connectivity, the local connectivity around some given query q has more direct impact on recall. Specifically, we observed that high clustering coefficient makes most of the k nearest neighbors of q sit in a maximum strongly connected component (SCC) in the graph. From the algorithmic point of view, we show that the search procedure is actually composed of two phases - the one outside the maximum SCC and the other one in it, which is different from the widely accepted single or multiple paths search models. We proved that the commonly used graph-based search algorithm is guaranteed to traverse the maximum SCC once visiting any point in it. Our analysis reveals that high clustering coefficient leads to large size of the maximum SCC, and thus provides good answer quality with the help of the two-phase search procedure. Extensive empirical results over a comprehensive collection of datasets validate our findings.
ResizeMix: Mixing Data with Preserved Object Information and True Labels
Venue: arXiv cs.CV 2020
Year: 2020

Data augmentation is a powerful technique to increase the diversity of data, which can effectively improve the generalization ability of neural networks in image recognition tasks. Recent data mixing based augmentation strategies have achieved great success. Especially, CutMix uses a simple but effective method to improve the classifiers by randomly cropping a patch from one image and pasting it on another image. To further promote the performance of CutMix, a series of works explore to use the saliency information of the image to guide the mixing. We systematically study the importance of the saliency information for mixing data, and find that the saliency information is not so necessary for promoting the augmentation performance. Furthermore, we find that the cutting based data mixing methods carry two problems of label misallocation and object information missing, which cannot be resolved simultaneously. We propose a more effective but very easily implemented method, namely ResizeMix. We mix the data by directly resizing the source image to a small patch and paste it on another image. The obtained patch preserves more substantial object information compared with conventional cut-based methods. ResizeMix shows evident advantages over CutMix and the saliency-guided methods on both image classification and object detection tasks without additional computation cost, which even outperforms most costly search-based automatic augmentation methods.
Hop-Hop Relation-aware Graph Neural Networks
Venue: arXiv cs.LG 2020
Year: 2020

Graph Neural Networks (GNNs) are widely used in graph representation learning. However, most GNN methods are designed for either homogeneous or heterogeneous graphs. In this paper, we propose a new model, Hop-Hop Relation-aware Graph Neural Network (HHR-GNN), to unify representation learning for these two types of graphs. HHR-GNN learns a personalized receptive field for each node by leveraging knowledge graph embedding to learn relation scores between the central node's representations at different hops. In neighborhood aggregation, our model simultaneously allows for hop-aware projection and aggregation. This mechanism enables the central node to learn a hop-wise neighborhood mixing that can be applied to both homogeneous and heterogeneous graphs. Experimental results on five benchmarks show the competitive performance of our model compared to state-of-the-art GNNs, e.g., up to 13K faster in terms of time cost per training epoch on large heterogeneous graphs.
A complete, parallel and autonomous photonic neural network in a semiconductor multimode laser
Venue: arXiv cs.NE 2020
Year: 2020

Neural networks are one of the disruptive computing concepts of our time. However, they fundamentally differ from classical, algorithmic computing in a number of fundamental aspects. These differences result in equally fundamental, severe and relevant challenges for neural network computing using current computing substrates. Neural networks urge for parallelism across the entire processor and for a co-location of memory and arithmetic, i.e. beyond von Neumann architectures. Parallelism in particular made photonics a highly promising platform, yet until now scalable and integratable concepts are scarce. Here, we demonstrate for the first time how a fully parallel and fully implemented photonic neural network can be realized using spatially distributed modes of an efficient and fast semiconductor laser. Importantly, all neural network connections are realized in hardware, and our processor produces results without pre- or post-processing. 130+ nodes are implemented in a large-area vertical cavity surface emitting laser, input and output weights are realized via the complex transmission matrix of a multimode fiber and a digital micro-mirror array, respectively. We train the readout weights to perform 2-bit header recognition, a 2-bit XOR and 2-bit digital analog conversion, and obtain < 0.9 10^-3 and 2.9 10^-2 error rates for digit recognition and XOR, respectively. Finally, the digital analog conversion can be realized with a standard deviation of only 5.4 10^-2. Our system is scalable to much larger sizes and to bandwidths in excess of 20 GHz.
Repairing Reed-Solomon Codes Evaluated on Subspaces
Venue: arXiv cs.IT 2020
Year: 2020

We consider the repair problem for Reed--Solomon (RS) codes, evaluated on an $\mathbb{F}_q$-linear subspace $U\subseteq\mathbb{F}_{q^m}$ of dimension $d$, where $q$ is a prime power, $m$ is a positive integer, and $\mathbb{F}_q$ is the Galois field of size $q$. For the case of $q\geq 3$, we show the existence of a linear repair scheme for the RS code of length $n=q^d$ and codimension $q^s$, $s< d$, evaluated on $U$, in which each of the $n-1$ surviving nodes transmits only $r$ symbols of $\mathbb{F}_q$, provided that $ms\geq d(m-r)$. For the case of $q=2$, we prove a similar result, with some restrictions on the evaluation linear subspace $U$. Our proof is based on a probabilistic argument, however the result is not merely an existence result; the success probability is fairly large (at least $1/3$) and there is a simple criterion for checking the validity of the randomly chosen linear repair scheme. Our result extend the construction of Dau--Milenkovich to the range $r<m-s$, for a wide range of parameters.
Learn molecular representations from large-scale unlabeled molecules for drug discovery
Venue: arXiv cs.LG 2020
Year: 2020

How to produce expressive molecular representations is a fundamental challenge in AI-driven drug discovery. Graph neural network (GNN) has emerged as a powerful technique for modeling molecular data. However, previous supervised approaches usually suffer from the scarcity of labeled data and have poor generalization capability. Here, we proposed a novel Molecular Pre-training Graph-based deep learning framework, named MPG, that leans molecular representations from large-scale unlabeled molecules. In MPG, we proposed a powerful MolGNet model and an effective self-supervised strategy for pre-training the model at both the node and graph-level. After pre-training on 11 million unlabeled molecules, we revealed that MolGNet can capture valuable chemistry insights to produce interpretable representation. The pre-trained MolGNet can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of drug discovery tasks, including molecular properties prediction, drug-drug interaction, and drug-target interaction, involving 13 benchmark datasets. Our work demonstrates that MPG is promising to become a novel approach in the drug discovery pipeline.
Diverse Knowledge Distillation for End-to-End Person Search
Venue: arXiv cs.CV 2020
Year: 2020

Person search aims to localize and identify a specific person from a gallery of images. Recent methods can be categorized into two groups, i.e., two-step and end-to-end approaches. The former views person search as two independent tasks and achieves dominant results using separately trained person detection and re-identification (Re-ID) models. The latter performs person search in an end-to-end fashion. Although the end-to-end approaches yield higher inference efficiency, they largely lag behind those two-step counterparts in terms of accuracy. In this paper, we argue that the gap between the two kinds of methods is mainly caused by the Re-ID sub-networks of end-to-end methods. To this end, we propose a simple yet strong end-to-end network with diverse knowledge distillation to break the bottleneck. We also design a spatial-invariant augmentation to assist model to be invariant to inaccurate detection results. Experimental results on the CUHK-SYSU and PRW datasets demonstrate the superiority of our method against existing approaches -- it achieves on par accuracy with state-of-the-art two-step methods while maintaining high efficiency due to the single joint model. Code is available at: https://git.io/DKD-PersonSearch.
Personalized fall detection monitoring system based on learning from the user movements
Venue: arXiv cs.LG 2020
Year: 2020

Personalized fall detection system is shown to provide added and more benefits compare to the current fall detection system. The personalized model can also be applied to anything where one class of data is hard to gather. The results show that adapting to the user needs, improve the overall accuracy of the system. Future work includes detection of the smartphone on the user so that the user can place the system anywhere on the body and make sure it detects. Even though the accuracy is not 100% the proof of concept of personalization can be used to achieve greater accuracy. The concept of personalization used in this paper can also be extended to other research in the medical field or where data is hard to come by for a particular class. More research into the feature extraction and feature selection module should be investigated. For the feature selection module, more research into selecting features based on one class data.
Neural Joint Entropy Estimation
Venue: arXiv cs.IT 2020
Year: 2020

Estimating the entropy of a discrete random variable is a fundamental problem in information theory and related fields. This problem has many applications in various domains, including machine learning, statistics and data compression. Over the years, a variety of estimation schemes have been suggested. However, despite significant progress, most methods still struggle when the sample is small, compared to the variable's alphabet size. In this work, we introduce a practical solution to this problem, which extends the work of McAllester and Statos (2020). The proposed scheme uses the generalization abilities of cross-entropy estimation in deep neural networks (DNNs) to introduce improved entropy estimation accuracy. Furthermore, we introduce a family of estimators for related information-theoretic measures, such as conditional entropy and mutual information. We show that these estimators are strongly consistent and demonstrate their performance in a variety of use-cases. First, we consider large alphabet entropy estimation. Then, we extend the scope to mutual information estimation. Next, we apply the proposed scheme to conditional mutual information estimation, as we focus on independence testing tasks. Finally, we study a transfer entropy estimation problem. The proposed estimators demonstrate improved performance compared to existing methods in all tested setups.
Conceptual Software Engineering Applied to Movie Scripts and Stories
Venue: arXiv cs.AI 2020
Year: 2020

This study introduces another application of software engineering tools, conceptual modeling, which can be applied to other fields of research. One way to strengthen the relationship between software engineering and other fields is to develop a good way to perform conceptual modeling that is capable of addressing the peculiarities of these fields of study. This study concentrates on humanities and social sciences, which are usually considered softer and further away from abstractions and (abstract) machines. Specifically, we focus on conceptual modeling as a software engineering tool (e.g., UML) in the area of stories and movie scripts. Researchers in the humanities and social sciences might not use the same degree of formalization that engineers do, but they still find conceptual modeling useful. Current modeling techniques (e.g., UML) fail in this task because they are geared toward the creation of software systems. Similar Conceptual Modeling Language (e.g., ConML) has been proposed with the humanities and social sciences in mind and, as claimed, can be used to model anything. This study is a venture in this direction, where a software modeling technique, Thinging Machine (TM), is applied to movie scripts and stories. The paper presents a novel approach to developing diagrammatic static/dynamic models of movie scripts and stories. The TM model diagram serves as a neutral and independent representation for narrative discourse and can be used as a communication instrument among participants. The examples presented include examples from Propp s model of fairytales; the railway children and an actual movie script seem to point to the viability of the approach.
Recoloring graphs of treewidth 2
Venue: arXiv cs.DM 2020
Year: 2020

Two (proper) colorings of a graph are adjacent if they differ on exactly one vertex. Jerrum proved that any $(d + 2)$-coloring of any d-degenerate graph can be transformed into any other via a sequence of adjacent colorings. A result of Bonamy et al. ensures that a shortest transformation can have a quadratic length even for $d = 1$. Bousquet and Perarnau proved that a linear transformation exists for between $(2d + 2)$-colorings. It is open to determine if this bound can be reduced. In this note, we prove that it can be reduced for graphs of treewidth 2, which are 2-degenerate. There exists a linear transformation between 5-colorings. It completes the picture for graphs of treewidth 2 since there exist graphs of treewidth 2 such a linear transformation between 4-colorings does not exist.