OpenRaiser/Intern-Atlas huggingface dataset:
Extreme Theory of Functional Connections: A Physics-Informed Neural Network Method for Solving Parametric Differential Equations
Venue: arXiv cs.LG 2020
Year: 2020

In this work we present a novel, accurate, and robust physics-informed method for solving problems involving parametric differential equations (DEs) called the Extreme Theory of Functional Connections, or X-TFC. The proposed method is a synergy of two recently developed frameworks for solving problems involving parametric DEs, 1) the Theory of Functional Connections, TFC, and the Physics-Informed Neural Networks, PINN. Although this paper focuses on the solution of exact problems involving parametric DEs (i.e. problems where the modeling error is negligible) with known parameters, X-TFC can also be used for data-driven solutions and data-driven discovery of parametric DEs. In the proposed method, the latent solution of the parametric DEs is approximated by a TFC constrained expression that uses a Neural Network (NN) as the free-function. This approximate solution form always analytically satisfies the constraints of the DE, while maintaining a NN with unconstrained parameters, like the Deep-TFC method. X-TFC differs from PINN and Deep-TFC; whereas PINN and Deep-TFC use a deep-NN, X-TFC uses a single-layer NN, or more precisely, an Extreme Learning Machine, ELM. This choice is based on the properties of the ELM algorithm. In order to numerically validate the method, it was tested over a range of problems including the approximation of solutions to linear and non-linear ordinary DEs (ODEs), systems of ODEs (SODEs), and partial DEs (PDEs). Furthermore, a few of these problems are of interest in physics and engineering such as the Classic Emden-Fowler equation, the Radiative Transfer (RT) equation, and the Heat-Transfer (HT) equation. The results show that X-TFC achieves high accuracy with low computational time and thus it is comparable with the other state-of-the-art methods.
Neural ODEs for Informative Missingness in Multivariate Time Series
Venue: arXiv cs.LG 2020
Year: 2020

Informative missingness is unavoidable in the digital processing of continuous time series, where the value for one or more observations at different time points are missing. Such missing observations are one of the major limitations of time series processing using deep learning. Practical applications, e.g., sensor data, healthcare, weather, generates data that is in truth continuous in time, and informative missingness is a common phenomenon in these datasets. These datasets often consist of multiple variables, and often there are missing values for one or many of these variables. This characteristic makes time series prediction more challenging, and the impact of missing input observations on the accuracy of the final output can be significant. A recent novel deep learning model called GRU-D is one early attempt to address informative missingness in time series data. On the other hand, a new family of neural networks called Neural ODEs (Ordinary Differential Equations) are natural and efficient for processing time series data which is continuous in time. In this paper, a deep learning model is proposed that leverages the effective imputation of GRU-D, and the temporal continuity of Neural ODEs. A time series classification task performed on the PhysioNet dataset demonstrates the performance of this architecture.
Distributed Verifiers in PCP
Venue: arXiv cs.CC 2020
Year: 2020

Traditional proof systems involve a resource-bounded verifier communicating with a powerful (but untrusted) prover. Distributed verifier proof systems are a new family of proof models that involve a network of verifier nodes communicating with a single independent prover that has access to the complete network structure of the verifiers. The prover is tasked with convincing all verifiers of some global property of the network graph. In addition, each individual verifier may be given some input string they will be required to verify during the course of computation. Verifier nodes are allowed to exchange messaged with nodes a constant distance away, and accept / reject the input after some computation. Because individual nodes are limited to a local view, communication with the prover is potentially necessary to prove global properties about the network graph of nodes, which only the prover has access to. In this system of models, the entire model accepts the input if and only if every individual node has accepted. There are three models in the distributed verifier proof system family: $\mathsf{LCP}$, $\mathsf{dIP}$, and our proposed $\mathsf{dPCP}$, with the fundamental difference between these coming from the type of communication established between the verifiers and the prover. In this paper, we will first go over the past work in the $\mathsf{LCP}$ and $\mathsf{dIP}$ space before showing properties and proofs in our $\mathsf{dPCP}$ system.
Efficient Ensemble Model Generation for Uncertainty Estimation with Bayesian Approximation in Segmentation
Venue: arXiv cs.CV 2020
Year: 2020

Recent studies have shown that ensemble approaches could not only improve accuracy and but also estimate model uncertainty in deep learning. However, it requires a large number of parameters according to the increase of ensemble models for better prediction and uncertainty estimation. To address this issue, a generic and efficient segmentation framework to construct ensemble segmentation models is devised in this paper. In the proposed method, ensemble models can be efficiently generated by using the stochastic layer selection method. The ensemble models are trained to estimate uncertainty through Bayesian approximation. Moreover, to overcome its limitation from uncertain instances, we devise a new pixel-wise uncertainty loss, which improves the predictive performance. To evaluate our method, comprehensive and comparative experiments have been conducted on two datasets. Experimental results show that the proposed method could provide useful uncertainty information by Bayesian approximation with the efficient ensemble model generation and improve the predictive performance.
rTop-k: A Statistical Estimation Approach to Distributed SGD
Venue: arXiv cs.LG 2020
Year: 2020

The large communication cost for exchanging gradients between different nodes significantly limits the scalability of distributed training for large-scale learning models. Motivated by this observation, there has been significant recent interest in techniques that reduce the communication cost of distributed Stochastic Gradient Descent (SGD), with gradient sparsification techniques such as top-k and random-k shown to be particularly effective. The same observation has also motivated a separate line of work in distributed statistical estimation theory focusing on the impact of communication constraints on the estimation efficiency of different statistical models. The primary goal of this paper is to connect these two research lines and demonstrate how statistical estimation models and their analysis can lead to new insights in the design of communication-efficient training techniques. We propose a simple statistical estimation model for the stochastic gradients which captures the sparsity and skewness of their distribution. The statistically optimal communication scheme arising from the analysis of this model leads to a new sparsification technique for SGD, which concatenates random-k and top-k, considered separately in the prior literature. We show through extensive experiments on both image and language domains with CIFAR-10, ImageNet, and Penn Treebank datasets that the concatenated application of these two sparsification methods consistently and significantly outperforms either method applied alone.
Chang's lemma via Pinsker's inequality
Venue: arXiv cs.DM 2020
Year: 2020

Extending the idea in [Impagliazzo, R., Moore, C. and Russell, A., An entropic proof of Chang's inequality. SIAM Journal on Discrete Mathematics, 28(1), pp.173-176.] we give a short information theoretic proof for Chang's lemma that is based on Pinsker's inequality.
Online Coloring and a New Type of Adversary for Online Graph Problems
Venue: arXiv cs.DS 2020
Year: 2020

We introduce a new type of adversary for online graph problems. The new adversary is parameterized by a single integer $κ$, which upper bounds the number of connected components that the adversary can use at any time during the presentation of the online graph $G$. We call this adversary "$κ$ components bounded", or $κ$-CB for short. On one hand, this adversary is restricted compared to the classical adversary because of the $κ$-CB constraint. On the other hand, we seek competitive ratios parameterized only by $κ$ with no dependence on the input length $n$, thereby giving the new adversary power to use arbitrarily large inputs. We study online coloring under the $κ$-CB adversary. We obtain finer analysis of the existing algorithms $FirstFit$ and $CBIP$ by computing their competitive ratios on trees and bipartite graphs under the new adversary. Surprisingly, $FirstFit$ outperforms $CBIP$ on trees. When it comes to bipartite graphs $FirstFit$ is no longer competitive under the new adversary, while $CBIP$ uses at most $2κ$ colors. We also study several well known classes of graphs, such as $3$-colorable, $C_k$-free, $d$-inductive, planar, and bounded treewidth, with respect to online coloring under the $κ$-CB adversary. We demonstrate that the extra adversarial power of unbounded input length outweighs the restriction on the number of connected components leading to non existence of competitive algorithms for these classes.
Blind Two-Dimensional Super Resolution in Multiple Input Single Output Linear Systems
Venue: arXiv cs.IT 2020
Year: 2020

In this paper, we consider a multiple-input single-output (MISO) linear time-varying system whose output is a superposition of scaled and time-frequency shifted versions of inputs. The goal of this paper is to determine system characteristics and input signals from the single output signal. More precisely, we want to recover the continuous time-frequency shift pairs, the corresponding (complex-valued) amplitudes and the input signals from only one output vector. This problem arises in a variety of applications such as radar imaging, microscopy, channel estimation and localization problems. While this problem is naturally ill-posed, by constraining the unknown input waveforms to lie in separate known low-dimensional subspaces, it becomes tractable. More explicitly, we propose a semidefinite program which exactly recovers time-frequency shift pairs and input signals. We prove uniqueness and optimality of the solution to this program. Moreover, we provide a grid-based approach which can significantly reduce computational complexity in exchange for adding a small gridding error. Numerical results confirm the ability of our proposed method to exactly recover the unknowns.
Evaluating Neural Morphological Taggers for Sanskrit
Venue: arXiv cs.CL 2020
Year: 2020

Neural sequence labelling approaches have achieved state of the art results in morphological tagging. We evaluate the efficacy of four standard sequence labelling models on Sanskrit, a morphologically rich, fusional Indian language. As its label space can theoretically contain more than 40,000 labels, systems that explicitly model the internal structure of a label are more suited for the task, because of their ability to generalise to labels not seen during training. We find that although some neural models perform better than others, one of the common causes for error for all of these models is mispredictions due to syncretism.
CHEER: Rich Model Helps Poor Model via Knowledge Infusion
Venue: arXiv cs.LG 2020
Year: 2020

There is a growing interest in applying deep learning (DL) to healthcare, driven by the availability of data with multiple feature channels in rich-data environments (e.g., intensive care units). However, in many other practical situations, we can only access data with much fewer feature channels in a poor-data environments (e.g., at home), which often results in predictive models with poor performance. How can we boost the performance of models learned from such poor-data environment by leveraging knowledge extracted from existing models trained using rich data in a related environment? To address this question, we develop a knowledge infusion framework named CHEER that can succinctly summarize such rich model into transferable representations, which can be incorporated into the poor model to improve its performance. The infused model is analyzed theoretically and evaluated empirically on several datasets. Our empirical results showed that CHEER outperformed baselines by 5.60% to 46.80% in terms of the macro-F1 score on multiple physiological datasets.
A Concise Review of Recent Few-shot Meta-learning Methods
Venue: arXiv cs.LG 2020
Year: 2020

Few-shot meta-learning has been recently reviving with expectations to mimic humanity's fast adaption to new concepts based on prior knowledge. In this short communication, we give a concise review on recent representative methods in few-shot meta-learning, which are categorized into four branches according to their technical characteristics. We conclude this review with some vital current challenges and future prospects in few-shot meta-learning.
RankPose: Learning Generalised Feature with Rank Supervision for Head Pose Estimation
Venue: arXiv cs.CV 2020
Year: 2020

We address the challenging problem of RGB image-based head pose estimation. We first reformulate head pose representation learning to constrain it to a bounded space. Head pose represented as vector projection or vector angles shows helpful to improving performance. Further, a ranking loss combined with MSE regression loss is proposed. The ranking loss supervises a neural network with paired samples of the same person and penalises incorrect ordering of pose prediction. Analysis on this new loss function suggests it contributes to a better local feature extractor, where features are generalised to Abstract Landmarks which are pose-related features instead of pose-irrelevant information such as identity, age, and lighting. Extensive experiments show that our method significantly outperforms the current state-of-the-art schemes on public datasets: AFLW2000 and BIWI. Our model achieves significant improvements over previous SOTA MAE on AFLW2000 and BIWI from 4.50 to 3.66 and from 4.0 to 3.71 respectively. Source code will be made available at: https://github.com/seathiefwang/RankHeadPose.
A Convolutional Neural Network with Parallel Multi-Scale Spatial Pooling to Detect Temporal Changes in SAR Images
Venue: arXiv cs.CV 2020
Year: 2020

In synthetic aperture radar (SAR) image change detection, it is quite challenging to exploit the changing information from the noisy difference image subject to the speckle. In this paper, we propose a multi-scale spatial pooling (MSSP) network to exploit the changed information from the noisy difference image. Being different from the traditional convolutional network with only mono-scale pooling kernels, in the proposed method, multi-scale pooling kernels are equipped in a convolutional network to exploit the spatial context information on changed regions from the difference image. Furthermore, to verify the generalization of the proposed method, we apply our proposed method to the cross-dataset bitemporal SAR image change detection, where the MSSP network (MSSP-Net) is trained on a dataset and then applied to an unknown testing dataset. We compare the proposed method with other state-of-arts and the comparisons are performed on four challenging datasets of bitemporal SAR images. Experimental results demonstrate that our proposed method obtains comparable results with S-PCA-Net on YR-A and YR-B dataset and outperforms other state-of-art methods, especially on the Sendai-A and Sendai-B datasets with more complex scenes. More important, MSSP-Net is more efficient than S-PCA-Net and convolutional neural networks (CNN) with less executing time in both training and testing phases.
Reinforcement learning with human advice: a survey
Venue: arXiv cs.AI 2020
Year: 2020

In this paper, we provide an overview of the existing methods for integrating human advice into a Reinforcement Learning process. We first propose a taxonomy of the different forms of advice that can be provided to a learning agent. We then describe the methods that can be used for interpreting advice when its meaning is not determined beforehand. Finally, we review different approaches for integrating advice into the learning process.
Classification and Clustering of arXiv Documents, Sections, and Abstracts, Comparing Encodings of Natural and Mathematical Language
Venue: arXiv cs.DL 2020
Year: 2020

In this paper, we show how selecting and combining encodings of natural and mathematical language affect classification and clustering of documents with mathematical content. We demonstrate this by using sets of documents, sections, and abstracts from the arXiv preprint server that are labeled by their subject class (mathematics, computer science, physics, etc.) to compare different encodings of text and formulae and evaluate the performance and runtimes of selected classification and clustering algorithms. Our encodings achieve classification accuracies up to $82.8\%$ and cluster purities up to $69.4\%$ (number of clusters equals number of classes), and $99.9\%$ (unspecified number of clusters) respectively. We observe a relatively low correlation between text and math similarity, which indicates the independence of text and formulae and motivates treating them as separate features of a document. The classification and clustering can be employed, e.g., for document search and recommendation. Furthermore, we show that the computer outperforms a human expert when classifying documents. Finally, we evaluate and discuss multi-label classification and formula semantification.
Discovering Frequent Gradual Itemsets with Imprecise Data
Venue: arXiv cs.LG 2020
Year: 2020

The gradual patterns that model the complex co-variations of attributes of the form "The more/less X, The more/less Y" play a crucial role in many real world applications where the amount of numerical data to manage is important, this is the biological data. Recently, these types of patterns have caught the attention of the data mining community, where several methods have been defined to automatically extract and manage these patterns from different data models. However, these methods are often faced the problem of managing the quantity of mined patterns, and in many practical applications, the calculation of all these patterns can prove to be intractable for the user-defined frequency threshold and the lack of focus leads to generating huge collections of patterns. Moreover another problem with the traditional approaches is that the concept of gradualness is defined just as an increase or a decrease. Indeed, a gradualness is considered as soon as the values of the attribute on both objects are different. As a result, numerous quantities of patterns extracted by traditional algorithms can be presented to the user although their gradualness is only a noise effect in the data. To address this issue, this paper suggests to introduce the gradualness thresholds from which to consider an increase or a decrease. In contrast to literature approaches, the proposed approach takes into account the distribution of attribute values, as well as the user's preferences on the gradualness threshold and makes it possible to extract gradual patterns on certain databases where literature approaches fail due to too large search space. Moreover, results from an experimental evaluation on real databases show that the proposed algorithm is scalable, efficient, and can eliminate numerous patterns that do not verify specific gradualness requirements to show a small set of patterns to the user.
Risk scoring calculation for the current NHSx contact tracing app
Venue: arXiv cs.CY 2020
Year: 2020

We consider how the NHS COVID-19 application will initially calculate a risk score for an individual based on their recent contact with people who report that they have coronavirus symptoms.
A Comparative Evaluation of Heart Rate Estimation Methods using Face Videos
Venue: arXiv cs.CV 2020
Year: 2020

This paper presents a comparative evaluation of methods for remote heart rate estimation using face videos, i.e., given a video sequence of the face as input, methods to process it to obtain a robust estimation of the subjects heart rate at each moment. Four alternatives from the literature are tested, three based in hand crafted approaches and one based on deep learning. The methods are compared using RGB videos from the COHFACE database. Experiments show that the learning-based method achieves much better accuracy than the hand crafted ones. The low error rate achieved by the learning based model makes possible its application in real scenarios, e.g. in medical or sports environments.
Optimal Lower Bounds for Matching and Vertex Cover in Dynamic Graph Streams
Venue: arXiv cs.DS 2020
Year: 2020

In this paper, we give simple optimal lower bounds on the one-way two-party communication complexity of approximate Maximum Matching and Minimum Vertex Cover with deletions. In our model, Alice holds a set of edges and sends a single message to Bob. Bob holds a set of edge deletions, which form a subset of Alice's edges, and needs to report a large matching or a small vertex cover in the graph spanned by the edges that are not deleted. Our results imply optimal space lower bounds for insertion-deletion streaming algorithms for Maximum Matching and Minimum Vertex Cover. Previously, Assadi et al. [SODA 2016] gave an optimal space lower bound for insertion-deletion streaming algorithms for Maximum Matching via the simultaneous model of communication. Our lower bound is simpler and stronger in several aspects: The lower bound of Assadi et al. only holds for algorithms that (1) are able to process streams that contain a triple exponential number of deletions in $n$, the number of vertices of the input graph; (2) are able to process multi-graphs; and (3) never output edges that do not exist in the input graph when the randomized algorithm errs. In contrast, our lower bound even holds for algorithms that (1) rely on short ($O(n^2)$-length) input streams; (2) are only able to process simple graphs; and (3) may output non-existing edges when the algorithm errs.
Hermes: Enabling Energy-efficient IoT Networks with Generalized Deduplication
Venue: arXiv cs.DC 2020
Year: 2020

With the advent of the Internet of Things (IoT), the ever growing number of connected devices observed in recent years and foreseen for the next decade suggests that more and more data will have to be transmitted over a network, before being processed and stored in data centers. Generalized deduplication (GD) is a novel technique to effectively reduce the data storage cost by identifying similar data chunks, and able to gradually reduce the pressure from the network infrastructure by limiting the data that needs to be transmitted. This paper presents Hermes, an application-level protocol for the data-plane that can operate over generalized deduplication, as well as over classic deduplication. Hermes significantly reduces the data transmission traffic while effectively decreasing the energy footprint, a relevant matter to consider in the context of IoT deployments. We fully implemented Hermes and evaluated its performance using consumer-grade IoT devices (e.g., Raspberry Pi 4B models). Our results highlight several trade-offs that must be taken into account when considering real-world workloads.