OpenRaiser/Intern-Atlas huggingface dataset:
Searching for Accurate Binary Neural Architectures
Venue: arXiv cs.LG 2019
Year: 2019

Binary neural networks have attracted tremendous attention due to the efficiency for deploying them on mobile devices. Since the weak expression ability of binary weights and features, their accuracy is usually much lower than that of full-precision (i.e. 32-bit) models. Here we present a new frame work for automatically searching for compact but accurate binary neural networks. In practice, number of channels in each layer will be encoded into the search space and optimized using the evolutionary algorithm. Experiments conducted on benchmark datasets and neural architectures demonstrate that our searched binary networks can achieve the performance of full-precision models with acceptable increments on model sizes and calculations.
Truthful and Faithful Monetary Policy for a Stablecoin Conducted by a Decentralised, Encrypted Artificial Intelligence
Venue: arXiv cs.CR 2019
Year: 2019

The Holy Grail of a decentralised stablecoin is achieved on rigorous mathematical frameworks, obtaining multiple advantageous proofs: stability, convergence, truthfulness, faithfulness, and malicious-security. These properties could only be attained by the novel and interdisciplinary combination of previously unrelated fields: model predictive control, deep learning, alternating direction method of multipliers (consensus-ADMM), mechanism design, secure multi-party computation, and zero-knowledge proofs. For the first time, this paper proves: - the feasibility of decentralising the central bank while securely preserving its independence in a decentralised computation setting - the benefits for price stability of combining mechanism design, provable security, and control theory, unlike the heuristics of previous stablecoins - the implementation of complex monetary policies on a stablecoin, equivalent to the ones used by central banks and beyond the current fixed rules of cryptocurrencies that hinder their price stability - methods to circumvent the impossibilities of Guaranteed Output Delivery (G.O.D.) and fairness: standing on truthfulness and faithfulness, we reach G.O.D. and fairness under the assumption of rational parties As a corollary, a decentralised artificial intelligence is able to conduct the monetary policy of a stablecoin, minimising human intervention.
Reproducibility of an airway tapering measurement in CT with application to bronchiectasis
Venue: arXiv cs.CV 2019
Year: 2019

Purpose: This paper proposes a pipeline to acquire a scalar tapering measurement from the carina to the most distal point of an individual airway visible on CT. We show the applicability of using tapering measurements on clinically acquired data by quantifying the reproducibility of the tapering measure. Methods: We generate a spline from the centreline of an airway to measure the area and arclength at contiguous intervals. The tapering measurement is the gradient of the linear regression between area in log space and arclength. The reproducibility of the measure was assessed by analysing different radiation doses, voxel sizes and reconstruction kernel on single timepoint and longitudinal CT scans and by evaluating the effct of airway bifurcations. Results: Using 74 airways from 10 CT scans, we show a statistical difference, p = 3.4 $\times$ 10$^{-4}$ in tapering between healthy airways (n = 35) and those affected by bronchiectasis (n = 39). The difference between the mean of the two populations was 0.011mm$^{-1}$ and the difference between the medians of the two populations was 0.006mm$^{-1}$. The tapering measurement retained a 95\% confidence interval of $\pm$0.005mm$^{-1}$ in a simulated 25 mAs scan and retained a 95% confidence of $\pm$0.005mm$^{-1}$ on simulated CTs up to 1.5 times the original voxel size. Conclusion: We have established an estimate of the precision of the tapering measurement and estimated the effect on precision of simulated voxel size and CT scan dose. We recommend that the scanner calibration be undertaken with the phantoms as described, on the specific CT scanner, radiation dose and reconstruction algorithm that is to be used in any quantitative studies. Our code is available at https://github.com/quan14/AirwayTaperingInCT
Short-Text Classification Using Unsupervised Keyword Expansion
Venue: arXiv cs.CL 2019
Year: 2019

Short-text classification, like all data science, struggles to achieve high performance using limited data. As a solution, a short sentence may be expanded with new and relevant feature words to form an artificially enlarged dataset, and add new features to testing data. This paper applies a novel approach to text expansion by generating new words directly for each input sentence, thus requiring no additional datasets or previous training. In this unsupervised approach, new keywords are formed within the hidden states of a pre-trained language model and then used to create extended pseudo documents. The word generation process was assessed by examining how well the predicted words matched to topics of the input sentence. It was found that this method could produce 3-10 relevant new words for each target topic, while generating just 1 word related to each non-target topic. Generated words were then added to short news headlines to create extended pseudo headlines. Experimental results have shown that models trained using the pseudo headlines can improve classification accuracy when limiting the number of training examples.
Probing Natural Language Inference Models through Semantic Fragments
Venue: arXiv cs.CL 2019
Year: 2019

Do state-of-the-art models for language understanding already have, or can they easily learn, abilities such as boolean coordination, quantification, conditionals, comparatives, and monotonicity reasoning (i.e., reasoning about word substitutions in sentential contexts)? While such phenomena are involved in natural language inference (NLI) and go beyond basic linguistic understanding, it is unclear the extent to which they are captured in existing NLI benchmarks and effectively learned by models. To investigate this, we propose the use of semantic fragments---systematically generated datasets that each target a different semantic phenomenon---for probing, and efficiently improving, such capabilities of linguistic models. This approach to creating challenge datasets allows direct control over the semantic diversity and complexity of the targeted linguistic phenomena, and results in a more precise characterization of a model's linguistic behavior. Our experiments, using a library of 8 such semantic fragments, reveal two remarkable findings: (a) State-of-the-art models, including BERT, that are pre-trained on existing NLI benchmark datasets perform poorly on these new fragments, even though the phenomena probed here are central to the NLI task. (b) On the other hand, with only a few minutes of additional fine-tuning---with a carefully selected learning rate and a novel variation of "inoculation"---a BERT-based model can master all of these logic and monotonicity fragments while retaining its performance on established NLI benchmarks.
Generalized Dictionary Matching under Substring Consistent Equivalence Relations
Venue: arXiv cs.DS 2019
Year: 2019

Given a set of patterns called a dictionary and a text, the dictionary matching problem is a task to find all occurrence positions of all patterns in the text. The dictionary matching problem can be solved efficiently by using the Aho-Corasick algorithm. Recently, Matsuoka et al. [TCS, 2016] proposed a generalization of pattern matching problem under substring consistent equivalence relations and presented a generalization of the Knuth-Morris-Pratt algorithm to solve this problem. An equivalence relation $\approx$ is a substring consistent equivalence relation (SCER) if for two strings $X,Y$, $X \approx Y$ implies $|X| = |Y|$ and $X[i:j] \approx Y[i:j]$ for all $1 \le i \le j \le |X|$. In this paper, we propose a generalization of the dictionary matching problem and present a generalization of the Aho-Corasick algorithm for the dictionary matching under SCER. We present an algorithm that constructs SCER automata and an algorithm that performs dictionary matching under SCER by using the automata. Moreover, we show the time and space complexity of our algorithms with respect to the size of input strings.
Hierarchical Reinforcement Learning for Open-Domain Dialog
Venue: arXiv cs.LG 2019
Year: 2019

Open-domain dialog generation is a challenging problem; maximum likelihood training can lead to repetitive outputs, models have difficulty tracking long-term conversational goals, and training on standard movie or online datasets may lead to the generation of inappropriate, biased, or offensive text. Reinforcement Learning (RL) is a powerful framework that could potentially address these issues, for example by allowing a dialog model to optimize for reducing toxicity and repetitiveness. However, previous approaches which apply RL to open-domain dialog generation do so at the word level, making it difficult for the model to learn proper credit assignment for long-term conversational rewards. In this paper, we propose a novel approach to hierarchical reinforcement learning, VHRL, which uses policy gradients to tune the utterance-level embedding of a variational sequence model. This hierarchical approach provides greater flexibility for learning long-term, conversational rewards. We use self-play and RL to optimize for a set of human-centered conversation metrics, and show that our approach provides significant improvements -- in terms of both human evaluation and automatic metrics -- over state-of-the-art dialog models, including Transformers.
Conformal Prediction based Spectral Clustering
Venue: arXiv cs.LG 2019
Year: 2019

Spectral Clustering(SC) is a prominent data clustering technique of recent times which has attracted much attention from researchers. It is a highly data-driven method and makes no strict assumptions on the structure of the data to be clustered. One of the central pieces of spectral clustering is the construction of an affinity matrix based on a similarity measure between data points. The way the similarity measure is defined between data points has a direct impact on the performance of the SC technique. Several attempts have been made in the direction of strengthening the pairwise similarity measure to enhance the spectral clustering. In this work, we have defined a novel affinity measure by employing the concept of non-conformity used in Conformal Prediction(CP) framework. The non-conformity based affinity captures the relationship between neighborhoods of data points and has the power to generalize the notion of contextual similarity. We have shown that this formulation of affinity measure gives good results and compares well with the state of the art methods.
K-BERT: Enabling Language Representation with Knowledge Graph
Venue: arXiv cs.CL 2019
Year: 2019

Pre-trained language representation models, such as BERT, capture a general language representation from large-scale corpora, but lack domain-specific knowledge. When reading a domain text, experts make inferences with relevant knowledge. For machines to achieve this capability, we propose a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge. However, too much knowledge incorporation may divert the sentence from its correct meaning, which is called knowledge noise (KN) issue. To overcome KN, K-BERT introduces soft-position and visible matrix to limit the impact of knowledge. K-BERT can easily inject domain knowledge into the models by equipped with a KG without pre-training by-self because it is capable of loading model parameters from the pre-trained BERT. Our investigation reveals promising results in twelve NLP tasks. Especially in domain-specific tasks (including finance, law, and medicine), K-BERT significantly outperforms BERT, which demonstrates that K-BERT is an excellent choice for solving the knowledge-driven problems that require experts.
Alleviating Feature Confusion for Generative Zero-shot Learning
Venue: arXiv cs.CV 2019
Year: 2019

Lately, generative adversarial networks (GANs) have been successfully applied to zero-shot learning (ZSL) and achieved state-of-the-art performance. By synthesizing virtual unseen visual features, GAN-based methods convert the challenging ZSL task into a supervised learning problem. However, GAN-based ZSL methods have to train the generator on the seen categories and further apply it to unseen instances. An inevitable issue of such a paradigm is that the synthesized unseen features are prone to seen references and incapable to reflect the novelty and diversity of real unseen instances. In a nutshell, the synthesized features are confusing. One cannot tell unseen categories from seen ones using the synthesized features. As a result, the synthesized features are too subtle to be classified in generalized zero-shot learning (GZSL) which involves both seen and unseen categories at the test stage. In this paper, we first introduce the feature confusion issue. Then, we propose a new feature generating network, named alleviating feature confusion GAN (AFC-GAN), to challenge the issue. Specifically, we present a boundary loss which maximizes the decision boundary of seen categories and unseen ones. Furthermore, a novel metric named feature confusion score (FCS) is proposed to quantify the feature confusion. Extensive experiments on five widely used datasets verify that our method is able to outperform previous state-of-the-arts under both ZSL and GZSL protocols.
Refined $α$-Divergence Variational Inference via Rejection Sampling
Venue: arXiv cs.LG 2019
Year: 2019

We present an approximate inference method, based on a synergistic combination of Rényi $α$-divergence variational inference (RDVI) and rejection sampling (RS). RDVI is based on minimization of Rényi $α$-divergence $D_α(p||q)$ between the true distribution $p(x)$ and a variational approximation $q(x)$; RS draws samples from a distribution $p(x) = \tilde{p}(x)/Z_{p}$ using a proposal $q(x)$, s.t. $Mq(x) \geq \tilde{p}(x), \forall x$. Our inference method is based on a crucial observation that $D_\infty(p||q)$ equals $\log M(θ)$ where $M(θ)$ is the optimal value of the RS constant for a given proposal $q_θ(x)$. This enables us to develop a \emph{two-stage} hybrid inference algorithm. Stage-1 performs RDVI to learn $q_θ$ by minimizing an estimator of $D_α(p||q)$, and uses the learned $q_θ$ to find an (approximately) optimal $\tilde{M}(θ)$. Stage-2 performs RS using the constant $\tilde{M}(θ)$ to improve the approximate distribution $q_θ$ and obtain a sample-based approximation. We prove that this two-stage method allows us to learn considerably more accurate approximations of the target distribution as compared to RDVI. We demonstrate our method's efficacy via several experiments on synthetic and real datasets.
Towards Efficient and Secure Delivery of Data for Deep Learning with Privacy-Preserving
Venue: arXiv cs.CR 2019
Year: 2019

Privacy recently emerges as a severe concern in deep learning, that is, sensitive data must be prohibited from being shared with the third party during deep neural network development. In this paper, we propose Morphed Learning (MoLe), an efficient and secure scheme to deliver deep learning data. MoLe has two main components: data morphing and Augmented Convolutional (Aug-Conv) layer. Data morphing allows data providers to send morphed data without privacy information, while Aug-Conv layer helps deep learning developers to apply their networks on the morphed data without performance penalty. MoLe provides stronger security while introducing lower overhead compared to GAZELLE (USENIX Security 2018), which is another method with no performance penalty on the neural network. When using MoLe for VGG-16 network on CIFAR dataset, the computational overhead is only 9% and the data transmission overhead is 5.12%. As a comparison, GAZELLE has computational overhead of 10,000 times and data transmission overhead of 421,000 times. In this setting, the attack success rate of adversary is 7.9 x 10^{-90} for MoLe and 2.9 x 10^{-30} for GAZELLE, respectively.
Communication-Efficient Weighted Sampling and Quantile Summary for GBDT
Venue: arXiv cs.DS 2019
Year: 2019

Gradient boosting decision tree (GBDT) is a powerful and widely-used machine learning model, which has achieved state-of-the-art performance in many academic areas and production environment. However, communication overhead is the main bottleneck in distributed training which can handle the massive data nowadays. In this paper, we propose two novel communication-efficient methods over distributed dataset to mitigate this problem, a weighted sampling approach by which we can estimate the information gain over a small subset efficiently, and distributed protocols for weighted quantile problem used in approximate tree learning.
Resource-Aware Automata and Games for Optimal Synthesis
Venue: arXiv cs.LO 2019
Year: 2019

We consider quantitative notions of parity automaton and parity game aimed at modelling resource-aware behaviour, and study (memory-full) strategies for exhibiting accepting runs that require a minimum amount of initial resources, respectively for winning a game with minimum initial resources. We also show how such strategies can be simplified to consist of only two types of moves: the former aimed at increasing resources, the latter aimed at satisfying the acceptance condition.
State Complexity of the Multiples of the Thue-Morse Set
Venue: arXiv cs.FL 2019
Year: 2019

The Thue-Morse set T is the set of those non-negative integers whose binary expansions have an even number of 1. The name of this set comes from the fact that its characteristic sequence is given by the famous Thue-Morse word abbabaabbaababba..., which is the fixed point starting with a of the word morphism sending a to ab and b to ba. The numbers in T are sometimes called the evil numbers. We obtain an exact formula for the state complexity (i.e. the number of states of its minimal automaton) of the multiplication by a constant of the Thue-Morse set with respect to any integer base b which is a power of 2. Our proof is constructive and we are able to explicitly provide the minimal automaton of the language of all 2^p-expansions of the set mT for any positive integers m and p. The used method is general for any b-recognizable set of integers. As an application, we obtain a decision procedure running in quadratic time for the problem of deciding whether a given 2^p-recognizable set is equal to some multiple of the Thue-Morse set.
Synchronous Maneuver Searching and Trajectory Planning for Autonomous Vehicles in Dynamic Traffic Environments
Venue: arXiv cs.RO 2019
Year: 2019

In the real-time decision-making and local planning process of autonomous vehicles in dynamic environments, the autonomous driving system may fail to find a reasonable policy or even gets trapped in some situation due to the complexity of global tasks and the incompatibility between upper-level maneuver decisions with the low-level lower level trajectory planning. To solve this problem, this paper presents a synchronous maneuver searching and trajectory planning (SMSTP) algorithm based on the topological concept of homotopy. Firstly, a set of alternative maneuvers with boundary limits are enumerated on a multi-lane road. Instead of sampling numerous paths in the whole spatio-temporal space, we, for the first time, propose using Trajectory Profiles (TPs) to quickly construct the topological maneuvers represented by different routes, and put forward a corridor generation algorithm based on graph-search. The bounded corridor further constrains the maneuver's space in the spatial space. A step-wise heuristic optimization algorithm is then proposed to synchronously generate a feasible trajectory for each maneuver. To achieve real-time performance, we initialize the states to be optimized with the boundary constraints of maneuvers, and we set some heuristic states as terminal targets in the quadratic cost function. The solution of a feasible trajectory is always guaranteed only if a specific maneuver is given. The simulation and realistic driving-test experiments verified that the proposed SMSTP algorithm has a short computation time which is less than 37ms, and the experimental results showed the validity and effectiveness of the SMSTP algorithm.
Multi-Task Learning for Automotive Foggy Scene Understanding via Domain Adaptation to an Illumination-Invariant Representation
Venue: arXiv cs.LG 2019
Year: 2019

Joint scene understanding and segmentation for automotive applications is a challenging problem in two key aspects:- (1) classifying every pixel in the entire scene and (2) performing this task under unstable weather and illumination changes (e.g. foggy weather), which results in poor outdoor scene visibility. This poor outdoor scene visibility leads to a non-optimal performance of deep convolutional neural network-based scene understanding and segmentation. In this paper, we propose an efficient end-to-end contemporary automotive semantic scene understanding approach under foggy weather conditions, employing domain adaptation and illumination-invariant image per-transformation. As a multi-task pipeline, our proposed model provides:- (1) transferring images from extreme to clear-weather condition using domain transfer approach and (2) semantically segmenting a scene using a competitive encoder-decoder convolutional neural network (CNN) with dense connectivity, skip connections and fusion-based techniques. We evaluate our approach on challenging foggy datasets, including synthetic dataset (Foggy Cityscapes) as well as real-world datasets (Foggy Zurich and Foggy Driving). By incorporating RGB, depth, and illumination-invariant information, our approach outperforms the state-of-the-art within automotive scene understanding, under foggy weather condition.
Building Change Detection for Remote Sensing Images Using a Dual Task Constrained Deep Siamese Convolutional Network Model
Venue: arXiv cs.CV 2019
Year: 2019

In recent years, building change detection methods have made great progress by introducing deep learning, but they still suffer from the problem of the extracted features not being discriminative enough, resulting in incomplete regions and irregular boundaries. To tackle this problem, we propose a dual task constrained deep Siamese convolutional network (DTCDSCN) model, which contains three sub-networks: a change detection network and two semantic segmentation networks. DTCDSCN can accomplish both change detection and semantic segmentation at the same time, which can help to learn more discriminative object-level features and obtain a complete change detection map. Furthermore, we introduce a dual attention module (DAM) to exploit the interdependencies between channels and spatial positions, which improves the feature representation. We also improve the focal loss function to suppress the sample imbalance problem. The experimental results obtained with the WHU building dataset show that the proposed method is effective for building change detection and achieves a state-of-the-art performance in terms of four metrics: precision, recall, F1-score, and intersection over union.
SocialNLP EmotionX 2019 Challenge Overview: Predicting Emotions in Spoken Dialogues and Chats
Venue: arXiv cs.CL 2019
Year: 2019

We present an overview of the EmotionX 2019 Challenge, held at the 7th International Workshop on Natural Language Processing for Social Media (SocialNLP), in conjunction with IJCAI 2019. The challenge entailed predicting emotions in spoken and chat-based dialogues using augmented EmotionLines datasets. EmotionLines contains two distinct datasets: the first includes excerpts from a US-based TV sitcom episode scripts (Friends) and the second contains online chats (EmotionPush). A total of thirty-six teams registered to participate in the challenge. Eleven of the teams successfully submitted their predictions performance evaluation. The top-scoring team achieved a micro-F1 score of 81.5% for the spoken-based dialogues (Friends) and 79.5% for the chat-based dialogues (EmotionPush).