OpenRaiser/Intern-Atlas huggingface dataset:
Ruminating Word Representations with Random Noised Masker
Venue: arXiv cs.LG 2019
Year: 2019

We introduce a training method for both better word representation and performance, which we call GROVER (Gradual Rumination On the Vector with maskERs). The method is to gradually and iteratively add random noises to word embeddings while training a model. GROVER first starts from conventional training process, and then extracts the fine-tuned representations. Next, we gradually add random noises to the word representations and repeat the training process from scratch, but initialize with the noised word representations. Through the re-training process, we can mitigate some noises to be compensated and utilize other noises to learn better representations. As a result, we can get word representations further fine-tuned and specialized on the task. When we experiment with our method on 5 text classification datasets, our method improves model performances on most of the datasets. Moreover, we show that our method can be combined with other regularization techniques, further improving the model performance.
IoTNetSim: A Modelling and Simulation Platform for End-to-End IoT Services and Networking
Venue: arXiv cs.DC 2019
Year: 2019

Internet-of-Things (IoT) systems are becoming increasingly complex, heterogeneous and pervasive, integrating a variety of physical devices and virtual services that are spread across architecture layers (cloud, fog, edge) using different connection types. As such, research and design of such systems have proven to be challenging. Despite the influx in IoT research and the significant benefits of simulation-based approaches in supporting research, there is a general lack of appropriate modelling and simulation platforms to create a detailed representation of end-to-end IoT services, i.e. from the underlying IoT nodes to the application layer in the cloud along with the underlying networking infrastructure. To aid researchers and practitioners in overcoming these challenges, we propose IoTNetSim, a novel self-contained extendable platform for modelling and simulation of end-to-end IoT services. The platform supports modelling heterogeneous IoT nodes (sensors, actuators, gateways, etc.) with their fine-grained details (mobility, energy profile, etc.), as well as different models of application logic and network connectivity. The proposed work is distinct from the current literature, being an all-in-one tool for end-to-end IoT services with a multi-layered architecture that allows modelling IoT systems with different structures. We experimentally validate and evaluate our IoTNetSim implementation using two very large-scale real-world cases from the natural environment and disaster monitoring IoT domains.
Face Detection in Camera Captured Images of Identity Documents under Challenging Conditions
Venue: arXiv cs.CV 2019
Year: 2019

Benefiting from the advance of deep convolutional neural network approaches (CNNs), many face detection algorithms have achieved state-of-the-art performance in terms of accuracy and very high speed in unconstrained applications. However, due to the lack of public datasets and due to the variation of the orientation of face images, the complex background and lighting, defocus and the varying illumination of camera captured images, face detection on identity documents under unconstrained environments has not been sufficiently studied. To address this problem more efficiently, we survey three state-of-the-art face detection methods based on general images, i.e. Cascade-CNN, MTCNN and PCN, for face detection in camera captured images of identity documents, given different image quality assessments. For that, The MIDV-500 dataset, which is the largest and most challenging dataset for identity documents, is used to evaluate the three methods. The evaluation results show the performance and the limitations of the current methods for face detection on identity documents under the wild complex environments. These results show that the face detection task in camera captured images of identity documents is challenging, providing a space to improve in the future works.
DeepMask: an algorithm for cloud and cloud shadow detection in optical satellite remote sensing images using deep residual network
Venue: arXiv cs.CV 2019
Year: 2019

Detecting and masking cloud and cloud shadow from satellite remote sensing images is a pervasive problem in the remote sensing community. Accurate and efficient detection of cloud and cloud shadow is an essential step to harness the value of remotely sensed data for almost all downstream analysis. DeepMask, a new algorithm for cloud and cloud shadow detection in optical satellite remote sensing imagery, is proposed in this study. DeepMask utilizes ResNet, a deep convolutional neural network, for pixel-level cloud mask generation. The algorithm is trained and evaluated on the Landsat 8 Cloud Cover Assessment Validation Dataset distributed across 8 different land types. Compared with CFMask, the most widely used cloud detection algorithm, land-type-specific DeepMask models achieve higher accuracy across all land types. The average accuracy is 93.56%, compared with 85.36% from CFMask. DeepMask also achieves 91.02% accuracy on all-land-type dataset. Compared with other CNN-based cloud mask algorithms, DeepMask benefits from the parsimonious architecture and the residual connection of ResNet. It is compatible with input of any size and shape. DeepMask still maintains high performance when using only red, green, blue, and NIR bands, indicating its potential to be applied to other satellite platforms that only have limited optical bands.
Cellular Automata: Reversibility, Semi-reversibility and Randomness
Venue: arXiv cs.FL 2019
Year: 2019

In this dissertation, we study two of the global properties of 1-dimensional cellular automata (CAs) under periodic boundary condition, namely, reversibility and randomness. To address reversibility of finite CAs, we develop a mathematical tool, named reachability tree, which can efficiently characterize those CAs. A decision algorithm is proposed using minimized reachability tree which takes a CA rule and size n as input and verifies whether the CA is reversible for that n. To decide reversibility of a finite CA, we need to know both the rule and the CA size. However, for infinite CAs, reversibility is decided based on the local rule only. Therefore, apparently, these two cases seem to be divergent. This dissertation targets to construct a bridge between these two cases. To do so, reversibility of CAs is redefined and the notion of semi-reversible CAs is introduced. Hence, we propose a new classification of finite CAs -(1) reversible CAs, (2) semi-reversible CAs and (3) strictly irreversible CAs. Finally, relation between reversibility of finite and infinite CAs is established. This dissertation also explores CAs as source of randomness and build pseudo-random number generators (PRNGs) based on CAs. We identify a list of properties for a CA to be a good source of randomness. Two heuristic algorithms are proposed to synthesize candidate (decimal) CAs which have great potentiality as PRNGs. Two schemes tare developed o use these CAs as window-based PRNGs - (1) as decimal number generators and as (2) binary number generators. We empirically observe that in comparison to the best PRNG SFMT19937-64, average performance of our proposed PRNGs are slightly better. Hence, our decimal CAs based PRNGs are one of the best PRNGs today.
Hate Speech Detection on Vietnamese Social Media Text using the Bi-GRU-LSTM-CNN Model
Venue: arXiv cs.CL 2019
Year: 2019

In recent years, Hate Speech Detection has become one of the interesting fields in natural language processing or computational linguistics. In this paper, we present the description of our system to solve this problem at the VLSP shared task 2019: Hate Speech Detection on Social Networks with the corpus which contains 20,345 human-labeled comments/posts for training and 5,086 for public-testing. We implement a deep learning method based on the Bi-GRU-LSTM-CNN classifier into this task. Our result in this task is 70.576% of F1-score, ranking the 5th of performance on public-test set.
Conditioned Query Generation for Task-Oriented Dialogue Systems
Venue: arXiv cs.CL 2019
Year: 2019

Scarcity of training data for task-oriented dialogue systems is a well known problem that is usually tackled with costly and time-consuming manual data annotation. An alternative solution is to rely on automatic text generation which, although less accurate than human supervision, has the advantage of being cheap and fast. In this paper we propose a novel controlled data generation method that could be used as a training augmentation framework for closed-domain dialogue. Our contribution is twofold. First we show how to optimally train and control the generation of intent-specific sentences using a conditional variational autoencoder. Then we introduce a novel protocol called query transfer that allows to leverage a broad, unlabelled dataset to extract relevant information. Comparison with two different baselines shows that our method, in the appropriate regime, consistently improves the diversity of the generated queries without compromising their quality.
Error Analysis for Vietnamese Dependency Parsing
Venue: arXiv cs.CL 2019
Year: 2019

Dependency parsing is needed in different applications of natural language processing. In this paper, we present a thorough error analysis for dependency parsing for the Vietnamese language, using two state-of-the-art parsers: MSTParser and MaltParser. The error analysis results provide us insights in order to improve the performance of dependency parsing for the Vietnamese language.
Universal Communication, Universal Graphs, and Graph Labeling
Venue: arXiv cs.CC 2019
Year: 2019

We introduce a communication model called universal SMP, in which Alice and Bob receive a function $f$ belonging to a family $\mathcal{F}$, and inputs $x$ and $y$. Alice and Bob use shared randomness to send a message to a third party who cannot see $f, x, y$, or the shared randomness, and must decide $f(x,y)$. Our main application of universal SMP is to relate communication complexity to graph labeling, where the goal is to give a short label to each vertex in a graph, so that adjacency or other functions of two vertices $x$ and $y$ can be determined from the labels $\ell(x),\ell(y)$. We give a universal SMP protocol using $O(k^2)$ bits of communication for deciding whether two vertices have distance at most $k$ on distributive lattices (generalizing the $k$-Hamming Distance problem in communication complexity), and explain how this implies an $O(k^2\log n)$ labeling scheme for determining $\mathrm{dist}(x,y) \leq k$ on distributive lattices with size $n$; in contrast, we show that a universal SMP protocol for determining $\mathrm{dist}(x,y) \leq 2$ in modular lattices (a superset of distributive lattices) has super-constant $Ω(n^{1/4})$ communication cost. On the other hand, we demonstrate that many graph families known to have efficient adjacency labeling schemes, such as trees, low-arboricity graphs, and planar graphs, admit constant-cost communication protocols for adjacency. Trees also have an $O(k)$ protocol for deciding $\mathrm{dist}(x,y) \leq k$ and planar graphs have an $O(1)$ protocol for $\mathrm{dist}(x,y) \leq 2$, which implies a new $O(\log n)$ labeling scheme for the same problem on planar graphs.
Prospect Theoretic Analysis of Privacy-Preserving Mechanism
Venue: arXiv cs.GT 2019
Year: 2019

We study a problem of privacy-preserving mechanism design. A data collector wants to obtain data from individuals to perform some computations. To relieve the privacy threat to the contributors, the data collector adopts a privacy-preserving mechanism by adding random noise to the computation result, at the cost of reduced accuracy. Individuals decide whether to contribute data when faced with the privacy issue. Due to the intrinsic uncertainty in privacy protection, we model individuals' privacy-related decision using Prospect Theory. Such a theory more accurately models individuals' behavior under uncertainty than the traditional expected utility theory, whose prediction always deviates from practical human behavior. We show that the data collector's utility maximization problem involves a polynomial of high and fractional order, the root of which is difficult to compute analytically. We get around this issue by considering a large population approximation, and obtain a closed-form solution that well approximates the precise solution. We discover that the data collector who considers the more realistic Prospect Theory based individual decision modeling would adopt a more conservative privacy-preserving mechanism, compared with the case based on the expected utility theory modeling. We also study the impact of Prospect Theory parameters, and concludes that more loss-averse or risk-seeking individuals will trigger a more conservative mechanism. When individuals have different Prospect Theory parameters, simulations demonstrate that the privacy protection first becomes stronger and then becomes weaker as the heterogeneity increases from a low value to a high one.
Embedded Neural Networks for Robot Autonomy
Venue: arXiv cs.RO 2019
Year: 2019

We present a library to automatically embed signal processing and neural network predictions into the material robots are made of. Deep and shallow neural network models are first trained offline using state-of-the-art machine learning tools and then transferred onto general purpose microcontrollers that are co-located with a robot's sensors and actuators. We validate this approach using multiple examples: a smart robotic tire for terrain classification, a robotic finger sensor for load classification and a smart composite capable of regressing impact source localization. In each example, sensing and computation are embedded inside the material, creating artifacts that serve as stand-in replacement for otherwise inert conventional parts. The open source software library takes as inputs trained model files from higher level learning software, such as Tensorflow/Keras, and outputs code that is readable in a microcontroller that supports C. We compare the performance of this approach for various embedded platforms. In particular, we show that low-cost off-the-shelf microcontrollers can match the accuracy of a desktop computer, while being fast enough for real-time applications at different neural network configurations. We provide means to estimate the maximum number of parameters that the hardware will support based on the microcontroller's specifications.
Understanding Multi-Head Attention in Abstractive Summarization
Venue: arXiv cs.CL 2019
Year: 2019

Attention mechanisms in deep learning architectures have often been used as a means of transparency and, as such, to shed light on the inner workings of the architectures. Recently, there has been a growing interest in whether or not this assumption is correct. In this paper we investigate the interpretability of multi-head attention in abstractive summarization, a sequence-to-sequence task for which attention does not have an intuitive alignment role, such as in machine translation. We first introduce three metrics to gain insight in the focus of attention heads and observe that these heads specialize towards relative positions, specific part-of-speech tags, and named entities. However, we also find that ablating and pruning these heads does not lead to a significant drop in performance, indicating redundancy. By replacing the softmax activation functions with sparsemax activation functions, we find that attention heads behave seemingly more transparent: we can ablate fewer heads and heads score higher on our interpretability metrics. However, if we apply pruning to the sparsemax model we find that we can prune even more heads, raising the question whether enforced sparsity actually improves transparency. Finally, we find that relative positions heads seem integral to summarization performance and persistently remain after pruning.
Customized video filtering on YouTube
Venue: arXiv cs.LG 2019
Year: 2019

Inappropriate and profane content on social media is exponentially increasing and big corporations are becoming more aware of the type of content on which they are advertising and how it may affect their brand reputation. But with a huge surge in content being posted online it becomes seemingly difficult to filter out related videos on which they can run their ads without compromising brand name. Advertising on youtube videos generates a huge amount of revenue for corporations. It becomes increasingly important for such corporations to advertise on only the videos that don't hurt the feelings, community or harmony of the audience at large. In this paper, we propose a system to identify inappropriate content on YouTube and leverage it to perform a first of its kind, large scale, quantitative characterization that reveals some of the risks of YouTube ads consumption on inappropriate videos. Customization of the architecture have also been included to serve different requirements of corporations. Our analysis reveals that YouTube is still plagued by such disturbing videos and its currently deployed countermeasures are ineffective in terms of detecting them in a timely manner. Our framework tries to fill this gap by providing a handy, add on solution to filter the videos and help corporations and companies to push ads on the platform without worrying about the content on which the ads are displayed.
DRiLLS: Deep Reinforcement Learning for Logic Synthesis
Venue: arXiv cs.AI 2019
Year: 2019

Logic synthesis requires extensive tuning of the synthesis optimization flow where the quality of results (QoR) depends on the sequence of optimizations used. Efficient design space exploration is challenging due to the exponential number of possible optimization permutations. Therefore, automating the optimization process is necessary. In this work, we propose a novel reinforcement learning-based methodology that navigates the optimization space without human intervention. We demonstrate the training of an Advantage Actor Critic (A2C) agent that seeks to minimize area subject to a timing constraint. Using the proposed methodology, designs can be optimized autonomously with no-humans in-loop. Evaluation on the comprehensive EPFL benchmark suite shows that the agent outperforms existing exploration methodologies and improves QoRs by an average of 13%.
Decompressing Knowledge Graph Representations for Link Prediction
Venue: arXiv cs.CL 2019
Year: 2019

This paper studies the problem of predicting missing relationships between entities in knowledge graphs through learning their representations. Currently, the majority of existing link prediction models employ simple but intuitive scoring functions and relatively small embedding size so that they could be applied to large-scale knowledge graphs. However, these properties also restrict the ability to learn more expressive and robust features. Therefore, diverging from most of the prior works which focus on designing new objective functions, we propose, DeCom, a simple but effective mechanism to boost the performance of existing link predictors such as DistMult, ComplEx, etc, through extracting more expressive features while preventing overfitting by adding just a few extra parameters. Specifically, embeddings of entities and relationships are first decompressed to a more expressive and robust space by decompressing functions, then knowledge graph embedding models are trained in this new feature space. Experimental results on several benchmark knowledge graphs and advanced link prediction systems demonstrate the generalization and effectiveness of our method. Especially, RESCAL + DeCom achieves state-of-the-art performance on the FB15k-237 benchmark across all evaluation metrics. In addition, we also show that compared with DeCom, explicitly increasing the embedding size significantly increase the number of parameters but could not achieve promising performance improvement.
Invariant Representations through Adversarial Forgetting
Venue: arXiv cs.LG 2019
Year: 2019

We propose a novel approach to achieving invariance for deep neural networks in the form of inducing amnesia to unwanted factors of data through a new adversarial forgetting mechanism. We show that the forgetting mechanism serves as an information-bottleneck, which is manipulated by the adversarial training to learn invariance to unwanted factors. Empirical results show that the proposed framework achieves state-of-the-art performance at learning invariance in both nuisance and bias settings on a diverse collection of datasets and tasks.
Generative Autoregressive Networks for 3D Dancing Move Synthesis from Music
Venue: arXiv cs.LG 2019
Year: 2019

This paper proposes a framework which is able to generate a sequence of three-dimensional human dance poses for a given music. The proposed framework consists of three components: a music feature encoder, a pose generator, and a music genre classifier. We focus on integrating these components for generating a realistic 3D human dancing move from music, which can be applied to artificial agents and humanoid robots. The trained dance pose generator, which is a generative autoregressive model, is able to synthesize a dance sequence longer than 5,000 pose frames. Experimental results of generated dance sequences from various songs show how the proposed method generates human-like dancing move to a given music. In addition, a generated 3D dance sequence is applied to a humanoid robot, showing that the proposed framework can make a robot to dance just by listening to music.
Program Synthesis by Type-Guided Abstraction Refinement
Venue: arXiv cs.PL 2019
Year: 2019

We consider the problem of type-directed component based synthesis where, given a set of (typed) components and a query type, the goal is to synthesize a term that inhabits the query. Classical approaches based on proof search in intuitionistic logics do not scale up to the standard libraries of modern languages, which span hundreds or thousands of components. Recent graph reachability based methods proposed for languages like Java do scale, but only apply to components over monomorphic data and functions: polymorphic data and functions infinitely explode the size of the graph that must be searched, rendering synthesis intractable. We introduce type-guided abstraction refinement (TYGAR), a new approach for scalable type-directed synthesis over polymorphic datatypes and components. Our key insight is that we can overcome the explosion by building a graph over abstract types which represent a potentially unbounded set of concrete types. We show how to use graph reachability to search for candidate terms over abstract types, and introduce a new algorithm that uses proofs of untypeability of ill-typed candidates to iteratively refine the abstraction until a well-typed result is found. We have implemented TYGAR in H+, a tool that takes as input a set of Haskell libraries and a query type, and returns a Haskell term that uses functions from the provided libraries to implement the query type. We have evaluated H+ on a set of 44 queries using a set of popular Haskell libraries with a total of 291 components. Our results demonstrate that H+ returns an interesting solution within the first five results for 33 out of 44 queries. Moreover, TYGAR allows H+ to rapidly return well-typed terms, with the median time to first solution of just 1.4 seconds.