OpenRaiser/Intern-Atlas huggingface dataset:
Overcoming Overfitting and Large Weight Update Problem in Linear Rectifiers: Thresholded Exponential Rectified Linear Units
Venue: arXiv cs.LG 2020
Year: 2020

In past few years, linear rectified unit activation functions have shown its significance in the neural networks, surpassing the performance of sigmoid activations. RELU (Nair & Hinton, 2010), ELU (Clevert et al., 2015), PRELU (He et al., 2015), LRELU (Maas et al., 2013), SRELU (Jin et al., 2016), ThresholdedRELU, all these linear rectified activation functions have its own significance over others in some aspect. Most of the time these activation functions suffer from bias shift problem due to non-zero output mean, and high weight update problem in deep complex networks due to unit gradient, which results in slower training, and high variance in model prediction respectively. In this paper, we propose, "Thresholded exponential rectified linear unit" (TERELU) activation function that works better in alleviating in overfitting: large weight update problem. Along with alleviating overfitting problem, this method also gives good amount of non-linearity as compared to other linear rectifiers. We will show better performance on the various datasets using neural networks, considering TERELU activation method compared to other activations.
Refined Continuous Control of DDPG Actors via Parametrised Activation
Venue: arXiv cs.LG 2020
Year: 2020

In this paper, we propose enhancing actor-critic reinforcement learning agents by parameterising the final actor layer which produces the actions in order to accommodate the behaviour discrepancy of different actuators, under different load conditions during interaction with the environment. We propose branching the action producing layer in the actor to learn the tuning parameter controlling the activation layer (e.g. Tanh and Sigmoid). The learned parameters are then used to create tailored activation functions for each actuator. We ran experiments on three OpenAI Gym environments, i.e. Pendulum-v0, LunarLanderContinuous-v2 and BipedalWalker-v2. Results have shown an average of 23.15% and 33.80% increase in total episode reward of the LunarLanderContinuous-v2 and BipedalWalker-v2 environments, respectively. There was no significant improvement in Pendulum-v0 environment but the proposed method produces a more stable actuation signal compared to the state-of-the-art method. The proposed method allows the reinforcement learning actor to produce more robust actions that accommodate the discrepancy in the actuators' response functions. This is particularly useful for real life scenarios where actuators exhibit different response functions depending on the load and the interaction with the environment. This also simplifies the transfer learning problem by fine tuning the parameterised activation layers instead of retraining the entire policy every time an actuator is replaced. Finally, the proposed method would allow better accommodation to biological actuators (e.g. muscles) in biomechanical systems.
Event-based visual place recognition with ensembles of temporal windows
Venue: arXiv cs.CV 2020
Year: 2020

Event cameras are bio-inspired sensors capable of providing a continuous stream of events with low latency and high dynamic range. As a single event only carries limited information about the brightness change at a particular pixel, events are commonly accumulated into spatio-temporal windows for further processing. However, the optimal window length varies depending on the scene, camera motion, the task being performed, and other factors. In this research, we develop a novel ensemble-based scheme for combining temporal windows of varying lengths that are processed in parallel. For applications where the increased computational requirements of this approach are not practical, we also introduce a new "approximate" ensemble scheme that achieves significant computational efficiencies without unduly compromising the original performance gains provided by the ensemble approach. We demonstrate our ensemble scheme on the visual place recognition (VPR) task, introducing a new Brisbane-Event-VPR dataset with annotated recordings captured using a DAVIS346 color event camera. We show that our proposed ensemble scheme significantly outperforms all the single-window baselines and conventional model-based ensembles, irrespective of the image reconstruction and feature extraction methods used in the VPR pipeline, and evaluate which ensemble combination technique performs best. These results demonstrate the significant benefits of ensemble schemes for event camera processing in the VPR domain and may have relevance to other related processes, including feature tracking, visual-inertial odometry, and steering prediction in driving.
Network size and weights size for memorization with two-layers neural networks
Venue: arXiv cs.LG 2020
Year: 2020

In 1988, Eric B. Baum showed that two-layers neural networks with threshold activation function can perfectly memorize the binary labels of $n$ points in general position in $\mathbb{R}^d$ using only $\ulcorner n/d \urcorner$ neurons. We observe that with ReLU networks, using four times as many neurons one can fit arbitrary real labels. Moreover, for approximate memorization up to error $ε$, the neural tangent kernel can also memorize with only $O\left(\frac{n}{d} \cdot \log(1/ε) \right)$ neurons (assuming that the data is well dispersed too). We show however that these constructions give rise to networks where the magnitude of the neurons' weights are far from optimal. In contrast we propose a new training procedure for ReLU networks, based on complex (as opposed to real) recombination of the neurons, for which we show approximate memorization with both $O\left(\frac{n}{d} \cdot \frac{\log(1/ε)}ε\right)$ neurons, as well as nearly-optimal size of the weights.
ToGCom: An Asymmetric Sybil Defense
Venue: arXiv cs.DC 2020
Year: 2020

Proof-of-work (PoW) is one of the most common techniques to defend against Sybil attacks. Unfortunately, current PoW defenses have two main drawbacks. First, they require work to be done even in the absence of an attack. Second, during an attack, they require good identities (IDs) to spend as much as the attacker. Recent theoretical work by Gupta, Saia, and Young suggests the possibility of overcoming these two drawbacks. In particular, they describe a new algorithm, GMCom, that always ensures that a minority of IDs are Sybil. They show that rate at which all good IDs perform computation is $O(J_G + \sqrt{T(J_G+1)})$, where $J_G$ is the join rate of good IDs, and $T$ is the rate at which the adversary performs computation. Unfortunately, this cost bound only holds in the case where (1) GMCom always knows the join rate of good IDs; and (2) there is a fixed constant amount of time that separates join events by good IDs. Here, we present ToGCom, which removes these two shortcomings. To do so, we design and analyze a mechanism for estimating the join rate of good IDs; and also devise a new method for setting the computational cost to join the system. Additionally, we evaluate the performance of ToGCom alongside prior PoW-based defenses. Based on our experiments, we design heuristics that further improve the performance of ToGCom by up to $3$ orders of magnitude over these previous Sybil defenses.
Jamming-Aware Control Plane in Elastic Optical Networks
Venue: arXiv cs.NI 2020
Year: 2020

Physical layer security is essential in optical networks. In this paper, we study a jamming-aware control plane, in which a high power jamming attack exists in the network. The studied control plane considers that the jammed connections can be detected and avoided. We used a physical layer model, in which we embedded the additional jamming power, to evaluate different security in scenarios, such as a jamming-free scenario, jamming with an unaware controller, and jamming with an aware controller. The performance is analyzed in terms of the blocking rate and slots utilization. We analyze the impact of jamming attacks in the least used link and in the most used link on the network. The results demonstrates that the jamming avoidance by the control plane can reach performance near the not jammed scenario.
Syntactic Search by Example
Venue: arXiv cs.CL 2020
Year: 2020

We present a system that allows a user to search a large linguistically annotated corpus using syntactic patterns over dependency graphs. In contrast to previous attempts to this effect, we introduce a light-weight query language that does not require the user to know the details of the underlying syntactic representations, and instead to query the corpus by providing an example sentence coupled with simple markup. Search is performed at an interactive speed due to an efficient linguistic graph-indexing and retrieval engine. This allows for rapid exploration, development and refinement of syntax-based queries. We demonstrate the system using queries over two corpora: the English wikipedia, and a collection of English pubmed abstracts. A demo of the wikipedia system is available at: https://allenai.github.io/spike
SOLO: A Corpus of Tweets for Examining the State of Being Alone
Venue: arXiv cs.CL 2020
Year: 2020

The state of being alone can have a substantial impact on our lives, though experiences with time alone diverge significantly among individuals. Psychologists distinguish between the concept of solitude, a positive state of voluntary aloneness, and the concept of loneliness, a negative state of dissatisfaction with the quality of one's social interactions. Here, for the first time, we conduct a large-scale computational analysis to explore how the terms associated with the state of being alone are used in online language. We present SOLO (State of Being Alone), a corpus of over 4 million tweets collected with query terms 'solitude', 'lonely', and 'loneliness'. We use SOLO to analyze the language and emotions associated with the state of being alone. We show that the term 'solitude' tends to co-occur with more positive, high-dominance words (e.g., enjoy, bliss) while the terms 'lonely' and 'loneliness' frequently co-occur with negative, low-dominance words (e.g., scared, depressed), which confirms the conceptual distinctions made in psychology. We also show that women are more likely to report on negative feelings of being lonely as compared to men, and there are more teenagers among the tweeters that use the word 'lonely' than among the tweeters that use the word 'solitude'.
Hierarchical Optimal Transport for Robust Multi-View Learning
Venue: arXiv cs.LG 2020
Year: 2020

Traditional multi-view learning methods often rely on two assumptions: ($i$) the samples in different views are well-aligned, and ($ii$) their representations in latent space obey the same distribution. Unfortunately, these two assumptions may be questionable in practice, which limits the application of multi-view learning. In this work, we propose a hierarchical optimal transport (HOT) method to mitigate the dependency on these two assumptions. Given unaligned multi-view data, the HOT method penalizes the sliced Wasserstein distance between the distributions of different views. These sliced Wasserstein distances are used as the ground distance to calculate the entropic optimal transport across different views, which explicitly indicates the clustering structure of the views. The HOT method is applicable to both unsupervised and semi-supervised learning, and experimental results show that it performs robustly on both synthetic and real-world tasks.
Evaluating Text Coherence at Sentence and Paragraph Levels
Venue: arXiv cs.CL 2020
Year: 2020

In this paper, to evaluate text coherence, we propose the paragraph ordering task as well as conducting sentence ordering. We collected four distinct corpora from different domains on which we investigate the adaptation of existing sentence ordering methods to a paragraph ordering task. We also compare the learnability and robustness of existing models by artificially creating mini datasets and noisy datasets respectively and verifying the efficiency of established models under these circumstances. Furthermore, we carry out human evaluation on the rearranged passages from two competitive models and confirm that WLCS-l is a better metric performing significantly higher correlations with human rating than tau, the most prevalent metric used before. Results from these evaluations show that except for certain extreme conditions, the recurrent graph neural network-based model is an optimal choice for coherence modeling.
A Characterization of Semi-Synchrony for Asynchronous Robots with Limited Visibility, and its Application to Luminous Synchronizer Design
Venue: arXiv cs.DC 2020
Year: 2020

A mobile robot system consists of anonymous mobile robots, each of which autonomously performs sensing, computation, and movement according to a common algorithm, so that the robots collectively achieve a given task. There are two main models of time and activation of the robots. In the semi-synchronous model (SSYNC), the robots share a common notion of time; at each time unit, a subset of the robots is activated, and each performs all three actions (sensing, computation, and movement) in that time unit. In the asynchronous model (ASYNC), there is no common notion of time, the robots are activated at arbitrary times, and the duration of each action is arbitrary but finite. In this paper, we investigate the problem of synchronizing ASNYC robots with limited sensing range, i.e., limited visibility. We first present a sufficient condition for an ASYNC execution of a common algorithm ${\cal A}$ to have a corresponding SSYNC execution of ${\cal A}$; our condition imposes timing constraints on the activation schedule of the robots and visibility constraints during movement. Then, we prove that this condition is necessary (with probability $1$) under a randomized ASYNC adversary. Finally, we present a synchronization algorithm for luminous ASYNC robots with limited visibility, each equipped with a light that can take a constant number of colors. Our algorithm enables luminous ASYNC robots to simulate any algorithm ${\cal A}$, designed for the (non-luminous) SSYNC robots and satisfying visibility constraints.
"To Target or Not to Target": Identification and Analysis of Abusive Text Using Ensemble of Classifiers
Venue: arXiv cs.CL 2020
Year: 2020

With rising concern around abusive and hateful behavior on social media platforms, we present an ensemble learning method to identify and analyze the linguistic properties of such content. Our stacked ensemble comprises of three machine learning models that capture different aspects of language and provide diverse and coherent insights about inappropriate language. The proposed approach provides comparable results to the existing state-of-the-art on the Twitter Abusive Behavior dataset (Founta et al. 2018) without using any user or network-related information; solely relying on textual properties. We believe that the presented insights and discussion of shortcomings of current approaches will highlight potential directions for future research.
HIPE-MAGIC: A Technology-Aware Synthesis and Mapping Flow for HIghly Parallel Execution of Memristor-Aided LoGIC
Venue: arXiv cs.ET 2020
Year: 2020

Recent efforts for finding novel computing paradigms that meet today's design requirements have given rise to a new trend of processing-in-memory relying on non-volatile memories. In this paper, we present HIPE-MAGIC, a technology-aware synthesis and mapping flow for highly parallel execution of the memristor-based logic. Our framework is built upon two fundamental contributions: balancing techniques during the logic synthesis, mainly targeting benefits of the parallelism offered by memristive crossbar arrays (MCAs), and an efficient technology mapping framework to maximize the performance and area-efficiency of the memristor-based logic. Our experimental evaluations across several benchmark suites demonstrate the superior performance of HIPE-MAGIC in terms of throughput and energy efficiency compared to recently developed synthesis and mapping flows targeting MCAs, as well as the conventional CPU computing.
PLANS: Robust Program Learning from Neurally Inferred Specifications
Venue: arXiv cs.LG 2020
Year: 2020

Recent years have seen the rise of statistical program learning based on neural models as an alternative to traditional rule-based systems for programming by example. Rule-based approaches offer correctness guarantees in an unsupervised way as they inherently capture logical rules, while neural models are more realistically scalable to raw, high-dimensional input, and provide resistance to noisy I/O specifications. We introduce PLANS (Program LeArning from Neurally inferred Specifications), a hybrid model for program synthesis from visual observations that gets the best of both worlds, relying on (i) a neural architecture trained to extract abstract, high-level information from each raw individual input (ii) a rule-based system using the extracted information as I/O specifications to synthesize a program capturing the different observations. In order to address the key challenge of making PLANS resistant to noise in the network's output, we introduce a filtering heuristic for I/O specifications based on selective classification techniques. We obtain state-of-the-art performance at program synthesis from diverse demonstration videos in the Karel and ViZDoom environments, while requiring no ground-truth program for training. We make our implementation available at github.com/rdang-nhu/PLANS.
Multi-modal Feature Fusion with Feature Attention for VATEX Captioning Challenge 2020
Venue: arXiv cs.CV 2020
Year: 2020

This report describes our model for VATEX Captioning Challenge 2020. First, to gather information from multiple domains, we extract motion, appearance, semantic and audio features. Then we design a feature attention module to attend on different feature when decoding. We apply two types of decoders, top-down and X-LAN and ensemble these models to get the final result. The proposed method outperforms official baseline with a significant gap. We achieve 76.0 CIDEr and 50.0 CIDEr on English and Chinese private test set. We rank 2nd on both English and Chinese private test leaderboard.
Gandhipedia: A one-stop AI-enabled portal for browsing Gandhian literature, life-events and his social network
Venue: arXiv cs.IR 2020
Year: 2020

We introduce an AI-enabled portal that presents an excellent visualization of Mahatma Gandhi's life events by constructing temporal and spatial social networks from the Gandhian literature. Applying an ensemble of methods drawn from NLTK, Polyglot and Spacy we extract the key persons and places that find mentions in Gandhi's written works. We visualize these entities and connections between them based on co-mentions within the same time frame as networks in an interactive web portal. The nodes in the network, when clicked, fire search queries about the entity and all the information about the entity presented in the corresponding book from which the network is constructed, are retrieved and presented back on the portal. Overall, this system can be used as a digital and user-friendly resource to study Gandhian literature.
Securing IoT Applications using Blockchain: A Survey
Venue: arXiv cs.CR 2020
Year: 2020

The Internet of Things (IoT) has become a guiding technology behind automation and smart computing. One of the major concerns with the IoT systems is the lack of privacy and security preserving schemes for controlling access and ensuring the security of the data. A majority of security issues arise because of the centralized architecture of IoT systems. Another concern is the lack of proper authentication and access control schemes to moderate access to information generated by the IoT devices. So the question that arises is how to ensure the identity of the equipment or the communicating node. The answer to secure operations in a trustless environment brings us to the decentralized solution of Blockchain. A lot of research has been going on in the area of convergence of IoT and Blockchain, and it has resulted in some remarkable progress in addressing some of the significant issues in the IoT arena. This work reviews the challenges and threats in the IoT environment and how integration with Blockchain can resolve some of them.
ELITR Non-Native Speech Translation at IWSLT 2020
Venue: arXiv cs.CL 2020
Year: 2020

This paper is an ELITR system submission for the non-native speech translation task at IWSLT 2020. We describe systems for offline ASR, real-time ASR, and our cascaded approach to offline SLT and real-time SLT. We select our primary candidates from a pool of pre-existing systems, develop a new end-to-end general ASR system, and a hybrid ASR trained on non-native speech. The provided small validation set prevents us from carrying out a complex validation, but we submit all the unselected candidates for contrastive evaluation on the test set.
Structure preserving deep learning
Venue: arXiv cs.LG 2020
Year: 2020

Over the past few years, deep learning has risen to the foreground as a topic of massive interest, mainly as a result of successes obtained in solving large-scale image processing tasks. There are multiple challenging mathematical problems involved in applying deep learning: most deep learning methods require the solution of hard optimisation problems, and a good understanding of the tradeoff between computational effort, amount of data and model complexity is required to successfully design a deep learning approach for a given problem. A large amount of progress made in deep learning has been based on heuristic explorations, but there is a growing effort to mathematically understand the structure in existing deep learning methods and to systematically design new deep learning methods to preserve certain types of structure in deep learning. In this article, we review a number of these directions: some deep neural networks can be understood as discretisations of dynamical systems, neural networks can be designed to have desirable properties such as invertibility or group equivariance, and new algorithmic frameworks based on conformal Hamiltonian systems and Riemannian manifolds to solve the optimisation problems have been proposed. We conclude our review of each of these topics by discussing some open problems that we consider to be interesting directions for future research.
The Baggage Belt Assignment Problem
Venue: arXiv cs.DS 2020
Year: 2020

We consider the problem of assigning flights to baggage belts in the baggage reclaim area of an airport. The problem is originated by a real-life application in Copenhagen airport. The objective is to construct a robust schedule taking passenger and airline preferences into account. We consider a number of business and fairness constraints, avoiding congestions, and ensuring a good passenger flow. Robustness of the solutions is achieved by matching the delivery time with the expected arrival time of passengers, and by adding buffer time between two flights scheduled on the same belt. We denote this problem as the Baggage Belt Assignment Problem (BBAP). We first derive a general Integer Linear Programming (ILP) formulation for the problem. Then, we propose a Branch-and-Price (B&P) algorithm based on a reformulation of the ILP model tackled by Column Generation. Our approach relies on an effective dynamic programming algorithm for handling the pricing problems. We tested the proposed algorithm on a set of real-life data from Copenhagen airport as well as on a set of instances inspired by the real data. Our B&P scheme outperforms a commercial solver launched on the ILP formulation of the problem and is effective in delivering high quality solutions in limited computational times, making it possible its use in daily operations in medium-sized and large airports.