OpenRaiser/Intern-Atlas huggingface dataset:
Learning Cross-Context Entity Representations from Text
Venue: arXiv cs.CL 2020
Year: 2020

Language modeling tasks, in which words, or word-pieces, are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases or human readable encyclopedias tend to be entity-centric, we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the text contexts in which those entities were mentioned. We show that large scale training of neural models allows us to learn high quality entity representations, and we demonstrate successful results on four domains: (1) existing entity-level typing benchmarks, including a 64% error reduction over previous work on TypeNet (Murty et al., 2018); (2) a novel few-shot category reconstruction task; (3) existing entity linking benchmarks, where we match the state-of-the-art on CoNLL-Aida without linking-specific features and obtain a score of 89.8% on TAC-KBP 2010 without using any alias table, external knowledge base or in domain training data and (4) answering trivia questions, which uniquely identify entities. Our global entity representations encode fine-grained type categories, such as Scottish footballers, and can answer trivia questions such as: Who was the last inmate of Spandau jail in Berlin?
Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
Venue: arXiv cs.LG 2020
Year: 2020

How can we enable machines to make sense of the world, and become better at learning? To approach this goal, I believe viewing intelligence in terms of many integral aspects, and also a universal two-term tradeoff between task performance and complexity, provides two feasible perspectives. In this thesis, I address several key questions in some aspects of intelligence, and study the phase transitions in the two-term tradeoff, using strategies and tools from physics and information. Firstly, how can we make the learning models more flexible and efficient, so that agents can learn quickly with fewer examples? Inspired by how physicists model the world, we introduce a paradigm and an AI Physicist agent for simultaneously learning many small specialized models (theories) and the domain they are accurate, which can then be simplified, unified and stored, facilitating few-shot learning in a continual way. Secondly, for representation learning, when can we learn a good representation, and how does learning depend on the structure of the dataset? We approach this question by studying phase transitions when tuning the tradeoff hyperparameter. In the information bottleneck, we theoretically show that these phase transitions are predictable and reveal structure in the relationships between the data, the model, the learned representation and the loss landscape. Thirdly, how can agents discover causality from observations? We address part of this question by introducing an algorithm that combines prediction and minimizing information from the input, for exploratory causal discovery from observational time series. Fourthly, to make models more robust to label noise, we introduce Rank Pruning, a robust algorithm for classification with noisy labels. I believe that building on the work of my thesis we will be one step closer to enable more intelligent machines that can make sense of the world.
Decentralized sequential active hypothesis testing and the MAC feedback capacity
Venue: arXiv cs.IT 2020
Year: 2020

We consider the problem of decentralized sequential active hypothesis testing (DSAHT), where two transmitting agents, each possessing a private message, are actively helping a third agent--and each other--to learn the message pair over a discrete memoryless multiple access channel (DM-MAC). The third agent (receiver) observes the noisy channel output, which is also available to the transmitting agents via noiseless feedback. We formulate this problem as a decentralized dynamic team, show that optimal transmission policies have a time-invariant domain, and characterize the solution through a dynamic program. Several alternative formulations are discussed involving time-homogenous cost functions and/or variable-length codes, resulting in solutions described through fixed-point, Bellman-type equations. Subsequently, we make connections with the problem of simplifying the multi-letter capacity expressions for the noiseless feedback capacity of the DM-MAC. We show that restricting attention to distributions induced by optimal transmission schemes for the DSAHT problem, without loss of optimality, transforms the capacity expression, so that it can be thought of as the average reward received by an appropriately defined stochastic dynamical system with time-invariant state space.
Secure Decentralized Pliable Index Coding
Venue: arXiv cs.IT 2020
Year: 2020

This paper studies a variant of the Pliable Index CODing (PICOD) problem, i.e., an index coding problem where a user can be satisfied by decoding any message that is not in its side information set, where communication is decentralized, i.e., it occurs among users rather than by the central server, and secure, i.e., each user is allowed to decode only one message outside its side information set and must not be able to collect any information about any other message that is not its decoded one. Given the difficulty of the general version of this problem, this paper focuses on the case where the side information sets are `$s$~circular shifts', namely, user $u$'s side information set is the set of messages indexed by $\{u, u+1, \ldots, u+s-1\}$ for some fixed $s$ and where the indices are intended modulo the cardinality of the message set. This particular setting has been studied in the `decentralized non-secure' and in the `centralized secure' settings, thus allows one to quantify the cost of decentralized communication under security constraints on the number of transmissions. Interestingly, the decentralized vs the centralized secure setting incurs a multiplicative gap of approximately~three. This is in contrast to the cases without security constraint, where the multiplicative gap is known to be at most two.
Permissioned Blockchain Revisited: A Byzantine Game-Theoretical Perspective
Venue: arXiv cs.GT 2020
Year: 2020

Despite the popularity and practical applicability of blockchains, there is very limited work on the theoretical foundation of blockchains: The lack of rigorous theory and analysis behind the curtain of blockchains has severely staggered its broader applications. This paper attempts to lay out a theoretical foundation for a specific type of blockchains---the ones requiring basic authenticity from the participants, also called \textit{permissioned blockchain}. We formulate permissioned blockchain systems and operations into a game-theoretical problem by incorporating constraints implied by the wisdom from distributed computing and Byzantine systems. We show that in a noncooperative blockchain game (NBG), a Nash equilibrium can be efficiently found in a closed-form even though the game involves more than two players. Somewhat surprisingly, the simulation results of the Nash equilibrium implies that the game can reach a stable status regardless of the number of Byzantine nodes and trustworthy players. We then study a harder problem where players are allowed to form coalitions: the coalitional blockchain game (CBG). We show that although the Shapley value for a CBG can be expressed in a more succinct form, its core is empty.
The Capacity of Private Information Retrieval Under Arbitrary Collusion Patterns
Venue: arXiv cs.IT 2020
Year: 2020

We study the private information retrieval (PIR) problem under arbitrary collusion pattern for replicated databases. We find its capacity, which is the same as the capacity of the original PIR problem with the number of databases $N$ replaced by a number $S^*$. The number $S^*$ is the optimal solution to a linear programming problem that is a function of the collusion pattern. Hence, the collusion pattern affects the capacity of the PIR problem only through the number $S^*$.
Learning to drive via Apprenticeship Learning and Deep Reinforcement Learning
Venue: arXiv cs.RO 2020
Year: 2020

With the implementation of reinforcement learning (RL) algorithms, current state-of-art autonomous vehicle technology have the potential to get closer to full automation. However, most of the applications have been limited to game domains or discrete action space which are far from the real world driving. Moreover, it is very tough to tune the parameters of reward mechanism since the driving styles vary a lot among the different users. For instance, an aggressive driver may prefer driving with high acceleration whereas some conservative drivers prefer a safer driving style. Therefore, we propose an apprenticeship learning in combination with deep reinforcement learning approach that allows the agent to learn the driving and stopping behaviors with continuous actions. We use gradient inverse reinforcement learning (GIRL) algorithm to recover the unknown reward function and employ REINFORCE as well as Deep Deterministic Policy Gradient algorithm (DDPG) to learn the optimal policy. The performance of our method is evaluated in simulation-based scenario and the results demonstrate that the agent performs human like driving and even better in some aspects after training.
Stochastic Natural Language Generation Using Dependency Information
Venue: arXiv cs.CL 2020
Year: 2020

This article presents a stochastic corpus-based model for generating natural language text. Our model first encodes dependency relations from training data through a feature set, then concatenates these features to produce a new dependency tree for a given meaning representation, and finally generates a natural language utterance from the produced dependency tree. We test our model on nine domains from tabular, dialogue act and RDF format. Our model outperforms the corpus-based state-of-the-art methods trained on tabular datasets and also achieves comparable results with neural network-based approaches trained on dialogue act, E2E and WebNLG datasets for BLEU and ERR evaluation metrics. Also, by reporting Human Evaluation results, we show that our model produces high-quality utterances in aspects of informativeness and naturalness as well as quality.
Error Exponents of Mismatched Likelihood Ratio Testing
Venue: arXiv cs.IT 2020
Year: 2020

We study the problem of mismatched likelihood ratio test. We analyze the type-\RNum{1} and \RNum{2} error exponents when the actual distributions generating the observation are different from the distributions used in the test. We derive the worst-case error exponents when the actual distributions generating the data are within a relative entropy ball of the test distributions. In addition, we study the sensitivity of the test for small relative entropy balls.
Attention Flow: End-to-End Joint Attention Estimation
Venue: arXiv cs.CV 2020
Year: 2020

This paper addresses the problem of understanding joint attention in third-person social scene videos. Joint attention is the shared gaze behaviour of two or more individuals on an object or an area of interest and has a wide range of applications such as human-computer interaction, educational assessment, treatment of patients with attention disorders, and many more. Our method, Attention Flow, learns joint attention in an end-to-end fashion by using saliency-augmented attention maps and two novel convolutional attention mechanisms that determine to select relevant features and improve joint attention localization. We compare the effect of saliency maps and attention mechanisms and report quantitative and qualitative results on the detection and localization of joint attention in the VideoCoAtt dataset, which contains complex social scenes.
Heterogeneous Computation Assignments in Coded Elastic Computing
Venue: arXiv cs.IT 2020
Year: 2020

We study the optimal design of a heterogeneous coded elastic computing (CEC) network where machines have varying relative computation speeds. CEC introduced by Yang {\it et al.} is a framework which mitigates the impact of elastic events, where machines join and leave the network. A set of data is distributed among storage constrained machines using a Maximum Distance Separable (MDS) code such that any subset of machines of a specific size can perform the desired computations. This design eliminates the need to re-distribute the data after each elastic event. In this work, we develop a process for an arbitrary heterogeneous computing network to minimize the overall computation time by defining an optimal computation load, or number of computations assigned to each machine. We then present an algorithm to define a specific computation assignment among the machines that makes use of the MDS code and meets the optimal computation load.
Finite-Level Quantization Procedures for Construction and Decoding of Polar Codes
Venue: arXiv cs.IT 2020
Year: 2020

We consider finite-level, symmetric quantization procedures for construction and decoding of polar codes. Whether polarization occurs in the presence of quantization is not known in general. Hassani and Urbanke have shown that a simple three-level quantization procedure polarizes and they have proposed a calculation method to obtain a lower bound for achievable rates. We find an improved calculation method for achievable rates and also the exact asymptotic behavior of the block error probability under the aforementioned simple case. We then prove that certain D-level quantization schemes polarize and give a lower bound on achievable rates. Furthermore, we show that a broad class of quantization procedures result in a weaker form of the polarization phenomenon.
Learning ergodic averages in chaotic systems
Venue: arXiv cs.LG 2020
Year: 2020

We propose a physics-informed machine learning method to predict the time average of a chaotic attractor. The method is based on the hybrid echo state network (hESN). We assume that the system is ergodic, so the time average is equal to the ergodic average. Compared to conventional echo state networks (ESN) (purely data-driven), the hESN uses additional information from an incomplete, or imperfect, physical model. We evaluate the performance of the hESN and compare it to that of an ESN. This approach is demonstrated on a chaotic time-delayed thermoacoustic system, where the inclusion of a physical model significantly improves the accuracy of the prediction, reducing the relative error from 48% to 7%. This improvement is obtained at the low extra cost of solving two ordinary differential equations. This framework shows the potential of using machine learning techniques combined with prior physical knowledge to improve the prediction of time-averaged quantities in chaotic systems.
SERAD: Soft Error Resilient Asynchronous Design using a Bundled Data Protocol
Venue: arXiv cs.AR 2020
Year: 2020

The risk of soft errors due to radiation continues to be a significant challenge for engineers trying to build systems that can handle harsh environments. Building systems that are Radiation Hardened by Design (RHBD) is the preferred approach, but existing techniques are expensive in terms of performance, power, and/or area. This paper introduces a novel soft-error resilient asynchronous bundled-data design template, SERAD, which uses a combination of temporal and spatial redundancy to mitigate Single Event Transients (SETs) and upsets (SEUs). SERAD uses Error Detecting Logic (EDL) to detect SETs at the inputs of sequential elements and correct them via re-sampling. Because SERAD only pays the delay penalty in the presence of an SET, which rarely occurs, its average performance is comparable to the baseline synchronous design. We tested the SERAD design using a combination of Spice and Verilog simulations and evaluated its impact on area, frequency, and power on an open-core MIPS-like processor using a NCSU 45nm cell library. Our post-synthesis results show that the SERAD design consumes less than half of the area of the Triple Modular Redundancy (TMR), exhibits significantly less performance degradation than Glitch Filtering (GF), and consumes no more total power than the baseline unhardened design.
ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
Venue: arXiv cs.CL 2020
Year: 2020

This paper presents a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of optimizing one-step-ahead prediction in the traditional sequence-to-sequence model, the ProphetNet is optimized by n-step ahead prediction that predicts the next n tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large-scale dataset (160GB), respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus.
Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey
Venue: arXiv cs.CV 2020
Year: 2020

From the autonomous car driving to medical diagnosis, the requirement of the task of image segmentation is everywhere. Segmentation of an image is one of the indispensable tasks in computer vision. This task is comparatively complicated than other vision tasks as it needs low-level spatial information. Basically, image segmentation can be of two types: semantic segmentation and instance segmentation. The combined version of these two basic tasks is known as panoptic segmentation. In the recent era, the success of deep convolutional neural networks (CNN) has influenced the field of segmentation greatly and gave us various successful models to date. In this survey, we are going to take a glance at the evolution of both semantic and instance segmentation work based on CNN. We have also specified comparative architectural details of some state-of-the-art models and discuss their training details to present a lucid understanding of hyper-parameter tuning of those models. We have also drawn a comparison among the performance of those models on different datasets. Lastly, we have given a glimpse of some state-of-the-art panoptic segmentation models.
Approximation smooth and sparse functions by deep neural networks without saturation
Venue: arXiv cs.IT 2020
Year: 2020

Constructing neural networks for function approximation is a classical and longstanding topic in approximation theory. In this paper, we aim at constructing deep neural networks (deep nets for short) with three hidden layers to approximate smooth and sparse functions. In particular, we prove that the constructed deep nets can reach the optimal approximation rate in approximating both smooth and sparse functions with controllable magnitude of free parameters. Since the saturation that describes the bottleneck of approximate is an insurmountable problem of constructive neural networks, we also prove that deepening the neural network with only one more hidden layer can avoid the saturation. The obtained results underlie advantages of deep nets and provide theoretical explanations for deep learning.
On the Gap between Scalar and Vector Solutions of Generalized Combination Networks
Venue: arXiv cs.IT 2020
Year: 2020

We study scalar-linear and vector-linear solutions to the generalized combination network. We derive new upper and lower bounds on the maximum number of nodes in the middle layer, depending on the network parameters. These bounds improve and extend the parameter range of known bounds. Using these new bounds we present a general lower bound on the gap in the alphabet size between scalar-linear and vector-linear solutions.
Multi-Sensor Data and Knowledge Fusion -- A Proposal for a Terminology Definition
Venue: arXiv cs.AI 2020
Year: 2020

Fusion is a common tool for the analysis and utilization of available datasets and so an essential part of data mining and machine learning processes. However, a clear definition of the type of fusion is not always provided due to inconsistent literature. In the following, the process of fusion is defined depending on the fusion components and the abstraction level on which the fusion occurs. The focus in the first part of the paper at hand is on the clear definition of the terminology and the development of an appropriate ontology of the fusion components and the fusion level. In the second part, common fusion techniques are presented.
Mining customer product reviews for product development: A summarization process
Venue: arXiv cs.CL 2020
Year: 2020

This research set out to identify and structure from online reviews the words and expressions related to customers' likes and dislikes to guide product development. Previous methods were mainly focused on product features. However, reviewers express their preference not only on product features. In this paper, based on an extensive literature review in design science, the authors propose a summarization model containing multiples aspects of user preference, such as product affordances, emotions, usage conditions. Meanwhile, the linguistic patterns describing these aspects of preference are discovered and drafted as annotation guidelines. A case study demonstrates that with the proposed model and the annotation guidelines, human annotators can structure the online reviews with high inter-agreement. As high inter-agreement human annotation results are essential for automatizing the online review summarization process with the natural language processing, this study provides materials for the future study of automatization.
Comparing the impact of subfields in scientific journals
Venue: arXiv cs.DL 2020
Year: 2020

The impact factor has been extensively used in the last years to assess journals visibility and prestige. While the impact factor is useful to compare journals, the specificities of subfields visibility in journals are overlooked whenever visibility is measured only at the journal level. In this paper, we analyze the subfields visibility in a subset of over 450,000 Physics papers. We show that the visibility of subfields is not regular in the considered dataset. In particular years, the variability in subfields impact factor in a journal reached 75% of the average subfields impact factor. We also found that the difference of subfields visibility in the same journal can be even higher than the difference of visibility between different journals. Our results show that subfields impact is an important factor accounting for journals visibility.
On the linear structure of cones
Venue: arXiv cs.LO 2020
Year: 2020

For encompassing the limitations of probabilistic coherence spaces which do not seem to provide natural interpretations of continuous data types such as the real line, Ehrhard and al. introduced a model of probabilistic higher order computation based on (positive) cones, and a class of totally monotone functions that they called "stable". Then Crubill{é} proved that this model is a conservative extension of the earlier probabilistic coherence space model. We continue these investigations by showing that the category of cones and linear and Scott-continuous functions is a model of intuitionistic linear logic. To define the tensor product, we use the special adjoint functor theorem, and we prove that this operation is and extension of the standard tensor product of probabilistic coherence spaces. We also show that these latter are dense in cones, thus allowing to lift the main properties of the tensor product of probabilistic coherence spaces to general cones. Last we define in the same way an exponential of cones and extend measurability to these new operations.