OpenRaiser/Intern-Atlas huggingface dataset:
Few Shot Learning for Information Verification
Venue: arXiv cs.CL 2021
Year: 2021

Information verification is quite a challenging task, this is because many times verifying a claim can require picking pieces of information from multiple pieces of evidence which can have a hierarchy of complex semantic relations. Previously a lot of researchers have mainly focused on simply concatenating multiple evidence sentences to accept or reject claims. These approaches are limited as evidence can contain hierarchical information and dependencies. In this research, we aim to verify facts based on evidence selected from a list of articles taken from Wikipedia. Pretrained language models such as XLNET are used to generate meaningful representations and graph-based attention and convolutions are used in such a way that the system requires little additional training to learn to verify facts.
Towards Personalised and Document-level Machine Translation of Dialogue
Venue: arXiv cs.CL 2021
Year: 2021

State-of-the-art (SOTA) neural machine translation (NMT) systems translate texts at sentence level, ignoring context: intra-textual information, like the previous sentence, and extra-textual information, like the gender of the speaker. Because of that, some sentences are translated incorrectly. Personalised NMT (PersNMT) and document-level NMT (DocNMT) incorporate this information into the translation process. Both fields are relatively new and previous work within them is limited. Moreover, there are no readily available robust evaluation metrics for them, which makes it difficult to develop better systems, as well as track global progress and compare different methods. This thesis proposal focuses on PersNMT and DocNMT for the domain of dialogue extracted from TV subtitles in five languages: English, Brazilian Portuguese, German, French and Polish. Three main challenges are addressed: (1) incorporating extra-textual information directly into NMT systems; (2) improving the machine translation of cohesion devices; (3) reliable evaluation for PersNMT and DocNMT.
Software Architecture for Next-Generation AI Planning Systems
Venue: arXiv cs.AI 2021
Year: 2021

Artificial Intelligence (AI) planning is a flourishing research and development discipline that provides powerful tools for searching a course of action that achieves some user goal. While these planning tools show excellent performance on benchmark planning problems, they represent challenging software systems when it comes to their use and integration in real-world applications. In fact, even in-depth understanding of their internal mechanisms does not guarantee that one can successfully set up, use and manipulate existing planning tools. We contribute toward alleviating this situation by proposing a service-oriented planning architecture to be at the core of the ability to design, develop and use next-generation AI planning systems. We collect and classify common planning capabilities to form the building blocks of the planning architecture. We incorporate software design principles and patterns into the architecture to allow for usability, interoperability and reusability of the planning capabilities. Our prototype planning system demonstrates the potential of our approach for rapid prototyping and flexibility of system composition. Finally, we provide insight into the qualitative advantages of our approach when compared to a typical planning tool.
Trust Computational Heuristic for Social Internet of Things: A Machine Learning-based Approach
Venue: arXiv cs.CR 2021
Year: 2021

The Internet of Things (IoT) is an evolving network of billions of interconnected physical objects, such as numerous sensors, smartphones, wearables, and embedded devices. These physical objects, generally referred to as the smart objects, when deployed in the real-world aggregates useful information from their surrounding environment. As-of-late, this notion of IoT has been extended to incorporate the social networking facets which have led to the promising paradigm of the `Social Internet of Things' (SIoT). In SIoT, the devices operate as an autonomous agent and provide an exchange of information and service discovery in an intelligent manner by establishing social relationships among them with respect to their owners. Trust plays an important role in establishing trustworthy relationships among the physical objects and reduces probable risks in the decision-making process. In this paper, a trust computational model is proposed to extract individual trust features in a SIoT environment. Furthermore, a machine learning-based heuristic is used to aggregate all the trust features in order to ascertain an aggregate trust score. Simulation results illustrate that the proposed trust-based model isolates the trustworthy and untrustworthy nodes within the network in an efficient manner.
Performance of Automatic De-identification Across Different Note Types
Venue: arXiv cs.CL 2021
Year: 2021

Free-text clinical notes detail all aspects of patient care and have great potential to facilitate quality improvement and assurance initiatives as well as advance clinical research. However, concerns about patient privacy and confidentiality limit the use of clinical notes for research. As a result, the information documented in these notes remains unavailable for most researchers. De-identification (de-id), i.e., locating and removing personally identifying protected health information (PHI), is one way of improving access to clinical narratives. However, there are limited off-the-shelf de-identification systems able to consistently detect PHI across different data sources and medical specialties. In this abstract, we present the performance of a state-of-the art de-id system called NeuroNER1 on a diverse set of notes from University of Washington (UW) when the models are trained on data from an external institution (Partners Healthcare) vs. from the same institution (UW). We present results at the level of PHI and note types.
Introducing the Hidden Neural Markov Chain framework
Venue: arXiv cs.CL 2021
Year: 2021

Nowadays, neural network models achieve state-of-the-art results in many areas as computer vision or speech processing. For sequential data, especially for Natural Language Processing (NLP) tasks, Recurrent Neural Networks (RNNs) and their extensions, the Long Short Term Memory (LSTM) network and the Gated Recurrent Unit (GRU), are among the most used models, having a "term-to-term" sequence processing. However, if many works create extensions and improvements of the RNN, few have focused on developing other ways for sequential data processing with neural networks in a "term-to-term" way. This paper proposes the original Hidden Neural Markov Chain (HNMC) framework, a new family of sequential neural models. They are not based on the RNN but on the Hidden Markov Model (HMM), a probabilistic graphical model. This neural extension is possible thanks to the recent Entropic Forward-Backward algorithm for HMM restoration. We propose three different models: the classic HNMC, the HNMC2, and the HNMC-CN. After describing our models' whole construction, we compare them with classic RNN and Bidirectional RNN (BiRNN) models for some sequence labeling tasks: Chunking, Part-Of-Speech Tagging, and Named Entity Recognition. For every experiment, whatever the architecture or the embedding method used, one of our proposed models has the best results. It shows this new neural sequential framework's potential, which can open the way to new models, and might eventually compete with the prevalent BiLSTM and BiGRU.
Semantic Parsing to Manipulate Relational Database For a Management System
Venue: arXiv cs.CL 2021
Year: 2021

Chatbots and AI assistants have claimed their importance in today life. The main reason behind adopting this technology is to connect with the user, understand their requirements, and fulfill them. This has been achieved but at the cost of heavy training data and complex learning models. This work is carried out proposes a simple algorithm, a model which can be implemented in different fields each with its own work scope. The proposed model converts human language text to computer-understandable SQL queries. The model requires data only related to the specific field, saving data space. This model performs linear computation hence solving the computational complexity. This work also defines the stages where a new methodology is implemented and what previous method was adopted to fulfill the requirement at that stage. Two datasets available online will be used in this work, the ATIS dataset, and WikiSQL. This work compares the computation time among the 2 datasets and also compares the accuracy of both. This paper works over basic Natural language processing tasks like semantic parsing, NER, parts of speech and tends to achieve results through these simple methods.
Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization
Venue: arXiv cs.LG 2021
Year: 2021

Action-constrained reinforcement learning (RL) is a widely-used approach in various real-world applications, such as scheduling in networked systems with resource constraints and control of a robot with kinematic constraints. While the existing projection-based approaches ensure zero constraint violation, they could suffer from the zero-gradient problem due to the tight coupling of the policy gradient and the projection, which results in sample-inefficient training and slow convergence. To tackle this issue, we propose a learning algorithm that decouples the action constraints from the policy parameter update by leveraging state-wise Frank-Wolfe and a regression-based policy update scheme. Moreover, we show that the proposed algorithm enjoys convergence and policy improvement properties in the tabular case as well as generalizes the popular DDPG algorithm for action-constrained RL in the general case. Through experiments, we demonstrate that the proposed algorithm significantly outperforms the benchmark methods on a variety of control tasks.
Hierarchical Graph Representations in Digital Pathology
Venue: arXiv cs.CV 2021
Year: 2021

Cancer diagnosis, prognosis, and therapy response predictions from tissue specimens highly depend on the phenotype and topological distribution of constituting histological entities. Thus, adequate tissue representations for encoding histological entities is imperative for computer aided cancer patient care. To this end, several approaches have leveraged cell-graphs that encode cell morphology and organization to denote the tissue information. These allow for utilizing machine learning to map tissue representations to tissue functionality to help quantify their relationship. Though cellular information is crucial, it is incomplete alone to comprehensively characterize complex tissue structure. We herein treat the tissue as a hierarchical composition of multiple types of histological entities from fine to coarse level, capturing multivariate tissue information at multiple levels. We propose a novel multi-level hierarchical entity-graph representation of tissue specimens to model hierarchical compositions that encode histological entities as well as their intra- and inter-entity level interactions. Subsequently, a graph neural network is proposed to operate on the hierarchical entity-graph representation to map the tissue structure to tissue functionality. Specifically, for input histology images we utilize well-defined cells and tissue regions to build HierArchical Cell-to-Tissue (HACT) graph representations, and devise HACT-Net, a graph neural network, to classify such HACT representations. As part of this work, we introduce the BReAst Carcinoma Subtyping (BRACS) dataset, a large cohort of H&E stained breast tumor images, to evaluate our proposed methodology against pathologists and state-of-the-art approaches. Through comparative assessment and ablation studies, our method is demonstrated to yield superior classification results compared to alternative methods as well as pathologists.
Polymorphic Automorphisms and the Picard Group
Venue: arXiv cs.LO 2021
Year: 2021

We investigate the concept of definable, or inner, automorphism in the logical setting of partial Horn theories. The central technical result extends a syntactical characterization of the group of such automorphisms (called the covariant isotropy group) associated with an algebraic theory to the wider class of quasi-equational theories. We apply this characterization to prove that the isotropy group of a strict monoidal category is precisely its Picard group of invertible objects. Furthermore, we obtain an explicit description of the covariant isotropy group of a presheaf category.
Point Cloud Subsampling Parallelization for Unified Memory Platforms
Venue: arXiv cs.RO 2021
Year: 2021

The exploration of unknown environments using robots is a task that integrates different areas such as location, mapping, and planning. For mapping, there are numerous methods to represent the environments through which a robot can travel, in two and three dimensions. The probabilistic occupation grid, Octomap, and STVL can be mentioned among the most important in recent years. Nowadays, RGB-D cameras are widely used to generate a detailed representation of the environment. RGB-D camera measurements present a large volume of data, which must be reduced in order to be used in platforms with limited computing resources. This work presents an implementation of the point cloud decimation method capable of being executed on platforms with unified memory. It consists of reducing the point cloud iteratively using a subdivision of space. Results were obtained for different sizes of grids, platforms, and scenarios, both real and simulated. The results indicate that in embedded systems it is convenient to have architectures that share memory between CPU and GPU to optimize data block communication processes.
Position Information in Transformers: An Overview
Venue: arXiv cs.CL 2021
Year: 2021

Transformers are arguably the main workhorse in recent Natural Language Processing research. By definition a Transformer is invariant with respect to reordering of the input. However, language is inherently sequential and word order is essential to the semantics and syntax of an utterance. In this article, we provide an overview and theoretical comparison of existing methods to incorporate position information into Transformer models. The objectives of this survey are to (1) showcase that position information in Transformer is a vibrant and extensive research area; (2) enable the reader to compare existing methods by providing a unified notation and systematization of different approaches along important model dimensions; (3) indicate what characteristics of an application should be taken into account when selecting a position encoding; (4) provide stimuli for future research.
Towards Causal Representation Learning
Venue: arXiv cs.LG 2021
Year: 2021

The two fields of machine learning and graphical causality arose and developed separately. However, there is now cross-pollination and increasing interest in both fields to benefit from the advances of the other. In the present paper, we review fundamental concepts of causal inference and relate them to crucial open problems of machine learning, including transfer and generalization, thereby assaying how causality can contribute to modern machine learning research. This also applies in the opposite direction: we note that most work in causality starts from the premise that the causal variables are given. A central problem for AI and causality is, thus, causal representation learning, the discovery of high-level causal variables from low-level observations. Finally, we delineate some implications of causality for machine learning and propose key research areas at the intersection of both communities.
Wider Vision: Enriching Convolutional Neural Networks via Alignment to External Knowledge Bases
Venue: arXiv cs.CV 2021
Year: 2021

Deep learning models suffer from opaqueness. For Convolutional Neural Networks (CNNs), current research strategies for explaining models focus on the target classes within the associated training dataset. As a result, the understanding of hidden feature map activations is limited by the discriminative knowledge gleaned during training. The aim of our work is to explain and expand CNNs models via the mirroring or alignment of CNN to an external knowledge base. This will allow us to give a semantic context or label for each visual feature. We can match CNN feature activations to nodes in our external knowledge base. This supports knowledge-based interpretation of the features associated with model decisions. To demonstrate our approach, we build two separate graphs. We use an entity alignment method to align the feature nodes in a CNN with the nodes in a ConceptNet based knowledge graph. We then measure the proximity of CNN graph nodes to semantically meaningful knowledge base nodes. Our results show that in the aligned embedding space, nodes from the knowledge graph are close to the CNN feature nodes that have similar meanings, indicating that nodes from an external knowledge base can act as explanatory semantic references for features in the model. We analyse a variety of graph building methods in order to improve the results from our embedding space. We further demonstrate that by using hierarchical relationships from our external knowledge base, we can locate new unseen classes outside the CNN training set in our embeddings space, based on visual feature activations. This suggests that we can adapt our approach to identify unseen classes based on CNN feature activations. Our demonstrated approach of aligning a CNN with an external knowledge base paves the way to reason about and beyond the trained model, with future adaptations to explainable models and zero-shot learning.
Domain Adaptation in Dialogue Systems using Transfer and Meta-Learning
Venue: arXiv cs.CL 2021
Year: 2021

Current generative-based dialogue systems are data-hungry and fail to adapt to new unseen domains when only a small amount of target data is available. Additionally, in real-world applications, most domains are underrepresented, so there is a need to create a system capable of generalizing to these domains using minimal data. In this paper, we propose a method that adapts to unseen domains by combining both transfer and meta-learning (DATML). DATML improves the previous state-of-the-art dialogue model, DiKTNet, by introducing a different learning technique: meta-learning. We use Reptile, a first-order optimization-based meta-learning algorithm as our improved training method. We evaluated our model on the MultiWOZ dataset and outperformed DiKTNet in both BLEU and Entity F1 scores when the same amount of data is available.
Phase Space Reconstruction Network for Lane Intrusion Action Recognition
Venue: arXiv cs.CV 2021
Year: 2021

In a complex road traffic scene, illegal lane intrusion of pedestrians or cyclists constitutes one of the main safety challenges in autonomous driving application. In this paper, we propose a novel object-level phase space reconstruction network (PSRNet) for motion time series classification, aiming to recognize lane intrusion actions that occur 150m ahead through a monocular camera fixed on moving vehicle. In the PSRNet, the movement of pedestrians and cyclists, specifically viewed as an observable object-level dynamic process, can be reconstructed as trajectories of state vectors in a latent phase space and further characterized by a learnable Lyapunov exponent-like classifier that indicates discrimination in terms of average exponential divergence of state trajectories. Additionally, in order to first transform video inputs into one-dimensional motion time series of each object, a lane width normalization based on visual object tracking-by-detection is presented. Extensive experiments are conducted on the THU-IntrudBehavior dataset collected from real urban roads. The results show that our PSRNet could reach the best accuracy of 98.0%, which remarkably exceeds existing action recognition approaches by more than 30%.
Deep Learning for THz Drones with Flying Intelligent Surfaces: Beam and Handoff Prediction
Venue: arXiv cs.IT 2021
Year: 2021

We consider the problem of proactive handoff and beam selection in Terahertz (THz) drone communication networks assisted with reconfigurable intelligent surfaces (RIS). Drones have emerged as critical assets for next-generation wireless networks to provide seamless connectivity and extend the coverage, and can largely benefit from operating in the THz band to achieve high data rates (such as considered for 6G). However, THz communications are highly susceptible to channel impairments and blockage effects that become extra challenging when accounting for drone mobility. RISs offer flexibility to extend coverage by adapting to channel dynamics. To integrate RISs into THz drone communications, we propose a novel deep learning solution based on a recurrent neural network, namely the Gated Recurrent Unit (GRU), that proactively predicts the serving base station/RIS and the serving beam for each drone based on the prior observations of drone location/beam trajectories. This solution has the potential to extend the coverage of drones and enhance the reliability of next-generation wireless communications. Predicting future beams based on the drone beam/position trajectory significantly reduces the beam training overhead and its associated latency, and thus emerges as a viable solution to serve time-critical applications. Numerical results based on realistic 3D ray-tracing simulations show that the proposed deep learning solution is promising for future RIS-assisted THz networks by achieving near-optimal proactive hand-off performance and more than 90% accuracy for beam prediction.
Modeling Multi-Destination Trips with Sketch-Based Model
Venue: arXiv cs.LG 2021
Year: 2021

The recently proposed EMDE (Efficient Manifold Density Estimator) model achieves state of-the-art results in session-based recommendation. In this work we explore its application to Booking Data Challenge competition. The aim of the challenge is to make the best recommendation for the next destination of a user trip, based on dataset with millions of real anonymized accommodation reservations. We achieve 2nd place in this competition. First, we use Cleora - our graph embedding method - to represent cities as a directed graph and learn their vector representation. Next, we apply EMDE to predict the next user destination based on previously visited cities and some features associated with each trip. We release the source code at: https://github.com/Synerise/booking-challenge.
You Only Compress Once: Optimal Data Compression for Estimating Linear Models
Venue: arXiv cs.LG 2021
Year: 2021

Linear models are used in online decision making, such as in machine learning, policy algorithms, and experimentation platforms. Many engineering systems that use linear models achieve computational efficiency through distributed systems and expert configuration. While there are strengths to this approach, it is still difficult to have an environment that enables researchers to interactively iterate and explore data and models, as well as leverage analytics solutions from the open source community. Consequently, innovation can be blocked. Conditionally sufficient statistics is a unified data compression and estimation strategy that is useful for the model development process, as well as the engineering deployment process. The strategy estimates linear models from compressed data without loss on the estimated parameters and their covariances, even when errors are autocorrelated within clusters of observations. Additionally, the compression preserves almost all interactions with the the original data, unlocking better productivity for both researchers and engineering systems.