Towards Multiple Tasks Integration with Reinforcement Learning

Historically, Natural Language Processing (NLP) systems have generally been built as sequential pipelines, where each module adds another layer of annotation, in order of (supposed) increasing complexity. For example, the system of [4], winner of the CoNLL-2005 Shared Task on Semantic Role Labeling (SRL) [2], starts with a syntactic analysis, followed by a pruning stage that collects argument candidates, an argument identification stage, an argument classification one and an inference stage that takes the final labelling decisions based on the classification probabilities output by the previous stage as well as a set of linguistic constraints. With the progress of neural networks, however, we have seen the apparition of state-of-the-art systems that completely bypass intermediate levels of analysis that were previously considered essential. For example, the systems of [16] and [6] performs respectively SRL and coreference resolution, both without referring to any explicit (morpho)syntactic information. A well known problem of sequential systems is error propagation. Error propagation happens when an incorrect action makes a subsequent correct action unavailable, because the two are logically inconsistent, or when the action leads to a state that is unusual (unseen during the training process) with the effect of disrupting the system so that a subsequent correct action fails to be predicted even though it is logically consistent [7]. Additionally, traditional pipelines are conceptually limited by the fact that the flow of annotation between their different stages is unidirectional. For example, a system could perform Word Sense Disambiguation (WSD) before semantic parsing and in that case WSD classification can be used during semantic parsing, but not semantic parsing decisions for WSD. Alternatively, the system can preform semantic parsing first, and in that case it is the other way around. However, it is not clear that one ordering is the “right” one, optimal for all inputs. Intuitively, one might prefer do what is easy first, considering both tasks, in order to maximise the information available when taking the hardest decisions. This kind of easy-first strategy has indeed been applied to various NLP problems, but only in single-task contexts [10, 3, 11, 15]. End-to-end systems with no intermediate level of analysis cannot, by definition, suffer from the first type of error propagation. However, by reducing the information in the training signal at their disposal, they might not leverage the full power of the neural networks they are usually based on. Indeed, by reintroducing syntactic dependencies in the training process, [12] managed to develop a state-of-the-art system for SRL that at the same time computes a good quality syntactic parse. The solution they implement, however, is very specific to the combination of tasks they consider. Furthermore, the interaction between the two types of information can only happen during the unique pass inside a neural network, where SRL and syntactic dependencies are only present under distributional forms, before they are independently discretised into symbolic structures. In this work — currently under progress —, we propose to build a system that performs part-ofspeech (POS) tagging, syntactic dependency parsing and semantic dependency parsing (SDP) in a nonsequential way. At each time step, the system can take actions related to any of these three annotation layers, with no order constraint. Each of these layers of annotation are fed as input for the next time steps, so they can fully interact with each other. Our model is inspired by the semantic dependency parser of [5], which was shown to develop an easy-first strategy (short semantic dependencies before longer ones) thanks to a Reinforcement Learning (RL) procedure. The most obvious difference between their work and ours is that we fully integrate different tasks into a unique system. This naturally leads to differences in architecture. The RL algorithm that we use is also slightly different from theirs.