HPSG/MRS-Based Sentence Generation with Transformer

In recent years, Natural Language Processing (NLP) has achieved significant improvements thanks to the introduction of neural networks. For instance, BERT (Devlin et al., 2018) achieved state-of-the-art performance in many tasks. However, BERT is not linguistically motivated but developed with a concept of Cloze Task (Taylor, 1953). Some people claim that natural language texts can be processed and generated successfully only with the text information by the success of NLP systems without linguistic motivations. For instance, a deep learning textbook introduces NLP as a field that does not require feature engineering (Subramanian, 2018, p.75). In doing so, it has often been postulated that the raw texts are enough to build up the NLP systems with. Meanwhile, Konstas et al. (2017) redefined natural language generation (NLG) as a machine translation (MT) task by training a model to parse to and generate from Abstract Meaning Representation (AMR). In other words, they built an MT model that translates AMR to natural language text using a sequence-to-sequence model with bidirectional LSTMs and Attention mechanisms. While AMR formalism abstracts away the surface form to represent human language sentences as a directed graph, other formalisms contain richer information. Minimal Recursion Semantics (MRS) formalism (Copestake et al., 2005), for instance, contains rich information about the surface form, including number, tense, and case information (Konstas et al., 2017). Using this richer representation, Hajdik et al. (2019) reproduced the work of Konstas et al. (2017) and showed significant improvement in translation when measured by the BLEU metric (Papineni et al., 2002). Konstas et al. (2017) and Hajdik et al. (2019) have their own significance in that they demonstrate using the linguistically motivated features works for a neural networkbased NLP task. The fact the linguistically richer representation significantly improved the performance also hints that linguistic features are still relevant when applied in a proper format. In other words, a proper use of grammars like Head-Driven Phrase Structure Grammar (HPSG), as in Hajdik et al. (2019), can improve NLP even in this age of neural networks. However, both studies used a bi-LSTM based sequenceto-sequence model for the MT task. RNNs “learn” from sequential data, such as natural language text divided by token, through the back-propagation of the error to the connected cells. In this process, the gradient information is used to minimize the loss of the model. As this gradient is acquired by differentiation, the gradient of the old cells vanishes, which impedes the RNNs from processing longer sentences. Later algorithms such as LSTM and GRU tried to bypass this problem by strategically “forgetting” some information. Meanwhile, Attention Mechanism helps a sequence-to-sequence model to process longer sentences by indicating which cells to “pay attention to”. Built solely upon this mechanism, Transformer performed better at MT tasks. Since MRS representation, linearized as in Hajdik et al. (2019), tends to be long, it is expected that applying Transformer would further improve the performance. In short, the current research aims to reproduce the work of Hajdik et al. (2019) using Transformer to substantiate that computational grammar inspired by rich syntactic and semantic formalism does improve neural NLG. Along the line of the previous studies, the present study draws more attention to the use of grammar-based representation. In so doing, the present study aims to demonstrate that Transformer-based NLG fits into the MRS representation.