Solving Three Czech NLP Tasks with End-to-end Neural Models

In this work, we focus on three different NLP tasks: image captioning, machine translation, and sentiment analysis. We reimplement successful approaches of other authors and adapt them to the Czech language. We provide end-to-end architectures that achieve state-ofthe-art results on all of the tasks within a single sequence learning toolkit. The trained models are available both for download as well as in an online demo. 1 End-to-End Training Traditionally, solving tasks such as machine translation or sentiment analysis required complex processing pipelines consisting of tools which transformed one explicit representation of the data into another, with the structure of the internal representations defined by the system designer. In machine translation, we would devise explicit word alignment links, extract phrase tables, train a language model, etc.; in sentiment analysis, we could label the data with part-of-speech tags, decode their syntactic structure, and/or assign them with semantic labels. All of these more-or-less linguistically motivated internal representations are not inherently required to produce the desired output, but have been devised as clever and useful ways to break down the large and hard task into smaller and manageable substeps. With the advent of end-to-end training of deep neural networks (DNN), the need for most of this has been eliminated. In the end-to-end learning paradigm, there is only one model, directly trained to produce the desired outputs from the inputs, without any explicit intermediate representations. The system designer now only has to design a rather generic architecture of the system. It mostly does not enforce any complex explicit representations and processing steps, but rather offers opportunities for the DNN to devise its own notion of intermediate representations and processing steps through training. This also means that similar architectures can be used to solve very different tasks. Rather than by the nature of the task itself, the structure of the DNN to use is mostly determined by the structure of the input and output – e.g. image inputs are processed by two-dimensional convolutions, while text inputs are processed by one-dimensional convolutions, recurrent units, and/or attentions; classification can produce its output in one step, while text generation is better done iteratively using recurrent decoders; etc. Thanks to that, a single general framework can be used to solve many different tasks. One just needs to transform the inputs and outputs into a suitable format, define an adequate network structure, and let the system train for a few weeks. Sadly, the burden of hyperparameter tuning has not been alleviated by DNNs, but rather made worse by the computational costliness of the training. However, with a bit of experience, one is often able to propose a suitable architecture and hyperparameter values at the first attempt, already achieving very competitive results even without any further tuning. 1.1 Our contribution Most of the papers in the field only evaluate their setups on English datasets. In our work, we try to rectify this shortcoming by reimplementing existing state-of-theart approaches in the Neural Monkey framework [14] and training them on existing Czech datasets. Neural Monkey is an open-source toolkit for sequenceto-sequence learning, implemented in the TensorFlow library [1]. The toolkit is designed to be easily extensible in order to support fast prototyping of architectures for various NLP tasks. It is freely available on GitHub1 under the BSD license, allowing both non-commercial and commercial use of the toolkit. We decided to focus on three rather varied tasks – sentiment analysis, machine translation, and image captioning. For each of the tasks, we reimplemented one or more existing state-of-the-art architectures within Neural Monkey and trained it on available datasets. Our evaluations show that we manage to reach or surpass state-of-the-art results for all the three tasks. As we wish to encourage other NLP researchers to focus on Czech language, we make sure that our source codes, our configuration files and our trained models are all freely available to anyone interested to use them, to study them and to build upon them. With our work, we hope to establish well-performing approaches for Czech NLP, as well as to allow e.g. investigation of the internals of the trained models to try to decipher in what ways language seems to be implicitly captured in them. Moreover, we have created a simple web-based demo that allows anyone to easily apply our models to any input data, intended to popularize deep learning and its applications for the Czech audience. 1https://github.com/ufal/neuralmonkey S. Krajči (ed.): ITAT 2018 Proceedings, pp. 138–143 CEUR Workshop Proceedings Vol. 2203, ISSN 1613-0073, c © 2018 Jindřich Libovický, Rudolf Rosa, Jindřich Helcl, and Martin Popel