Introduction
Being one of the prominent natural language generation tasks, neural abstractive text summarization (NATS) has gained a lot of popularity BIBREF0 , BIBREF1 , BIBREF2 . Different from extractive text summarization BIBREF3 , BIBREF4 , BIBREF5 , NATS relies on modern deep learning models, particularly sequence-to-sequence (Seq2Seq) models, to generate words from a vocabulary based on the representations/features of source documents BIBREF0 , BIBREF6 , so that it has the ability to generate high-quality summaries that are verbally innovative and can also easily incorporate external knowledge BIBREF1 . Many NATS models have achieved better performance in terms of the commonly used evaluation measures (such as ROUGE BIBREF7 score) compared to extractive text summarization approaches BIBREF2 , BIBREF8 , BIBREF9 .
We recently provided a comprehensive survey of the Seq2Seq models BIBREF10 , including their network structures, parameter inference methods, and decoding/generation approaches, for the task of abstractive text summarization. A variety of NATS models share many common properties and some of the key techniques are widely used to produce well-formed and human-readable summaries that are inferred from source articles, such as encoder-decoder framework BIBREF11 , word embeddings BIBREF12 , attention mechanism BIBREF13 , pointing mechanism BIBREF14 and beam-search algorithm BIBREF0 . Many of these features have also found applications in other language generation tasks, such as machine translation BIBREF13 and dialog systems BIBREF15 . In addition, other techniques that can also be shared across different tasks include training strategies BIBREF16 , BIBREF17 , BIBREF18 , data pre-processing, results post-processing and model evaluation. Therefore, having an open-source toolbox that modularizes different network components and unifies the learning framework for each training strategy can benefit researchers in language generation from various aspects, including efficiently implementing new models and generalizing existing models to different tasks.
In the past few years, different toolkits have been developed to achieve this goal. Some of them were designed specifically for a single task, such as ParlAI BIBREF19 for dialog research, and some have been further extended to other tasks. For example, OpenNMT BIBREF20 and XNMT BIBREF21 are primarily for neural machine translation (NMT), but have been applied to other areas. The bottom-up attention model BIBREF9 , which has achieved state-of-the-art performance for abstractive text summarization, is implemented in OpenNMT. There are also several other general purpose language generation packages, such as Texar BIBREF22 . Compared with these toolkits, LeafNATS is specifically designed for NATS research, but can also be adapted to other tasks. In this toolkit, we implement an end-to-end training framework that can minimize the effort in writing codes for training/evaluation procedures, so that users can focus on building models and pipelines. This framework also makes it easier for the users to transfer pre-trained parameters of user-specified modules to newly built models.
In addition to the learning framework, we have also developed a web application, which is driven by databases, web services and NATS models, to show a demo of deploying a new NATS idea to a real-life application using LeafNATS. Such an application can help front-end users (e.g., blog/news authors and editors) by providing suggestions of headlines and summaries for their articles.
The rest of this paper is organized as follows: Section SECREF2 introduces the structure and design of LeafNATS learning framework. In Section SECREF3 , we describe the architecture of the live system demo. Based on the request of the system, we propose and implement a new model using LeafNATS for headline and summary generation. We conclude this paper in Section SECREF4 .
LeafNATS Toolkithttps://github.com/tshi04/LeafNATS
In this section, we introduce the structure and design of LeafNATS toolkit, which is built upon the lower level deep learning platform – Pytorch BIBREF23 . As shown in Fig. FIGREF2 , it consists of four main components, i.e., engines, modules, data and tools and playground.
Engines: In LeafNATS, an engine represents a training algorithm. For example, end-to-end training BIBREF1 and adversarial training BIBREF16 are two different training frameworks. Therefore, we need to develop two different engines for them.
Specifically for LeafNATS, we implement a task-independent end-to-end training engine for NATS, but it can also be adapted to other NLP tasks, such as NMT, question-answering, sentiment classification, etc. The engine uses abstract data, models, pipelines, and loss functions to build procedures of training, validation, testing/evaluation and application, respectively, so that they can be completely reused when implementing a new model. For example, these procedures include saving/loading check-point files during training, selecting N-best models during validation, and using the best model for generation during testing, etc. Another feature of this engine is that it allows users to specify part of a neural network to train and reuse parameters from other models, which is convenient for transfer learning.
Modules: Modules are the basic building blocks of different models. In LeafNATS, we provide ready-to-use modules for constructing recurrent neural network (RNN)-based sequence-to-sequence (Seq2Seq) models for NATS, e.g., pointer-generator network BIBREF1 . These modules include embedder, RNN encoder, attention BIBREF24 , temporal attention BIBREF6 , attention on decoder BIBREF2 and others. We also use these basic modules to assemble a pointer-generator decoder module and the corresponding beam search algorithms. The embedder can also be used to realize the embedding-weights sharing mechanism BIBREF2 .
Data and Tools: Different models in LeafNATS are tested on three datasets (see Table TABREF5 ), namely, CNN/Daily Mail (CNN/DM) BIBREF25 , Newsroom BIBREF26 and Bytecup. The pre-processed CNN/DM data is available online. Here, we provide tools to pre-process the last two datasets. Data modules are used to prepare the input data for mini-batch optimization.
Playground: With the engine and modules, we can develop different models by just assembling these modules and building pipelines in playground. We re-implement different models in the NATS toolkit BIBREF10 to this framework. The performance (ROUGE scores BIBREF7 ) of the pointer-generator model on different datasets has been reported in Table TABREF6 , where we find that most of the results are better than our previous implementations BIBREF10 due to some minor changes to the neural network.
A Live System Demonstrationhttp://dmkdt3.cs.vt.edu/leafNATS
In this section, we present a real-world web application of the abstractive text summarization models, which can help front-end users to write headlines and summaries for their articles/posts. We will first discuss the architecture of the system, and then, provide more details of the front-end design and a new model built by LeafNATS that makes automatic summarization and headline generation possible.
Architecture
This is a news/blog website, which allows people to read, duplicate, edit, post, delete and comment articles. It is driven by web-services, databases and our NATS models. This web application is developed with PHP, HTML/CSS, and jQuery following the concept of Model-View-Controller (see Fig. FIGREF9 ).
In this framework, when people interact with the front-end views, they send HTML requests to controllers that can manipulate models. Then, the views will be changed with the updated information. For example, in NATS, we first write an article in a text-area. Then, this article along with the summarization request will be sent to the controller via jQuery Ajax call. The controller communicates with our NATS models asynchronously via JSON format data. Finally, generated headlines and summaries are shown in the view.
Design of Frontend
Fig. FIGREF13 presents the front-end design of our web application for creating a new post, where labels represent the sequence of actions. In this website, an author can first click on “New Post” (step 1) to bring a new post view. Then, he/she can write content of an article in the corresponding text-area (step 2) without specifying it's headline and highlights, i.e., summary. By clicking “NATS” button (step 3) and waiting for a few seconds, he/she will see the generated headlines and highlights for the article in a new tab on the right hand side of the screen. Here, each of the buttons in gray color denotes the resource of the training data. For example, “Bytecup” means the model is trained with Bytecup headline generation dataset. The tokenized article content is shown in the bottom. Apart from plain-text headlines and highlights, our system also enables users to get a visual understanding of how each word is generated via attention weights BIBREF24 . When placing the mouse tracker (step 4) on any token in the headlines or highlights, related content in the article will be labeled with red color. If the author would like to use one of the suggestions, he/she can click on the gray button (step 5) to add it to the text-area on the left hand side and edit it. Finally, he/she can click “Post” (step 6) to post the article.
The Proposed Model
As shown in the Fig. FIGREF11 , our system can suggest to the users two headlines (based on Newsroom headline and Bytecup datasets) and summaries (based on Newsroom summary and CNN/DM datasets). They are treated as four tasks in this section. To achieve this goal, we use the modules provided in LeafNATS toolkit to assemble a new model (see Fig. FIGREF13 ), which has a shared embedding layer, a shared encoder layer, a task specific encoder-decoder (Bi-LSTM encoder and pointer-generator decoder) layer and a shared output layer.
To train this model, we first build a multi-task learning pipeline for Newsroom dataset to learn parameters for the modules that are colored in orange in Fig. FIGREF13 , because (1) articles in this dataset have both headlines and highlights, (2) the size of the dataset is large, and (3) the articles come from a variety of news agents. Then, we build a transfer learning pipeline for CNN/Daily and Bytecup dataset, and learn the parameters for modules labeled with blue and green color, respectively. With LeafNATS, we can accomplish this work efficiently.
The performance of the proposed model on the corresponding testing sets are shown in Table TABREF14 . From the table, we observe that our model performs better in headline generation tasks. However, the ROUGE scores in summarization tasks are lower than the models without sharing embedding, encoder and output layers. It should be noted that by sharing the parameters, this model requires less than 20 million parameters to achieve such performance.
Conclusion
In this paper, we have introduced a LeafNATS toolkit for building, training, testing/evaluating, and deploying NATS models, as well as a live news blogging system to demonstrate how the NATS models can make the work of writing headlines and summaries for news articles more efficient. An extensive set of experiments on different benchmark datasets has demonstrated the effectiveness of our implementations. The newly proposed model for this system has achieved competitive results with fewer number of parameters.
Acknowledgments
This work was supported in part by the US National Science Foundation grants IIS-1619028, IIS-1707498 and IIS-1838730.

Question: What models are included in the toolkit?
Answer:  recurrent neural network (RNN)-based sequence-to-sequence (Seq2Seq) models for NATS