Many of these features have also found applications in other language generation tasks, such as machine translation  and dialog systems  . In addition, other techniques that can also be shared across different tasks include training strategies  ,  ,  , data pre-processing, results post-processing and model evaluation. Therefore, having an open-source toolbox that modularizes different network components and unifies the learning framework for each training strategy can benefit researchers in language generation from various aspects, including efficiently implementing new models and generalizing existing models to different tasks. In the past few years, different toolkits have been developed to achieve this goal. Some of them were designed specifically for a single task, such as ParlAI  for dialog research, and some have been further extended to other tasks. For example, OpenNMT  and XNMT  are primarily for neural machine translation (NMT), but have been applied to other areas. The bottom-up attention model  , which has achieved state-of-the-art performance for abstractive text summarization, is implemented in OpenNMT. There are also several other general purpose language generation packages, such as Texar  .

For example, these procedures include saving/loading check-point files during training, selecting N-best models during validation, and using the best model for generation during testing, etc. Another feature of this engine is that it allows users to specify part of a neural network to train and reuse parameters from other models, which is convenient for transfer learning. Modules: Modules are the basic building blocks of different models. In LeafNATS, we provide ready-to-use modules for constructing recurrent neural network (RNN)-based sequence-to-sequence (Seq2Seq) models for NATS, e.g., pointer-generator network  . These modules include embedder, RNN encoder, attention  , temporal attention  , attention on decoder  and others. We also use these basic modules to assemble a pointer-generator decoder module and the corresponding beam search algorithms. The embedder can also be used to realize the embedding-weights sharing mechanism  . Data and Tools: Different models in LeafNATS are tested on three datasets (see Table TABREF5 ), namely, CNN/Daily Mail (CNN/DM)  , Newsroom  and Bytecup. The pre-processed CNN/DM data is available online.

LeafNATS: An Open-Source Toolkit and Live Demo System for Neural Abstractive Text Summarization	Neural abstractive text summarization (NATS) has received a lot of attention in the past few years from both industry and academia. In this paper, we introduce an open-source toolkit, namely LeafNATS, for training and evaluation of different sequence-to-sequence based models for the NATS task, and for deploying the pre-trained models to real-world applications. The toolkit is modularized and extensible in addition to maintaining competitive performance in the NATS task. A live news blogging system has also been implemented to demonstrate how these models can aid blog/news editors by providing them suggestions of headlines and summaries of their articles.

Introduction	Being one of the prominent natural language generation tasks, neural abstractive text summarization (NATS) has gained a lot of popularity  ,  ,  . Different from extractive text summarization  ,  ,  , NATS relies on modern deep learning models, particularly sequence-to-sequence (Seq2Seq) models, to generate words from a vocabulary based on the representations/features of source documents  ,  , so that it has the ability to generate high-quality summaries that are verbally innovative and can also easily incorporate external knowledge  . Many NATS models have achieved better performance in terms of the commonly used evaluation measures (such as ROUGE  score) compared to extractive text summarization approaches  ,  ,  . We recently provided a comprehensive survey of the Seq2Seq models  , including their network structures, parameter inference methods, and decoding/generation approaches, for the task of abstractive text summarization. A variety of NATS models share many common properties and some of the key techniques are widely used to produce well-formed and human-readable summaries that are inferred from source articles, such as encoder-decoder framework  , word embeddings  , attention mechanism  , pointing mechanism  and beam-search algorithm  .

Conclusion	In this paper, we have introduced a LeafNATS toolkit for building, training, testing/evaluating, and deploying NATS models, as well as a live news blogging system to demonstrate how the NATS models can make the work of writing headlines and summaries for news articles more efficient. An extensive set of experiments on different benchmark datasets has demonstrated the effectiveness of our implementations. The newly proposed model for this system has achieved competitive results with fewer number of parameters.

Compared with these toolkits, LeafNATS is specifically designed for NATS research, but can also be adapted to other tasks. In this toolkit, we implement an end-to-end training framework that can minimize the effort in writing codes for training/evaluation procedures, so that users can focus on building models and pipelines. This framework also makes it easier for the users to transfer pre-trained parameters of user-specified modules to newly built models. In addition to the learning framework, we have also developed a web application, which is driven by databases, web services and NATS models, to show a demo of deploying a new NATS idea to a real-life application using LeafNATS. Such an application can help front-end users (e.g., blog/news authors and editors) by providing suggestions of headlines and summaries for their articles. The rest of this paper is organized as follows: Section SECREF2 introduces the structure and design of LeafNATS learning framework. In Section SECREF3 , we describe the architecture of the live system demo. Based on the request of the system, we propose and implement a new model using LeafNATS for headline and summary generation. We conclude this paper in Section SECREF4 .

Here, we provide tools to pre-process the last two datasets. Data modules are used to prepare the input data for mini-batch optimization. Playground: With the engine and modules, we can develop different models by just assembling these modules and building pipelines in playground. We re-implement different models in the NATS toolkit  to this framework. The performance (ROUGE scores  ) of the pointer-generator model on different datasets has been reported in Table TABREF6 , where we find that most of the results are better than our previous implementations  due to some minor changes to the neural network.

LeafNATS Toolkithttps://github.com/tshi04/LeafNATS	In this section, we introduce the structure and design of LeafNATS toolkit, which is built upon the lower level deep learning platform – Pytorch  . As shown in Fig. FIGREF2 , it consists of four main components, i.e., engines, modules, data and tools and playground. Engines: In LeafNATS, an engine represents a training algorithm. For example, end-to-end training  and adversarial training  are two different training frameworks. Therefore, we need to develop two different engines for them. Specifically for LeafNATS, we implement a task-independent end-to-end training engine for NATS, but it can also be adapted to other NLP tasks, such as NMT, question-answering, sentiment classification, etc. The engine uses abstract data, models, pipelines, and loss functions to build procedures of training, validation, testing/evaluation and application, respectively, so that they can be completely reused when implementing a new model.

The Proposed Model	As shown in the Fig. FIGREF11 , our system can suggest to the users two headlines (based on Newsroom headline and Bytecup datasets) and summaries (based on Newsroom summary and CNN/DM datasets). They are treated as four tasks in this section. To achieve this goal, we use the modules provided in LeafNATS toolkit to assemble a new model (see Fig. FIGREF13 ), which has a shared embedding layer, a shared encoder layer, a task specific encoder-decoder (Bi-LSTM encoder and pointer-generator decoder) layer and a shared output layer. To train this model, we first build a multi-task learning pipeline for Newsroom dataset to learn parameters for the modules that are colored in orange in Fig. FIGREF13 , because (1) articles in this dataset have both headlines and highlights, (2) the size of the dataset is large, and (3) the articles come from a variety of news agents. Then, we build a transfer learning pipeline for CNN/Daily and Bytecup dataset, and learn the parameters for modules labeled with blue and green color, respectively. With LeafNATS, we can accomplish this work efficiently.

The performance of the proposed model on the corresponding testing sets are shown in Table TABREF14 . From the table, we observe that our model performs better in headline generation tasks. However, the ROUGE scores in summarization tasks are lower than the models without sharing embedding, encoder and output layers. It should be noted that by sharing the parameters, this model requires less than 20 million parameters to achieve such performance.

Architecture	This is a news/blog website, which allows people to read, duplicate, edit, post, delete and comment articles. It is driven by web-services, databases and our NATS models. This web application is developed with PHP, HTML/CSS, and jQuery following the concept of Model-View-Controller (see Fig. FIGREF9 ). In this framework, when people interact with the front-end views, they send HTML requests to controllers that can manipulate models. Then, the views will be changed with the updated information. For example, in NATS, we first write an article in a text-area. Then, this article along with the summarization request will be sent to the controller via jQuery Ajax call. The controller communicates with our NATS models asynchronously via JSON format data. Finally, generated headlines and summaries are shown in the view.

A Live System Demonstrationhttp://dmkdt3.cs.vt.edu/leafNATS	In this section, we present a real-world web application of the abstractive text summarization models, which can help front-end users to write headlines and summaries for their articles/posts. We will first discuss the architecture of the system, and then, provide more details of the front-end design and a new model built by LeafNATS that makes automatic summarization and headline generation possible.

Design of Frontend	Fig. FIGREF13 presents the front-end design of our web application for creating a new post, where labels represent the sequence of actions. In this website, an author can first click on “New Post” (step 1) to bring a new post view. Then, he/she can write content of an article in the corresponding text-area (step 2) without specifying it's headline and highlights, i.e., summary. By clicking “NATS” button (step 3) and waiting for a few seconds, he/she will see the generated headlines and highlights for the article in a new tab on the right hand side of the screen. Here, each of the buttons in gray color denotes the resource of the training data. For example, “Bytecup” means the model is trained with Bytecup headline generation dataset. The tokenized article content is shown in the bottom. Apart from plain-text headlines and highlights, our system also enables users to get a visual understanding of how each word is generated via attention weights  . When placing the mouse tracker (step 4) on any token in the headlines or highlights, related content in the article will be labeled with red color.

Acknowledgments	This work was supported in part by the US National Science Foundation grants IIS-1619028, IIS-1707498 and IIS-1838730.

If the author would like to use one of the suggestions, he/she can click on the gray button (step 5) to add it to the text-area on the left hand side and edit it. Finally, he/she can click “Post” (step 6) to post the article.

Question: What models are included in the toolkit?
Answer: recurrent neural network (RNN)-based sequence-to-sequence (Seq2Seq) models for NATS