Venue: EMNLP 2020

# Q-learning with Language Model for Edit-based

Unsupervised Summarization

 Ryosuke Kohita

IBM Research

{kohi, akifumi.wachi, yangzhao}@ibm.com &Akifumi Wachi

IBM Research

Yang Zhao

IBM Research

Ryuki Tachibana

IBM Research

ryuki@jp.ibm.com

###### Abstract

Unsupervised methods are promising for abstractive textsummarization in that the parallel corpora is not required. However, their performance is still far from being satisfied, therefore research on promising solutions is on-going. In this paper, we propose a new approach based on Q-learning with an edit-based summarization. The method combines two key modules to form an Editorial Agent and Language Model converter (\(\mathsf{ELLM}\)). The agent predicts edit actions (e.t., delete, keep, and replace), and then the LM converter deterministically generates a summary on the basis of the action signals. Q-learning is leveraged to train the agent to produce proper edit actions. Experimental results show that \(\mathsf{ELLM}\) delivered competitive performance compared with the previous encoder-decoder-based methods, even with truly zero paired data (i.e., no validation set). Defining the task as Q-learning enables us not only to develop a competitive method but also to make the latest techniques in reinforcement learning available for unsupervised summarization. We also conduct qualitative analysis, providing insights into future study on unsupervised summarizers.1

Footnote 1: Our codes are available at [https://github.com/](https://github.com/) kohilin/ealm

## 1 Introduction

Automatic text summarization2 is an attractive technique for helping humans to grasp the content of documents effortlessly. While supervised neural methods have shown good performances See et al. (2017); Zhang et al. (2019), the unsupervised approach is starting to attract interest due to its advantage of not requiring costly parallel corpora. However, the empirical performance of unsupervised methods is currently behind that of state-of-the-art supervised models Zhao et al. (2018); Baziotis et al. (2019). Unsupervised text summarization is still developing and is now at the stage where various solutions should be actively explored.

Footnote 2: We refer to abstractive summarization in this paper.

One previous unsupervised approach extends neural encoder-decoder modeling to the zero paired data scenario, where a model is trained with a paradigm called compression-reconstruction (CR) learning Miao and Blunsom (2016); Fevry and Phang (2018); Zhao et al. (2018). The mechanism is similar to that of the back-translation Sennrich et al. (2016): the model consists of a compressor (i.e., summarizer) and a reconstructor, and they are co-trained so that the reconstructor can recover the original sentence from the summary generated by the compressor Miao and Blunsom (2016); the left side of Figure 1). Experimental results showed that such an unsupervised encoder-decoder-based summarizer is able to learn the mapping from a sentence to a summary without paired data Baziotis et al. (2019); Yang et al. (2020).

Reinforcement learning (RL) is also a potential

Figure 1: Overview of previous (left) and proposed (right) approaches on CR learning paradigm.



solution for the no paired data situation. In related fields, for example, there are unsupervised methods for text simplification and text compression with policy-gradient learning (Zhang and Lapata, 2017; Zhao et al., 2018). Recent RL techniques take a value-based approach (e.g., Q-learning) such as DQN (Mnih et al., 2015) or the combination of policy and value-based approaches such as Asynchronous Advantage Actor-Critic (Mnih et al., 2016). A critical requirement to leverage a value-based method is a value function that represents the goodness of an action on a given state (Sutton et al., 1998). We can naturally define the value function by utilizing the CR-learning paradigm, and it makes the latest value-based approaches available for unsupervised text summarization.

In this paper, we propose a new method based on Q-learning and an edit-based summarization Gu et al. (2019); Malmi et al. (2019); right side of Figure 1). The edit-based summarization generates a summary by operating an edit action (e.g., keep, remove, or replace) for each word in the input sentence. Our method implements the editing process with two modules: 1) an Editorial Agent that predicts edit actions, and 2) a Language Mmodel (LM) converter that deterministically decodes a sentence on the basis of action signals, which we call EALM. The CR learning is defined on the Q-learning framework to train the agent to predict edit actions that instruct the LM converter to produce a good summary. Although a vast action space causing sparsity in reward, such as the word generation of an encoder-decoder model, is generally difficult to be learned in RL, our method mitigates this issue thanks to its fewer edit actions and the deterministic decoding of a language model. Moreover, the formulation by Q-learning enables us to incorporate the latest techniques in RL.

The main contribution of this paper is that we provide a new solution in the form of an unsupervised edit-based summarization leveraging Q-learning and a language model. Experimental results show that our method achieved a competitive performance with encoder-decoder-based methods even with truly no paired data (i.e., no validation set), and qualitative analysis brings insights as to what current unsupervised models are missing. Also, the problem formulation on Q-learning enables us to import the latest techniques in RL, which leads to potential improvements in future research.

## 2 Task Definition

We begin by formally defining the problem of unsupervised summarization with the CR learning. The goal of the task is to produce an informative summary \({\mathbf{y}}\) consisting of \(M\) words \(y_{1},y_{2},...,y_{M}\) for a given input sentence \({\mathbf{x}}\) consisting of \(N\) words \(x_{1},x_{2},...,x_{N}\) where \(M<N\). The challenge in this task is to learn the transformation from \({\mathbf{x}}\) to \({\mathbf{y}}\) with only the input sentence \({\mathbf{x}}\).

To tackle this, the CR learning introduces an additional transformation called reconstruction. The reconstruction requests to reproduce the input sentence \(\hat{{\mathbf{x}}}\) from the generated summary \({\mathbf{y}}\) where \(\hat{{\mathbf{x}}}\) is the reproduced sentence consisting of \(N\) words \(\hat{x}_{1},\hat{x}_{2},...,\hat{x}_{N}\). In terms of the generated sentences \({\mathbf{y}}\) and \(\hat{{\mathbf{x}}}\), let \(C\) be the compression function and \(R\) be the reconstruction function:

\[{\mathbf{y}}=C({\mathbf{x}},\theta_{C})~{},~{}\hat{{\mathbf{x}}}=R({\mathbf{y}},\theta_{R})~{},\]

where \(\theta_{C}\) and \(\theta_{R}\) are their respective parameters. Thus, the task can be written as the following optimization problem:

\[\theta^{*}_{C},\theta^{*}_{R}=\operatorname*{arg\,max}_{\theta_{C},\theta_{R}} \{f({\mathbf{x}},{\mathbf{y}})+g({\mathbf{x}},\hat{{\mathbf{x}}})\}~{},\]

where \(f({\mathbf{x}},{\mathbf{y}})\) and \(g({\mathbf{x}},\hat{{\mathbf{x}}})\) are functions to return a higher value for favorable \({\mathbf{y}}\) and \(\hat{{\mathbf{x}}}\) in regard to the input sentence \({\mathbf{x}}\). According to the CR learning's hypothesis that the summary should contain enough information to guess the original contents, \({\mathbf{y}}\) becomes favorable when the difference between \({\mathbf{x}}\) and \(\hat{{\mathbf{x}}}\) is smaller while \({\mathbf{y}}\) maintains the essential aspects of a summary (e.g., shortness, fluency).

## 3 Previous Method

The previous approaches use a generative encoder-decoder model Sutskever et al. (2014), for the compression and reconstruction functions Miao and Blunsom (2016); Fevry and Phang (2018); Wang and Lee (2018); Baziotis et al. (2019). Although the objective functions and implementation details differ depending on the study, the underlying motivation entails the same hypothesis as the CR learning. For example, Baziotis et al. (2019) introduced four objective functions -- discrepancy of \({\mathbf{y}}\) from a pre-trained language model, topical distance of \({\mathbf{x}}\) and \({\mathbf{y}}\), and the length of \({\mathbf{y}}\) and probability difference of \(x_{i}\) and \(\hat{x}_{i}\) -- where the former threes can be regarded as \(f({\mathbf{x}},{\mathbf{y}})\) and the final one as \(g({\mathbf{x}},\hat{{\mathbf{x}}})\).

While such an encoder-decoder model has performed well on many generation tasks, it suffers 

from inherent difficulties related to repetition (See et al., 2017), length control Kikuchi et al. (2016), and exposure bias Ranzato et al. (2016). It also runs into convergence problems when co-training multiple generators Salimans et al. (2016).

## 4 Proposed Method

Our proposed method, which we call EALM, consists of two essential modules: the editorial agent and the LM converter. The agent sends action signals (keep, remove, or replace each word in a sentence) to the conveter, which then deterministically transforms the input sentence according to the signals. We train the agent to find action signals so that the LM converter produces sentences demanded by the CR learning. In the following sections, we first share the background of Q-learning (SS4.1) and then present how to put the task and our approach on the Q-learning framework (SS4.2). We next explain the core algorithmic details (SS4.3) and finish with explanations about training and inference (SS4.4).

### Preliminaries

Q-learning is a popular approach in RL as represented by Deep Q-Networks DQN Mnih et al. (2015). Q-learning leverages an action-value function to estimate the _value_ of a pair of state and action with respect to a policy \(\pi\). The action-value function (i.e., Q-function) is represented as the expected reward for the state-action pair:

\[Q^{\pi}(s,a)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})~{}|~{ }s_{0}=s,a_{0}=a\right],\]

where \(s\) is a state, \(a\) is an action, \(r\) is a reward function for the state-action pair, and \(\gamma\) is the discount factor. Hence, to solve a text summarization task via Q-learning, we first need to appropriately define the state, action, and reward function.

### Unsupervised Edit-based Summarization with Q-learning

In our approach, given an input sentence \(\mathbf{x}\), we define a state \(s_{i}\) in regard to each word \(x_{i}\). An action \(a_{i}\) for the state \(s_{i}\) is chosen from among the three options, \(\mathcal{A}=\{\texttt{Remove},\texttt{Keep},\texttt{Replace}\}\). The goal of the editorial agent is to provide the optimal action sequence, \(\mathbf{a}=\{a_{0},a_{1},\dots,a_{N}\}\), by iteratively making action decisions on each word (SS4.3.1). To obtain \(\mathbf{y}\) and \(\hat{\mathbf{x}}\), we propose a deterministic transformation algorithm based on \(\mathbf{a}\) and the LM converter (SS4.3.2). Finally, we define the reward function \(r\) to evaluate the action and action sequence in terms of the produced sentences \(\mathbf{y}\) and \(\mathbf{\hat{x}}\) (SS4.3.3). The reward function is designed to align with the CR learning paradigm and leads the agent into bringing the action sequence that generates an appropriate \(\mathbf{y}\) and \(\mathbf{\hat{x}}\).

### Algorithms

In this section, we describe three principle algorithms: 1) how to create \(s_{i}\) and to predict \(a_{i}\), 2) how to generate \(\mathbf{y}\) and \(\mathbf{\hat{x}}\) by means of \(\mathbf{a}\) and the LM converter, and 3) how to compute the reward.

#### 4.3.1 Iterative Action Prediction

The overall flow of iteratively predicting an action for each word is shown in Figure 2. The agent predicts an action for a state (i.e., a word) one by one, so we call one prediction a _step_ and express it with a subscription \((t)\). For example, \(s_{i(t)}\) and \(a_{i(t)}\) respectively denote the state and action for \(x_{i}\) at the \(t\)-th step. Note that \(a_{i(t)}\) has a predicted action if the agent has already done the prediction on \(x_{i}\) by the \(t\)-th step, otherwise \(a_{i(t)}\) is Keep. Also, we prepare a Boolean vector \(\mathbf{u}_{(t)}\) of length \(N\) representing the prediction statuses at the \(t\)-th step; \(u_{i(t)}\) is 1 if the prediction on the \(i\)-th word has been finished, otherwise, 0. The order to predict an action is determined by Q-values. Let \(s^{*}\) and \(a^{*}\) be a state-action pair to be operated next, which comes from the maximum Q-values over unoperated states:

\[s^{*},a^{*}=\operatorname*{arg\ max}_{s\in\mathcal{S}^{\prime},a\in\mathcal{ A}}Q(s,a),\]

where \(\mathcal{S}^{\prime}\) is defined as \(\mathcal{S}^{\prime}=(\forall_{i})\{s_{i(t)}~{}|~{}u_{i(t)}=0\}\). The agent then reiterates the predictions until it finishes determining an action on all words. By defining the state in regard to a single word instead of a whole sentence and asking the agent to determine the prediction order, we can handle variable sentence lengths in natural language. Note that this is not a left-to-right process; the agent conducts the prediction in the order of "confidence".

Next we explain how to encode \(s_{i(t)}\). To send the agent contextual information, such as the previous decisions, the prediction statuses, and the whole sentence, we dynamically create a state \(s_{i(t)}\) with a concatenation of two encodings; _local encoding_\(\mathbf{l}_{i(t)}\) and _global encoding_\(\mathbf{g}_{i(t)}\)

\[s_{i(t)}=[\mathbf{l}_{i(t)};\mathbf{g}_{i(t)}].\]

To create the two encodings, first, we map \(x_{i}\) to a fixed-sized vector \(\mathbf{e}_{i}\) with an arbitrary encoder (we 

use BERT; Devlin et al.2019), and \({\mathbf{e}_{i}}\) is repeatedly used throughout the process regardless of the steps. Then, we define the local encoding as

\[{\mathbf{l}_{i(t)}={\mathbf{e}_{i}}+{\mathbf{b}^{a_{i(t)}}}+{\mathbf{b}^{u_{i(t)}}}},\]

where \({\mathbf{b}^{a_{i(t)}}}\) and \({\mathbf{b}^{u_{i(t)}}}\) are learnable bias vectors for the action and prediction status of the \(i\)-th word, respectively. Next, we create the global encoding in a self-attention fashion as

\[{\mathbf{g}_{i(t)}}=\sum_{j}w_{j(t)}{\mathbf{I}_{j(t)}},\]

where \(w_{j(t)}\) is computed with ReLU:3

Footnote 3: We used \(\text{ReLU}(\cdot)\) instead of the conventional \(\exp(\cdot)\) because \(\exp(\cdot)\) caused the exploding gradient in our case.

\[w_{j(t)}=\frac{\text{ReLU}({\mathbf{l}_{i(t)}}\cdot{\mathbf{l}_{j(t)}})}{\sum_{k}\text {ReLU}({\mathbf{l}_{i(t)}}\cdot{\mathbf{l}_{k(t)}})}.\]

Thanks to the bias terms in \({\mathbf{l}_{i(t)}}\) and the self-attention in \({\mathbf{g}_{i(t)}}\), \(s_{i(t)}\) is aware of the previous decisions for each word and the interactions between those decisions. In addition, BERT encoding \({\mathbf{e}_{i}}\) enables us to take a whole sentence into account.

#### 4.3.2 Deterministic Decoding by Language Model with Action Signals

In this section, we explain how to compress and reconstruct sentences in a deterministic manner with the LM converter. For the LM converter, we use BERT (Devlin et al., 2019) which is a masked language model (MLM) trained to predict "masked" portions in a sentence. MLM can estimate the probability distribution of \(i\)-th word \(x_{i}\) in a sentence as \(p(x~{}|~{}{\mathbf{x}_{\backslash i}})\) where \({\mathbf{x}_{\backslash i}}\) is the same as \({\mathbf{x}}\) except that it has a mask at the \(i\)-th position (\(\langle...,x_{i-1},\texttt{[mask]},x_{i+1},...\rangle\); Wang and Cho 2019). \(L({\mathbf{x}_{\backslash i}})\) denotes a function to return a word with the highest probability for the \(i\)-th position.

The procedure to obtain \({\mathbf{y}}\) and \(\hat{{\mathbf{x}}}\) by using

Figure 3: Deterministic compression and reconstruction with masked language model

Figure 2: Algorithmic visualization of iterative action prediction



and MLM is shown in Figure 3. First, we convert \({\mathbf{x}}\) to a skeleton sequence \({\mathbf{z}}\) consisting of \(N\) tokens \(z_{1},z_{2},...,z_{N}\) where \(z_{i}\) is \(x_{i}\) if \(a_{i}\) is Keep, otherwise a null token \(\epsilon\). We then define our compression and reconstruction functions \(\tilde{C}\) and \(\tilde{R}\) as

A word is predicted only for \(\epsilon\) given by Replace in compression, but it does so for all \(\epsilon\) in reconstruction. Also, we set the original sentence as a prefixed context, which comes from \({\mathbf{x}}\) in compression and \({\mathbf{y}}\) in reconstruction, to make MLM aware of a former meaning. An example is shown in Figure 3, where MLM receives "_Machine learning is not perfect_ . [mask] _is_ [mask] ." as the compression input and predicts words for the [mask]s. If there are multiple masks, we conduct the prediction in an autoregressive fashion (see Appendix A.1). Note that while any language model can be used for the LM converter, MLM is advantageous because it utilizes before and after contexts, and there is no restriction on looking ahead at upcoming words.

#### 4.3.3 Stepwise Reward Computation

In this section, we explain the reward computation of the chosen action by referring to \({\mathbf{y}}\) and \(\hat{{\mathbf{x}}}\).

As stated in SS4.3.1, we have an action sequence \({\mathbf{a}}_{(t)}\) for every step \(t\). When we apply \(\tilde{C}\) and \(\tilde{R}\) to all the \({\mathbf{a}}_{(t)}\), we can obtain a list of tuples \((s,a,r,{\mathbf{x}},{\mathbf{y}},\hat{{\mathbf{x}}})_{(t)}\). A tuple -- let us say, _experience_ -- enables us to evaluate a state-action pair with respect to a single transition. In this section, we propose three techniques -- **step reward, violation penalty**, and **summary assessment** -- to evaluate the agent's behavior with the stepwise experiences. Refer to Table 1 to see how these work in reward computation with an actual example.

Before moving on to the details, let us define two important notions throughout this section, compression rate (\(cr\)) and reconstruction rate (\(rr\)):

\[cr_{(t)}=1-\frac{|{\mathbf{y}}_{(t)}|}{|{\mathbf{x}}|}\text{, }rr_{(t)}=\frac{|\{ i~{}|~{}x_{i}=\hat{x}_{i(t)}\}|}{|{\mathbf{x}}|}.\]

The CR learning assumes that the higher values of \(cr\) and \(rr\) are better. We use these for calculating rewards and pruning experiences.

Step Reward.The task of the agent is to produce an action sequence with which the LM converter generates an appropriately compressed sentence while keeping the reconstruction successful. As such, we define the reward function \(r\) as

\[r(s,a,{\mathbf{x}},{\mathbf{y}},\hat{{\mathbf{x}}})=r_{SR}+r_{SA},\]

where \(r_{SR}\) is the step reward that are designed to encourage the agent to improve the compression and reconstruction rate, respectively. \(r_{SA}\) is an additional score from the qualitative assessment of \({\mathbf{y}}\), which we explain later. Returning to the step reward \(r_{SR}\), it is a multiplication of \(r_{C}\) and \(r_{R}\) defined as

\[r_{SR}=r_{C}\times r_{R}\text{,}\]

\[r_{C}=1-\frac{|{\mathbf{y}}_{(t)}|}{|{\mathbf{y}}_{(t-1)}|}\text{, }r_{R}=\begin{cases}1&( rr_{(t)}>\tau_{(t)})\\ -1&(\text{otherwise})\end{cases},\]

where \(\tau_{(t)}\) is a minimum requirement for the reconstruction rate at the \(t\)-th step and is defined as \(\tau_{(t)}=1-t\frac{1-\tau}{N}\) with the hyperparameter \(\tau\in[0,1]\). If we set \(\tau=1\) that requests perfect reconstruction, then \(\tau_{(t)}=1\) regardless of \(t\). However, we need to forgive reconstruction failure to some extent because of the information loss in compression, and \(\tau\) adjusts the allowed number of failures. For example, \(\tau=0.5\) requests the model to recover at least half of the original sentence correctly.

Let us describe the behavior of the step reward \(r_{SR}\). First, the reward is 0 when the agent chooses Keep or Replace because \(r_{C}=0\) due to there being no change in the length of \({\mathbf{y}}\). Second, the reward gets a positive value when the agent chooses Remove and satisfies the requirement for the reconstruction rate (\(rr_{(t)}>\tau_{(t)}\)). Third, the reward gets a negative value when the agent chooses Remove, but the reconstruction rate is less than the requirement. In short, the step reward recommends Remove as long as the agent can recover the original word, and otherwise, Keep or Replace.

Violation Penalty.Sequential modeling, including that performed by our agent, essentially suffers from error propagation caused by incorrect predictions at an earlier stage Collins and Roark (2004). The violation penalty mitigates this issue by giving a negative reward to the latest problematic action and excluding experiences after the mistake 

Here, in addition to \(\tau\), we introduce the hyperparameter \(\rho\), which represents a minimum requirement for the compression rate. \(\rho_{(t)}\) denotes its threshold at the \(t\)-step defined as \(\rho_{(t)}=t\frac{\rho}{N}\), and the agent must satisfy the condition \(cr_{(t)}>\rho_{(t)}\). As the penalty, we forcibly assign \(-1\) reward for the state-action pair at the \(T\)-th step when the agent breaks either constraint of \(\tau_{(T)}\) or \(\rho_{(T)}\). In addition, we ignore experiences from step \((T+1)\) and onward. If the agent keeps predicting until the end, we define \(T=N\). Figure 4 shows how these constraints work for the experience sequence.

Summary Assessment.Although the step reward considers the compression and reconstruction ratios, it ignores the critical aspects of the generated summary such as replacement with a shorter synonym and fluency as a sentence. Here, we explain the \(r_{SA}\) mentioned in the previous paragraph and describe how to reflect such qualitative assessments to the reward given to the agent.

As the essential properties for \(y\), we take three perspectives into account: informativeness, shortness, and fluency. The informativeness refers to how much \(y\) retains the original meaning of \(x\), and the shortness and fluency are self-explanatory. To reflect these perspectives onto the agent's decision, we define \(r_{SA}\) as

\[r_{SA}=\frac{T}{N}\cdot [cr_{(T)}\times rr_{(T)}\] \[+\alpha\cdot sim(\mathbf{x},\mathbf{y}_{(T)})+\beta\cdot lh(\mathbf{y}_{(T)})],\]

where \(sim\) computes a similarity score of \(x\) and \(y\), and \(llh\) computes a log-likelihood of \(y\). \(\alpha\) and \(\beta\) are hyperparameters to adjust the importance of \(sim\) and \(lll\). In addition to \(r_{SR}\), we give \(r_{SA}\) to the experiences from the beginning to \(T\)-th steps as defined in the step reward paragraph.

Let us explain the terms inside the square brackets first. The first term, which is the multiplication of \(cr_{(T)}\) and \(rr_{(T)}\), aims for shortness and informativeness. It gets a higher value when the agent achieves the right balance of compression and reconstruction. The second term \(sim\) aims to evaluate informativeness brought about by Replace. Concretely, \(sim\) returns a semantic similarity score in the range of \([0,1]\) through the sentence vectors of \(\mathbf{x}\) and \(\mathbf{y}_{(T)}\) rather than just checking exact matches of words. The last term \(llh\) represents fluency via the log-likelihood of \(\mathbf{y}_{(T)}\) given by a pre-trained language model Zhao et al. (2018). We use BERT for the computation of \(sim\) and \(llh\)Devlin et al. (2019); Wang and Cho (2019); see Appendix A.3). Finally, \(T/N\) is the ratio of the number of operated words. It becomes closer to 1 when the agent is reaching a termination, i.e., finishing the prediction on all words by avoiding the violation penalty, which makes \(r_{SA}\) larger. In contrast, the agent who fails at an earlier stage gets a small value of \(r_{SA}\).

### Training and Inference

Training.Leveraging the experiences \((s,a,r,\mathbf{x},\mathbf{y},\hat{\mathbf{x}})\) in the replay buffer Lin (1992), the agent learns the policy for summarizing a sentence \(\mathbf{x}\) within the Q-learning framework. Specifically, we utilize DQN Mnih et al. (2015) to learn the Q-function \(Q^{*}\) corresponding to the optimal policy by minimizing the loss,

\[\mathcal{L}(\theta)=\mathbb{E}_{s,a,r,s^{\prime}}[(Q^{*}(s,a)-\psi)^{2}],\]

where \(\psi=r+\gamma\max_{a^{\prime}}\bar{Q}^{*}(s^{\prime},a^{\prime})\) and \(\bar{Q}\) is a target Q-function whose parameters are periodically updated in accordance with the latest network parameters. During the collection of experiences, RL requires the agent to explore an action on a given state for finding a better policy. As a unique point in this work, the agent must explore not only the action but also the order to predict. For both explorations, we use the \(\epsilon\)-greedy algorithm Watkins (1989) that stochastically forces the agent to ignore Q-values and to behave randomly (see Appendix A.2).

Inference.Our modeling that provides \(\mathbf{y}\) and \(\hat{\mathbf{x}}\) for each step has another advantage in terms of the inference. For the final output, we use \(\mathbf{y}\) at the \(t^{*}\)-th step that achieves the best balance of the compression and reconstruction ratios, where

Figure 4: Violation penalty for compression (left) and reconstruction (right). The x-axis is step and the y-axis is each ratio. The horizontal lines in the middle are \(\rho\) and \(\tau\), and the dashed lines represent \(\rho_{(t)}\) and \(\tau_{(t)}\). The circles represent a step where the agent breaks the constraints.

 

\(\arg\max_{t}\{cr_{(t)}+rr_{(t)}\}\). This is based on the trade-off relationship of compression and reconstruction as seen in the precision-recall curve.

## 5 Experiment

Baselines.We compare our proposed approach with three baselines: Lead-N, which simply takes the beginning N words as the summary, SEQ3, a recent encoder-decoder model Baziotis et al. (2019), and CMatch, a new approach without explicit reconstruction learning Zhou and Rush (2019). To conduct qualitative analysis on generated summaries, we ran the baselines ourselves with a replicated model for SEQ34 and the provided model for CMatch.5 Also, we test two types of SEQ3 models: one tuned with a validation set (SEQ3\({}^{+}\)) and the other with parameters at the last iteration in the training (SEQ3\({}^{-}\)). This is because EALM and CMatch do not need paired data even for validation.

Footnote 4: [https://github.com/cbaziotis/seq3](https://github.com/cbaziotis/seq3). We ran the training script with the same configuration as the original paper except for decreasing the batch size from 128 to 32 due to our GPU limitation. We trained three models and obtained slightly lower scores than the ones reported in the original paper. We report the averaged score among the three models.

Proposed method.We implemented EALM as follows. The Q-network of the agent consists of a two-layered MLP with 200 units per layer and ReLU. We used the Adam optimizer with the learning rate of 0.001 and apply gradient clipping by 1. For the epsilon-greedy strategy, we first set the exploring probability to 0.9 and decay it by multiplying by 0.995 every 100 updates until it reaches the minimum exploration rate of 0.03. We set the discount factor \(\gamma\) to 0.995. The size of the replay buffer is 2000, and we sample 128 experiences as a batch for one update. As the final model, we use parameters at a time when the averaged score of reward in the replay buffer is maximum, i.e., our model does not need a validation set. The hyperparameters of step reward (\(\tau\), \(\rho\); SS4.3.3) are set to 0.5 and 0.3, respectively. The hyperparameters of summary assessment (\(\alpha\), \(\beta\); SS4.3.3) are both set to 0.1. We train three models with the same configuration and report their averaged score, as Q-learning inherently contains randomness in training.

Dataset.The same as Baziotis et al. (2019), we train our model on the Gigaword corpus giga, Rush et al. (2015). However, we used only 30\(K\) sentences randomly picked from sentences with less than \(50\) words for the training of EALM. This is because the whole data, 3.8\(M\) sentences, is too large to expose the agent to different experiences from the same sentence.6 Note that we used the entirety of sentences for the training of the SEQ3 models.

Footnote 6: EALM can be trained with the large dataset, but it takes long time due to the exploitation and exploration learning strategy of Q-learning. 30K was better in the balance of the required time and the model performance.

We followed Baziotis et al. (2019) in the evaluation as well, using the test set consisting of the giga (1897 sentences) and DUC datasets (duc3 with 624 sentences, duc4 with 500 sentences; Over et al. 2007).

All models follow the same tokenization policy: the default tokenization in giga, duc3, and duc4. Although BERT (which EALM uses) has its own vocabulary based on subwords, we do not apply subwording to go along with a single tokenization policy. Therefore, words not in the BERT vocabulary are interpreted as unknown words, and the ratio of unknown words was around 10% in giga.

\begin{table}
\begin{tabular}{l|p{28.5pt}|p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}|p{28.5pt} p{28.5pt} p{28.5pt}|p{28.5pt}} \hline t & Action & Type & 1 & 2 & 3 & 4 & 5 & 6 & \(crP_{(t)}\) & \(rrf_{(t)}\) & \(\frac{1}{X}ferrrtsimllh\) & \(r_{SR}\) / \(rs_{SA}\) & \(r\) \\ \hline \multirow{2}{*}{1} & Remove (May) & \(\mathbf{y}_{(1)}\) & \multirow{2}{*}{day} & the & force & be & with & you & .17/.05 & 1.0/.91 & - & 1.0 / .20 & 1.2 \\ \cline{3-14}  & Remove & \(\mathbf{y}_{(2)}\) & & & force & be & with & you & & .33/.10 & 1.0/.83 & - & 1.0 / .20 & 1.2 \\ \cline{2-14}  & (the) & \(\mathbf{x}_{(2)}\) & May & the & force & be & with & you & & .50/.15 & .50/.75 & .50/.25/.50 /.10 & **-1.0** / .20 & -.80 \\ \hline \multirow{2}{*}{3} & Remove & \(\mathbf{y}_{(3)}\) & \multirow{2}{*}{1} & \multirow{2}{*}{will} & \multirow{2}{*}{always} & \multirow{2}{*}{be} & \multirow{2}{*}{with} & \multirow{2}{*}{you} & \multirow{2}{*}{.50/.15} & \multirow{2}{*}{.50/.75} & \multirow{2}{*}{.50/.25/.50} & \multirow{2}{*}{.10/.1} & \multirow{2}{*}{-} & \multirow{2}{*}{-} & \multirow{2}{*}{-} & \multirow{2}{*}{-} & \multirow{2}{*}{-} & \multirow{2}{*}{-} \\  & (force) & \(\mathbf{\hat{x}}_{(3)}\) & & & & & & & & & & & \\ \hline \multicolumn{14}{l}{} \\ \multicolumn{14}{l}{} \\ \multicolumn{14}{l}{} \\ \multicolumn{14}{l}{} \\ \multicolumn{14}{l}{} \\ \multicolumn{14}{l}{} \\ \multicolumn{14}{l}{} \\ \multicolumn{14}{l}{} \\ \multicolumn{14}{l}{} \\ \end{tabular} No experiences due to the violation occurred at the step 3.

\end{table}
Table 1: An example of stepwise reward computation. It breaks the reconstruction constraint at the step 3 when removing _force_, so \(r_{SR}=-1\). \(r_{SA}\) is computed at the step 3 by \(0.5\times(0.25+0.5\times 0.1+1.0\times 0.1)=0.20\), and it is used for the step 1 and 2 as well. The settings of hyperparameters are \(\tau=0.5\), \(\rho=0.3\), \(\alpha=0.1\), and \(\beta=0.1\).

 

Evaluation.In our quantitative analysis, we examine the ROUGE scores.7 To mitigate the bias to longer sentences in ROUGE calculation, we capped all summaries at the first 75 bytes. Note that the averaged sentence length of gold summaries after the capping were 8.58, 9.59, and 10.25 for giga , duc3, and duc4, respectively. Also, we examine sentence length (LEN) and count of new words (NW; number of words that are used in a generated summary but do not appear in the input sentence). Additionally, we show qualitative comparisons with a manual check of generated summaries. Although a questionnaire survey is often conducted to assess the deeper quality of summaries such as informativeness and readability, this still hides the exact points of model's strengths and weaknesses. We consider that specific indications provide insights on future work for the current unsupervised summarizers. We manually checked more than 200 summaries for each model and each dataset and include a few samples in Appendix A.6.

Footnote 7: We used files2rouge ([https://github.com/pltrady/files2rouge](https://github.com/pltrady/files2rouge)) following Baziotis et al., 2019.

Results.Table 2 lists the results of ROUGE scores, averaged lengths, and averaged counts of new words. EALM showed a better performance in duc3 and duc4 with respect to R-1 and R-L. In giga, it performed competitively with the baselines. However, the original length of the generated summaries tended to be longer, and the occurrence of new words was the lowest.

CMatch achieved the highest scores of ROUGE and meaningful length in giga. The scores of R-2 and R-L were superior to others by about two points, which means CMatch captured not only salient words but also word co-occurrences. However, for generating summaries, CMatch uses a language model trained with gold summaries in giga. In other words, it may just internally store the probable word distributions in summary sentences on giga. Actually, the results on duc3 and duc4 were not better than those on giga. Even though CMatch does not require paired data, it is not practical to collect enough summaries to train a language model for each domain.

SEQ3 showed a competitive performance with other models, but its scores dropped when no validation set was available. The requirement of a validation set is a keen disadvantage because creating input-summary pairs comes at a significant human labor cost.

While almost all of the best scores were given by the statistical models, Lead-15 also performed competiitively. This result indicates that unsupervised summarization methods can not yet overcome the trivial baseline. One significant barrier preventing the progress of unsupervised methods is presumably the difficulty of rephrasing. For writing a good summary, condensing a longer expression into a shorter form is essential. As seen in the NW column in Table 2, the number of new words was less than one in SEQ3\({}^{+}\), CMatch, and EALM. The current models tend to operate just by copy-and-paste, which is consistent with the report by Baziotis et al. (2019).

Finally, we manually assess the summaries produced by each model and sum up their pros and cons in Table 3. Also, actual examples are shown in 

Table 4. First, we found that a summary of SEQ3 was likely to be an exact copy of the input sentence from the top, but it kept sentences grammatical and informative. Rephrasing by SEQ3 did not meet our expectation in most cases, such as changing a week of the day (e.g., _Wednesday_ to _Thursday_) or a common adjective to a pronoun adjective (e.g., _astronomical_ to _her_). CMatch stably generated fluent summaries in giga, as seen in the ROUGE scores. It also generated grammatically correct sentences such as number agreement (e.g., _nec agrees_ ...). In the DUC datasets, however, meaningless summaries increased, such as containing no important information (e.g., _nasa observes_). Relatedly, CMatch's summaries on duc3 and duc4 were too short, and we found that more than half of the summaries consisted of less than or equal to 5 words. Finally, EALM's outputs tended to be longer due to containing non-informative portions (e.g., _nasa officials told_ ....). It was also likely to be ungrammatical due to leaving only a functional word (e.g., _mechanical problems that threaten_ ...) or deleting required prepositions (e.g., _... agreed (to join_ ...). Those failures resulted in lower readability. However, EALM tried to keep keywords from the whole input even though they exist at latter positions in a sentence, which is also supported by the relatively higher score of R-1 and R-L. Although this _challenge_ caused low readable and ungrammatical summaries, it is an interesting research direction to sophisticate such EALM's behavior.

## 6 Conclusion

We brought the Q-learning framework into unsupervised text summarization and proposed a new method EALM that is an edit-based unsupervised summarizer leveraging a Q-learning agent and a language model. The experiments showed that EALM performed competitively with the previous encoder-decoder-based methods. However, in qualitative analysis, we found that the quality of the generated summaries of any unsupervised model was not sufficient, and there are individual limitations for each model. These issue must be overcome as the step forward to generating practically available summaries without paired data. In particular for EALM, there is room for improvement by importing the latest techniques in RL. Our work paves the way for further research on bridging Q-learning and unsupervised text summarization.

\begin{table}
\begin{tabular}{p{72.3pt}|p{85.4pt}|p{90.6pt}|p{90.6pt}|p{90.6pt}} \hline INPUT & Human & SEQ3 & CMatch & EALM \\ \hline japan ’s nec corp. and **UNK** computer corp. of the united states said wednesday they had agreed to join forces in supercomputer sales & nec UNK in computer sales tie-up & Japan’s nec corp. and _her_ computer corp. of the united states said the united states said & nec computer united states said agreed _(to)_ joint forces in sales \\ \hline mechanical problems **that** threaten ** astronomical observations of the hubble space telescope may **prompt a repair mission** six months earlier than planned to the § 1.7 billion spacecraft , nasa officials told congress on Wednesday & mechanical problems that threaten to shut down the _her_ observations of the hubspace telescope may _repair mission_ six months earlier than planned to the § 1.7 billion spacecraft , nasa officials told congress on Wednesday & _nasa observes_ for the swatomised observations of space telescope may _ prompt mini_ mission six months earlier than planned to the § 1.7 billion spacecraft , nasa officials told congress on Wednesday & _that_ threaten to shut down astronomical observations of space telescope may _prompt repair mission_ six months earlier than planned to the § 1.7 billion spacecraft , nasa officials told congress on Wednesday & _that_ threaten to shut down astronomical observations of space telescope may _prompt repair mission_ six carried than planned bilion spacecraft _nasa officials told congress on_ & _down astronomical observations of space telescape may _prompt repair mission_ six earlier than planned bilition spacecraft _nasa officials told congress on_ & _repair mission_ six early time when planned billion spacecraft _nasa officials told congress on_ & _repair mission_ six months after two building blocks of the international space station station_ & _oneavour ‘s astronauts connected the first two _building blocks of the international space_ on _creating tower in shuttle bay_ & _conected first _building blocks of international space_ on _creating tower in shuttle boy_ & _nonceders_ & _no blocks of international space_ on _creating tower in shuttle boy_ \\ \hline \end{tabular}
\end{table}
Table 4: Summaries by Human (gold reference), SEQ3, CMatch and EALM from giga (top), duc3 (center), and duc4 (bottom).

 



## References

* Baziotis et al. (2019) Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, and Alexandros Potamianos. 2019. SEQ'3: Differentiable sequence-to-sequence-to-sequence autoencoder for unsupervised abstractive sentence compression. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 673-681. Cited by: SS4.
* M. Collins and B. Roark (2004)Incremental parsing with the perceptron algorithm. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pp. 111-118. Cited by: SS5.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171-4186. Cited by: SS1, SS2.
* T. Fevry and J. Phang (2018)Unsupervised sentence compression using denoising autoencoders. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pp. 413-422. Cited by: SS1, SS2.
* J. Gu, C. Wang, and J. Zhao (2019)Levenshtein transformer. In Proceedings of 32th International Conference on Neural Information Processing Systems, pp. 11179-11189. Cited by: SS1, SS2.
* Y. Kikuchi, G. Neubig, R. Sasano, H. Takamura, and M. Okumura (2016)Controlling output length in neural encoder-decoders. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1328-1338. Cited by: SS1, SS2.
* L. Ji Lin (1992)Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning8, pp. 293-321. Cited by: SS1, SS2.
* E. Malmi, S. Krause, S. Rothe, D. Mirylenka, and A. Severyn (2019)Encode, tag, realize: high-precision text editing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 5054-5065. External Links: Link, Document Cited by: SS1, SS2.
* Y. Miao and P. Blunsom (2016)Language as a latent variable: discrete generative models for sentence compression. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 319-328. Cited by: SS1, SS2.
* V. Mnih, A. Puigdomenech Badia, M. Mirza, A. Graves, T. Harley, T. P. Lillicrap, D. Silver, and K. Kavukcuoglu (2016)Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning, pp. 1928-1937. Cited by: SS1, SS2.
* Volodymyr Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis (2015)Human-level control through deep reinforcement learning. Nature518 (7540), pp. 529-533. Cited by: SS1, SS2.
* P. Over, H. Dang, and D. Harman (2007)Duc in context. Information Processing & Management43 (6), pp. 1506-1520. Cited by: SS1, SS2.
* M. Ranzato, S. Chopra, M. Auli, and W. Zaremba (2016)Sequence level training with recurrent neural networks. In Proceedings of 4th International Conference on Learning Representations, Cited by: SS1, SS2.
* A. M. Rush, S. Chopra, and J. Weston (2015)A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 379-389. Cited by: SS1, SS2.
* Tim Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen, and X. Chen (2016)Improved techniques for training gans. In Proceedings of the 29th Neural Information Processing Systems, pp. 2234-2242. Cited by: SS1, SS2.
* A. See, P. J. Liu, and C. D. Manning (2017)Get to the point: summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp. 1073-1083. Cited by: SS1, SS2.
* R. Sennrich, B. Haddow, and A. Birch (2016)Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, pp. 86-96. Cited by: SS1, SS2.
* I. Sutskever, O. Vinyals, and Q. V. Le (2014)Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems, pp. 3104-3112. Cited by: SS1, SS2.
* R. S. Sutton, A. G. Barto, et al. (1998)Introduction to reinforcement learning. Vol. 135, MIT press Cambridge. Cited by: SS1, SS2.
* A. Wang and K. Cho (2019)BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pp. 30-36.



Yaushan Wang and Hung-Yi Lee. 2018. Learning to encode text as human-readable summaries using generative adversarial networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4187-4195. Cited by: SS3.
* C. J. Cornish Hellaby Watkins (1989)Learning from delayed rewards. Ph.D. Thesis, King's College, Cambridge, UK. Cited by: SS3.
* Z. Yang, C. Zhu, R. Gmyr, M. Zeng, X. Huang, and E. Darve (2020)Ted: a pretrained unsupervised summarization model with theme modeling and denoising. External Links: Link Cited by: SS3.
* H. Zhang, J. Cai, J. Xu, and J. Wang (2019)Pretraining-based natural language generation for text summarization. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), Hong Kong, China, pp. 789-797. External Links: Document, ISSN 1674-17X Cited by: SS3.
* X. Zhang and M. Lapata (2017)Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 584-594. External Links: ISBN 978-3-319-5033-3, Link, Document Cited by: SS3.
* Y. Zhao, Z. Luo, and A. Aizawa (2018)A language model based evaluator for sentence compression. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 170-175. Cited by: SS3.
* J. Zhou and A. Rush (2019)Simple unsupervised summarization by contextual matching. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 5101-5106. External Links: Document, ISSN 1536-147X Cited by: SS3.

 

Appendices

### Autoregressive Prediction with MLM

Algorithm 1 describes the autoregressive prediction with MLM, which we used when an input contains multiple masks.

```
Input: a sentence \(x\) that includes [mask]s Output: a sentence \(x\) after replacing all [mask]s with predicted words \(I\leftarrow(\forall i)\{i~{}|~{}x_{i}=\texttt{[mask]}\}\) while\(I\neq\phi\)do for\(j\in I\)do \(w_{j}\gets L(\mathbf{x}_{\backslash j})\) endfor \(j^{*}~{}\leftarrow\operatorname*{arg\;max}_{j\in I}P(w_{j}|x_{\backslash j})\) \(x_{j^{*}}~{}\leftarrow~{}w_{j^{*}}\) \(I~{}\leftarrow~{}I_{\backslash j^{*}}\) endwhile
```

**Algorithm 1** Autoregressive prediction with MLM

### Exploration of Prediction Order

As explained in section 4.3 in the main paper, the editorial agent explores the order to predict. While the agent basically chooses a state with a maximum Q-value as the next state, we sometimes pick a most uncertain state instead. We define the uncertainty of a state by the entropy of action probabilities as \(H(s)=-\sum_{a\in\mathcal{A}}Q(s,a)\log Q(s,a)\), and then \(s^{*}\) and \(a^{*}\) are selected as

\[s^{*}=\operatorname*{arg\;max}_{s\in s^{t}}H(s)~{},\,a^{*}=\operatorname*{ arg\;max}_{a\in\mathcal{A}}Q(s^{*},a)~{}.\]

### Semantic Similarity and Log-likelihood Computation in Summary Assessment

#### Semantic Similarity.

We use a pre-trained model to predict the semantic similarity of paired-sentences with their BERT encodings.8 The model is trained in a supervised manner with a pair of sentences and their similarity score. The original library outputs a real-valued score in the range of \([0,5]\), whereas we normalize it to \([0,1]\).

Footnote 8: [https://github.com/AndriyMulvar/semantic-text-similarity](https://github.com/AndriyMulvar/semantic-text-similarity)

#### Log-likelihood.

We compute the log-likelihood of a compressed sentence by using BERT as follows Wang and Cho (2019):

\[\frac{1}{M}\sum_{i\in M}\log(P(y_{i}~{}|~{}\mathbf{y}_{\backslash i}))~{}.\]

However, our \(llh\) function performs thresholding -- namely, it returns 1 if the score is beyond a threshold, otherwise 0 -- because the raw log-likelihood score is not scaled with the other rewards. We empirically set the threshold to 0.005.

### Relaxations in \(rr_{(t)}\) Calculation

The calculation of the reconstruction rate introduced in section 4.3.3 is based on an exact match of each word of \(\mathbf{x}\) and \(\hat{\mathbf{x}}\). Given the ambiguity of natural language, this is very strict, so the agent rarely acquires rewards. We relax this situation by 1) excluding stop words in the calculation and 2) comparing with top-k candidates. Therefore, the equation of \(rr\) can be formally re-written as

\[rr_{(t)}=\frac{|\{i~{}|~{}x_{i}\in L^{k}(\mathbf{z}_{\setminus i})\cap x_{i}\notin W \}|}{|\{i~{}|~{}x_{i}\notin W\}|}~{},\]

where \(L^{k}(\mathbf{z}_{\setminus i})\) returns top-k probable words for the \(i\)-th position and \(W\) is a pre-defined set of words. We set \(k=10\). We used common stopwords (e.g., _him_, _the_) and infrequent words in giga for \(W\).

### Experimental Details

Computing Infrastructure.We run the models on a machine with the below specifications:

* Ubuntu 18.04
* Intel(R) Xeon(R) @ 2.60GHz
* RAM 120GB
* NVIDIA Tesla P100

Model Size.In EALM, the number of trainable parameters was 348208 in our experimental setting, which is all for the editorial agent. There are no trainable parameters for the language model.

Hyperparameter Search.We did not conduct a hyperparameter search. We empirically determined the values described in the main paper (the "Proposed method" paragraph in SS5).

Runtime Speed.ELM processes a sentence in three seconds pm average on the above GPU.

### Generated Summaries

Samples of the summaries generated by each model are listed in the tables on the following next pages. These examples are taken from the first sentences for giga and randomly picked for duc3 and duc4. We also include human-generated summaries (i.e., gold reference) 

* [25] M. C. Bader, R. J. Brogg, and A. K. Sridhar, "The automated automated computation of the number of events in a single event using data-driven control," _Journal of Statistical Analysis_, vol. 106, no. 3, pp. 479-484, 2000.
* [26] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in _Proceedings of the Thirtieth International Conference on Machine Learning Research (ICML)_, 2014.
* [27] G. Hinton, I. Goodfellow, Y. Bengio, and L. Bottou, "Deep learning with deep neural networks," _arXiv preprint arXiv:1512.01581_, 2015.
* [28] H. Li, X. Liu, Z. Chen, and Q. Weinberger, "Learning generative adversarial nets for large scale machine learning," _IEEE Transactions on Neural Networks_, vol. 10, no. 1, pp. 1-10, Jan. 2015.
* [29] T. Lin, W. Dong, and N. Yang, "A fast approach to image recognition via deep convolutional neural network," _CoRR_, vol. abs/1512.01581, 2015.
* [30] V. LeCun, E. Vanhoucke, and O. Zeil, "ImageNet: A simple framework for image segmentation," _IEEE transactions on pattern analysis and machine intelligence_, vol. 10, no. 1, pp. 1-10, Feb. 2015.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{c|p{142.3pt} p{56.9pt}|p{85.4pt}|p{70.7pt}|p{85.4pt}} \hline INPUT & Human & SEQ3 & CMatch & EALM \\ \hline mechanical problems that threaten to shut down the astronomical observations of the hubble space telescope may prompt a repair mission than planned to the S 1.7 billion spacecraft , nasa officials told congress on wednesday . & Problems may stop through mobile spacec telepsing a repair mission earlier than planned to the S 1.7 billion spacecraft , nasa officials told congress on Wednesday . & mechanical problems that threaten to shut down the her observations of the hubule space telescope . & nasa observes that threaten to shut down astronomical observations of space talescing may prompt regular mission six earlylier than planned bilition spacecraft nasa officials told congress on Wednesday . & that threaten to shut down astronomical observations of space talescing may prompt regular mission six earlylier than planned bilition spacecraft nasa officials told congress on Wednesday . & maneda acknowledges national congress violated human during apartheid setting africa & mandela acknowledggrees national congress violated human during apartheid setting africa & mandela acknowledggrees national congress violated human during setting africa & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting africa & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south affica . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress national congress violated human during setting at odds with deputy president over report that divided much of south . & mandela acknowledggrees national congress national congress national congress without additional congress national congress around around and rejected support that divided much of south . & mandela acknowledggrees national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congressional congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress international congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congress national congressress national congress national congress national congress national congress national congress national congressress national congress national congress national congress national congressress national congress national congress national congressress national congress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congressress national congress national congressress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congressress national congress national congress national congressress national congress



\begin{table}
\begin{tabular}{|p{56.9pt}|p{82.4pt}|p{71.3pt}|p{71.3pt}|p{71.3pt}|} \hline INPUT & Human & SECQ3 & CMatch & EALM \\ \hline endeavour ’s astronauts connected the first two building blocks of the international space stations on Sunday , creating a seven-story tower in the shuttle cargo bay . & First 2 building blocks of international space station successfully joined. & endeavour ’s astronants connected the first two building blocks of international space on creation tower in shuttle & connected first buildling blocks of international space on created one-stage on the shuttle \\ \hline in a cocoon of loyal and wealthy supporters , president Clinton said Friday that he must “ live with the consequences ” of his mistakes , although he contended that democrats should take pride in the achievements of his presidency and take heart from its possibilities . & in a her of loyal and wealthy supporters , president Clinton said that must of payload and wealthy supporters , Clinton “ . & democrats take pride in a of loyal and wealthy supporters , president Clinton said that must live with consequences of mistakes although he that should pride in achievements of fresibilities and heart from possibilities \\ \hline on the eve of a holiday that has been linked to antiabortion violence , the authorities on tues-day were investigating whether a picture of an aborted fetus sent to a canadian newspaper was connected to last month’s fatal shooting of a buffalo , n.y. doctor who provided abortions or four similar attacks in western new york and canada since 1994 . & on the eve of a holiday that has been linked to either violence or investigating picture of an sent to Canadian newspaper was connected to last month fatal shooting of buffalo , doctor who provided or similar attacks in western new york and canada since 1994 \\ \hline famine-threatened north korea ’s harvest will be no better this year than last and could be worse , a senior u.n. aid official said saturday .. & World Food Program reports famine may have killed 2 million North Koreans & her north korea ’s harvest will be no better this year than last worse & south korea ’s zhan better last could work s senior aid official said \\ \hline mathwe Wayne shepard , the gay student who was beaten in the dead of night , tied to a fence and left to die alone , was mounted at his funeral friday by 1,000 people , including many who had never then him . & Matthew Shepard utilized as one who wanted to make people’s lives better & matthew wayne her , the gay student who was beaten in the dead of night , gay student who was beaten in the dead of night , gay student who was beaten him & us coos , gay who beaten in dead of night tied to defence and left to die alone was at funeral by including who had met \\ \hline a delegation of chilegislators lobbying against the possible extradition of guinochet to spain to face thursday that chiles was on the brink of political turmoil . & Chilean legislators protest in Madrid against extradition of guchest to face trial & a delegation of chilean legislators lobbying against the possible extradition of augusto phonetic to face turnoil & delegation of chilean legislators faces trial & delegation of chilean legislators lobbying against possible of augusto pain to trial , warned chile on brink of political turmoil \\ \hline \end{tabular}
\end{table}
Table 7: Summaries by Human (gold reference), SEQ3, CMatch and EALM from duc4.