Venue: ACL 2017

# Japanese Sentence Compression with a Large Training Dataset

Shun Hasegawa

Tokyo Institute of Technology, Japan hasegawa.s@lr.pi.titech.ac.jp Yuta Kikuchi

Preferred Networks, Inc., Japan kikuchi@preferred.jp Hiroya Takamura

Tokyo Institute of Technology, Japan takamura@pi.titech.ac.jp Manabu Okumura

Tokyo Institute of Technology, Japan oku@pi.titech.ac.jp

###### Abstract

In English, high-quality sentence compression models by deleting words have been trained on automatically created large training datasets. We work on Japanese sentence compression by a similar approach. To create a large Japanese training dataset, a method of creating English training dataset is modified based on the characteristics of the Japanese language. The created dataset is used to train Japanese sentence compression models based on the recurrent neural network.

## 1 Introduction

Sentence compression is the task of shortening a sentence while preserving its important information and grammaticality. Robust sentence compression systems are useful by themselves and also as a module in an extractive summarization system Bergkirkpatrick et al. (2011); Thadani and McKeown (2013). In this paper, we work on Japanese sentence compression by deleting words. One advantage of compression by deleting words as opposed to abstractive compression lies in the small search space. Another one is that the compressed sentence is more likely to be free from incorrect information not mentioned in the source sentence.

There are many sentence compression models for Japanese Harashima and Kurohashi (2012); Hirao et al. (2009); Hori and Furui (2004) and for English Knight and Marcu (2000); Turner and Charniak (2005); Clarke and Lapata (2006). In recent years, a high-quality English sentence compression model by deleting words was trained on a large training dataset Filippova and Altun (2013); Filippova et al. (2015). While it is impractical to create a large training dataset by hand, one can be created automatically from news articles Filippova and Altun (2013). The procedure is as follows (where \(S\), \(H\), and \(C\) respectively denote the first sentence of an article, the headline, and the created compressed sentence of _S_). Firstly, to restrict the training data to grammatical and informative sentences, only news articles satisfying certain conditions are used. Then, nouns, verbs, adjectives, and adverbs (i.e., content words) shared by \(S\) and \(H\) are identified by matching word lemmas, and a rooted dependency subtree that contains all the shared content words is regarded as \(C\).

However, their method is designed for English, and cannot be applied to Japanese as it is. Thus, in this study, their method is modified based on the following three characteristics of the Japanese language: (a) Abbreviation of nouns and nominalization of verbs frequently occur in Japanese. (b) Words that are not verbs can also be the root node especially in headlines. (c) Subjects and objects that can be easily estimated from the context are often omitted.

The created training dataset is used to train three models. The first model is the original Filippova et al.'s model, an encoder-decoder model with a long short-term memory (LSTM), which we extend in this paper to make the other two models that can control the output length Kikuchi et al. (2016), because controlling the output length makes a compressed sentence more informative under the desired length.

## 2 Creating training dataset for Japanese

Filippova et al.'s method of creating training data consists of the conditions imposed on news articles and the following three steps: (1) identification of shared content words, (2) transformation of a dependency tree, and (3) extraction of the min

imum rooted subtree. We modified their method based on the characteristics of the Japanese language as follows. To explain our method, a dependency tree of \(S\) and a sequence of _bunsetsu_ chunks of \(H\) in Japanese are shown in Figures 1 and 2. Note that nodes of dependency trees in Japanese are _busetsu_ chunks each consisting of content words followed by function words.

### Identification of shared content words

Content words shared by \(S\) and \(H\) are identified by matching lemmas and pronominal anaphora resolution in Filippova et al.'s method. Abbreviation of nouns and nominalization of verbs frequently occur in Japanese (characteristic (a) in Introduction), and it is difficult to identify these transformations simply by matching lemmas. Thus, after the identification by matching lemmas, two identification methods (described below) using character-level information are applied. Note that pronominal anaphora resolution is not used, because pronouns are often omitted in Japanese.

**Abbreviation of nouns:** There are two types of abbreviations of nouns in Japanese. One is the abbreviation of a proper noun, which shortens the form by deleting characters (e.g., the pair with "+" in Figures 1 and 2). The other is the abbreviation of consecutive nouns, which deletes nouns that behave as adjectives (e.g., the pair with "-"). To deal with such cases, if the character sequence of the noun sequence in a chunk in \(H\) is identical to a subsequence of characters composing the noun sequence in a chunk in \(S\), the noun sequences in \(H\) and \(S\) are regarded as shared.

**Nominalization of verbs:** Many verbs in Japanese have corresponding nouns with similar meanings (e.g., the pair with # in Figures 1 and 2). Such pairs often share the same Chinese character, _kanji_. Kanji is an ideogram and is more informative than the other types of Japanese letters. Thus, if a noun in \(H\) and a verb in \(S^{2}\) share one kanji, the noun and the verb are regarded as shared.

### Transformation of a dependency tree

Some edges in dependency trees cannot be cut without changing the meaning or losing the grammaticality of the sentence. In Filippova et al.'s method, the nodes linked by such an edge are merged into a single node before extraction of the subtree. The method is adapted to Japanese as follows. If the function word of a chunk is one of the specific particles3, which often make obligatory cases, the chunk and its modifiee are merged into a single node. In Figure 1, the chunks at the start and end of a bold arrow are merged into a single node.

Footnote 3: “EUR (wo)”, “[“L� (ni)”, “?f� (ga)” and “lät (ha)”

### Extraction of the minimum rooted subtree

InFilippova et al.'s method, the minimum rooted subtree that contains all the shared content words is extracted from the transformed tree. We modify their method to take into account the characteristics (a) and (b).

**Deleting the global root:** In English, only verbs can be the root node of a subtree. However, in Japanese, words with other parts-of-speech can also be the root node in headlines (characteristics (b)). Therefore, the global root, which is the root node of \(S\), and the chunks including a word that can be located at the end of a sentence4, are the candidates for the root node of a subtree. Then, if the root node is not the global root, words succeeding the word that can be located at the end are removed from the root node. In Figure 1, among the two words with "*" that can be located at the

Figure 1: Dependency tree of \(S^{1}\). Squares and arrows respectively indicate chunks and dependency relations. Bold arrows mean that chunks are merged into a single node. Bold squares are extracted as \(C\).

 

end, the latter is extracted as the root, and the succeeding word is removed from the chunk.

**Reflecting abbreviated forms:** Abbreviation of nouns frequently occurs in Japanese (characteristic (a)). Thus, in \(C\), original forms are replaced by their abbreviated forms obtained as explained in Section 2.1 (e.g., the pair with "-" in Figures 1 and 2). However, we do not allow the head of a chunk to be deleted to keep the grammaticality. We also restrict here ourselves to word-level deletion and do not allow character-level deletion, because our purpose is to construct a training dataset for compression by word deletion. In the example of Figure 1, chunks in bold squares are extracted as \(C\).

### Conditions imposed on news articles

Filippova et al.'s method imposed eight conditions on news articles to restrict the training data to grammatical and informative sentences. In our method, these conditions are modified to adapt to Japanese. Firstly, the condition "\(S\) should include content words of \(H\) in the same order" is removed, because word order in Japanese is relatively free. Secondly, the condition "\(S\) should include all content words of _H_" is relaxed to "the ratio of shared content words to content words in \(H\) is larger than threshold \(\theta\)" because in Japanese, subjects and objects that can be easily estimated from the context are often omitted (characteristics (c)). In addition, two conditions "_H_ should have a verb" and "_H_ must not begin with a verb" are removed, leaving four conditions5.

Footnote 5: “_H_ is not a question”, “both \(H\) and \(S\) are not less than four words..”, ”S_ is more than 1.5 times longer than _H_”, and ”S_ is more than 1.5 times longer than _C_”.

## 3 Sentence compression with LSTM

Three models are used for sentence compression. Each model predicts a label sequence, where the label for each word is either "retain" or "delete" Filippova et al. (2015). The first is an encoder-decoder model with LSTM (_lstm_) Sutskever et al. (2014). Given an input sequence \({\bf x}=(x_{1},x_{2},\ldots,x_{n})\), \(y_{t}\) in label sequence \({\bf y}=(y_{1},y_{2},\ldots,y_{n})\) is computed as follows:

\[s_{0} =encoder({\bf x})\] \[(s_{t},m_{t}) =decoder(s_{t-1},m_{t-1},e_{1}(x_{t})\oplus e_{l}(y_{t-1}))\] \[y_{t} =softmax(W*s_{t}+b),\]

where \(\oplus\) indicates concatenation, \(s_{t}\) and \(m_{t}\) are respectively a hidden state and a memory cell at time \(t\), and \(e_{1}(\mathit{word})\) is embedding of \(word\). If \(label\) is "retain", \(e_{l}(label)\) is \((1;0)\); otherwise, \((0;1)\). \(m_{0}\) is a zero vector.

As the second and the third models, we extend the first model to control the output length Kikuchi et al. (2016). The second model, _lstm+leninit_, initializes the memory cell of the decoder as follows: \(m_{0}=tarlen*b_{len}\) where \(tarlen\) is the desired output length, and \(b_{len}\) is a trainable parameter. The third model, _lstm+lenemb_, uses the embedding of the potential desired length \(e_{2}(length)\) as an additional input. In this case, \(e_{1}(x_{t})\oplus e_{l}(y_{t-1})\oplus e_{2}(l_{t})\) is used as the input of the decoder where \(l_{t}\) is the potential desired length at time \(t\).

## 4 Experiments

The created training datasets were used to train three models for sentence compression. To see the effect of the modified subtree extraction method (Section 2.3), two training datasets were tested: _rooted_ and _multi-root+_. _rooted_ includes only deletion of the leafs in a dependency tree. In contrast, _multi-root+_ includes deleting the global root and reflecting abbreviated forms besides it.

**Setting:** Training datasets were created from seven million, 35-years' worth of news articles from the Mainichi, Nikkei, and Yomiuri newspapers, from which duplicate sentences and sentences in test data were filtered out. Gold-standard data were composed of the first sentences of the 1,000 news articles from Mainichi, each of which has 5 compressed sentences separately created by five human annotators. 100 sentences of the gold-standard data were used as development data, while the other sentences were used as test data. The three models were trained on datasets created with each value of threshold \(\theta\)6, which is the parameter used in the condition (introduced in Sec

\begin{table}
\begin{tabular}{l|r|r|r|r|r|r|r|r|r} threshold \(\theta\) & 0.1 & 0.2 & 0.3 & 0.4 & 0.5 & 0.6 & 0.7 & 0.8 & 0.9 & 1.0 \\ \hline size(10k) & 202 & 189 & 172 & 154 & 134 & 106 & 81 & 59 & 35 & 27 \\ vocab(k) & 105 & 102 & 98 & 94 & 90 & 81 & 72 & 62 & 48 & 42 \\ ROUGE-2 & 54.6 & 54.4 & 54.0 & 54.3 & 55.3 & 54.0 & 55.1 & 54.4 & 53.0 & 53.3 \\ \end{tabular}
\end{table}
Table 1: Created training data with each \(\theta\). ROUGE-2 is the score of compressed sentence generated by the model trained on each training data to target.

 

tion 2.4). The properties of the dataset in relation to \(\theta\) are shown in Table 1. \(\theta\) is tuned for ROUGE-2 score on the development data. In Table 1, we show the tendency of the ROUGE-2 scores when _lstm+leninit_ was trained on created _rooted_ with each \(\theta\). From Table 1, we chose 0.5 as \(\theta\). All models are three stacked LSTM layers7. Words with frequency lower than five are regarded as unknown words. ADAM8 was used as the optimization method. The desired length was set to the bytes of a compressed sentence randomly chosen from the five human-generated sentences. In the test step, beam-search was used (beam size: 20) and candidates exceeding the desired length were truncated.

Footnote 7: Dimension of word embedding and hidden layer:256, dropout:20%

**tree-base and prop-w-dpmd:** Existing methods for Japanese sentence compression are not based on the training on a large dataset. Therefore, the proposed method is compared with two methods, _tree-base_, Filippova and Strube [3] and _prop-w-dpnd_Harashima and Kuroshishi [9], which are not based on supervised learning. _tree-base_ is implemented as an integer linear programming problem that finds a subtree of a dependency tree. _prop-w-dpnd_ is also implemented as an integer linear programming problem, but modified based on characteristics of the Japanese language. _prop-w-dpnd_ allows the deletion inside the chunks.

### Automatic evaluation

Table 2 shows the ROUGE score of each model. We used ROUGE-1, ROUGE-2, and ROUGE-L (R-1, R-2, and R-L) for evaluation. In addition, we also show the precision and F-measure of ROUGE-2 because the output length of each model is not same. Compression ratio (CR) is the ratio of the bytes of the compressed sentence to the bytes of the source sentence. Average CR of the test data is approximately 61.0%. We think we should focus on F-measure of ROUGE-2 because of the different CR of the models. The models trained on a large training dataset achieved higher F-measure than the unsupervised models. Moreover, F-measure of _lstm+leninit_ and _lstm+lenemb_, which were trained on either _multi-root_+ or _rooted_, is significantly better than _prop-w-dpnd_ (p \(<\) 0.001). _lstm_ achieved lower R-1 and R-2 than the other models trained on a large training dataset, probably because it tends to generate too short sentences, as indicated by low CR. It is also noteworthy that CRs of _lstm+leninit_ and _lstm+lenemb_ are mostly closer to the average CR of the test data than _lstm_. Furthermore, _lstm+leninit_ and _lstm+lenembl_ trained on _multi-root_+ instead of _rooted_ worked better in terms of F-measure. We consider it is because various types of deletion make the compression model more flexible, as indicated by closer CR of the model trained on _multi-root_+ instead of _rooted_ to the average CR of the test data.

### Human evaluation

The difference between _lstm+leninit_ and _lstm+lenemb_ trained on _multi-root_+ was investigated first. With _lstm+leninit_, 2 out of 100 sentences, chosen randomly, ended with a word that cannot be located at the end of a sentence. In contrast, with _lstm+lenemb_, 24 sentences ended with such words and therefore are ungrammatical, although _lenemb_ has shown to be effective in abstractive sentence summarization Kikuchi et al. [11]. This result suggests that _lstm+lenemb_ is excessively affected by the desired length because _lenemb_ receives the potential desired length at each time of decoding. In fact, 21 out of the 24 sentences are as long as the desired length.

Then, _lstm+leninit_ trained on _multi-root_+ was evaluated by crowdsourcing in comparison with the gold-standard and _tree-base_. Each crowd

\begin{table}
\begin{tabular}{l|l|c|c c c|c|c} \hline \multirow{2}{*}{dataset} & \multicolumn{1}{c|}{model} & \multicolumn{3}{c|}{ROUGE-2} & \multirow{2}{*}{R-L} & \multirow{2}{*}{CR} \\ \cline{3-3} \cline{5-8}  & & & \multicolumn{1}{c|}{R(R-2)} & & P & \multicolumn{1}{c|}{F} \\ \hline \hline  & _prop-w-dpnd_ & 73.8 & 56.5 & 51.5 & 53.7 & 32.7 & 60.5 \\  & _tree-base_ & 68.6 & 52.6 & 50.4 & 51.4 & 34.4 & 59.1 \\ \hline _rooted_ & _lstm_ & 66.3 & 51.5 & 59.3 & 54.6 & 36.1 & 51.8 \\ _rooted_ & _lstm+leninit_ & 70.8 & 55.2 & 57.1 & 55.9 & 36.0 & 56.8 \\ _rooted_ & _lstm+lenemb_ & 71.0 & 55.2 & 56.7 & 55.7 & 35.9 & 57.8 \\ \hline _multi-root+_ & _lstm_ & 60.8 & 50.6 & 60.5 & 54.1 & 33.7 & 47.9 \\ _multi-root+_ & _lstm+leninit_ & 71.7 & 56.0 & 57.8 & 56.7 & 36.3 & 57.8 \\ _multi-root+_ & _lstm+lenemble_ & 72.6 & 56.2 & 56.4 & 56.1 & 35.8 & 59.4 \\ \hline \end{tabular}
\end{table}
Table 2: Automatic evaluation. R-1, R-2, and R-L are ROUGE-1, ROUGE-2, and ROUGE-L score. R, P and F are recall, precision and F-measure.

 

sourcing worker reads each source sentence and a set of the compressed sentences, reordered randomly, and gives a score from 1 to 5 (where 5 is best) to each compressed sentence in terms of informativeness (_info_) and readability (_read_). As shown in Table 3, in terms of both _read_ and _info_, _lstm+leninit_ archived higher scores than _prop-w-dpmd_, and the difference is significant (p \(<0.001\)).

Next, _lstm+leninit_ was compared with _lstm_ (both are trained on _multi-root_+), and _multi-root_+ was compared with _rooted_ (_lstm+leninit_ was used as the model), by human relative evaluation. In this evaluation, each worker votes for one of _lstm+leninit_, _lstm_, or tie, and for one of _multi-root_+, _rooted_, or tie. 250 votes in total were received (50 sentences \(\times\) 5 votes). The results are shown in Table 4. _lstm+leninit_ is better than _lstm_ in terms of _read_, which is not directly related to the output length, as well as of _info_. It is also clear that _multi-root_+ achieves much higher _info_ with a negligible reduction in _read_ than _rooted_.

## 5 Conclusion

Filippova et al.'s method of creating training data for English sentence compression was modified to create a training dataset for Japanese sentence compression. The effectiveness of the created Japanese training dataset was verified by automatic and human evaluation. Our method can refine the created dataset by giving more flexibility in compression and achieved better informativeness with negligible reduction in readability. Furthermore, it has been shown that controlling the output length improves the performance of the sentence compression models.

## Acknowledgment

This work was supported by Google Research Award, JSPS KAKENHI Grant Number JP26280080, and JPMJPR1655.

## References

* Volume 1_. pages 481-490.
* Clarke and Lapata (2006) James Clarke and Mirella Lapata. 2006. Models for sentence compression: A comparison across domains, training requirements and evaluation measures. In _Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics_. pages 377-384.
* Filippova et al. (2015) Katja Filippova, Enrique Alfonseca, Carlos A. Colmenares, Lukasz Kaiser, and Oriol Vinyals. 2015. Sentence compression by deletion with lstms. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. pages 360-368.
* Filippova and Altun (2013) Katja Filippova and Yasemin Altun. 2013. Overcoming the lack of parallel data in sentence compression. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. pages 1481-1491.
* Filippova and Strube (2008) Katja Filippova and Michael Strube. 2008. Dependency tree based sentence compression. In _Proceedings of the Fifth International Natural Language Generation Conference_. pages 25-32.
* Harashima and Kurohashi (2012) Jun Harashima and Sadao Kurohashi. 2012. Flexible Japanese sentence compression by relaxing unit constraints. In _Proceedings of COLING 2012_. pages 1097-1112.
* Hirao et al. (2009) Tsutomu Hirao, Jun Suzuki, and Hideki Isozaki. 2009. A syntax-free approach to japanese sentence compression. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_. pages 826-833.
* Hori and Furui (2004) Chiori Hori and Sadaoki Furui. 2004. Speech summarization : An approach through word extraction and a method for evaluation (!special section?the 2002 ieee excellent paper award). _IEICE transactions on information and systems_ pages 15-25.
* Kikuchi et al. (2016) Yuta Kikuchi, Graham Neubig, Rojhei Sasano, Hiroya Takamura, and Manabu Okumura. 2016. Controlling output length in neural encoder-decoders. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. pages 1328-1338.
* step one: Sentence compression. In _Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence_. pages 703-710.

\begin{table}
\begin{tabular}{l|c c} \hline model & _read_ & _info_ \\ \hline \hline gold-standard & 4.22 & 3.76 \\ _lstm+leninit_ & 3.99 & 3.09 \\ _prop-w-dpmd_ & 3.55 & 2.81 \\ \hline \end{tabular}
\end{table}
Table 3: Human absolute evaluation 

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Proceedings of the 27th International Conference on Neural Information Processing Systems_. pages 3104-3112.
* Thadani and McKeown (2013) Kapil Thadani and Kathleen McKeown. 2013. Sentence compression with joint structural inference. In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning_. pages 65-74.
* Turner and Charniak (2005) Jenine Turner and Eugene Charniak. 2005. Supervised and unsupervised learning for sentence compression. In _Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics_. pages 290-297.