Agreement on Target-Bidirectional LSTMs for Sequence-to-Sequence Learning

Recurrent neural networks, particularly the long short-term memory networks, are extremely appealing for sequence-tosequence learning tasks. Despite their great success, they typically suffer from a fundamental shortcoming: they are prone to generate unbalanced targets with good prefixes but bad suffixes, and thus performance suffers when dealing with long sequences. We propose a simple yet effective approach to overcome this shortcoming. Our approach relies on the agreement between a pair of target-directional LSTMs, which generates more balanced targets. In addition, we develop two efficient approximate search methods for agreement that are empirically shown to be almost optimal in terms of sequence-level losses. Extensive experiments were performed on two standard sequence-to-sequence transduction tasks: machine transliteration and grapheme-to-phoneme transformation. The results show that the proposed approach achieves consistent and substantial improvements, compared to six state-of-the-art systems. In particular, our approach outperforms the best reported error rates by a margin (up to 9% relative gains) on the grapheme-to-phoneme task. Our toolkit is publicly available on https://github.com/lemaoliu/Agtarbidir. Recurrent neural networks (RNNs) (Mikolov et al. 2010), particularly Long Short-term Memory networks (LSTMs)1 (Hochreiter and Schmidhuber 1997; Graves 2013), provide a universal and powerful solution for various tasks that have traditionally required carefully designed, task-specific solutions. On classification tasks (Graves and Schmidhuber 2008; Tai, Socher, and Manning 2015), they can readily summarize an unbounded context which is difficult for tranditional solutions, and this leads to more reliable prediciton. They have advantages over traditional solutions on a more general and challenging tasks such as sequence-to-sequence learning (Sutskever, Vinyals, and Le 2014), where a series of local but dependent predictions are required. RNNs make use of the contextual information for the entire source sequence and also critically are able to exploit the entire sequence of previous predictions. On various sequence-tosequence transduction tasks, RNNs have been shown to be comparable to the state-of-the-art (Bahdanau, Cho, and BenCopyright c  Throughout this paper, an LSTM network denote a particular RNN with LSTM hidden units.  Illustraction of the fundamental shortcoming of an LSTM in decoding. gio 2015; Meng et al. 2015) or superior (Jean et al. 2015; Luong et al. 2015). Despite their sucesses on sequence-to-sequnce learning, RNNs suffer from a fundamental and crucial shortcoming, which has surprisingly been overlooked. When making predictions (in decoding), an LSTM needs to encode the previous local predictions as a part of the contextual information. If some of previous predictions are incorrect, the context for subsequent predictions might include some noises, which undermine the quality of subsequent predicitons, as shown in Figure 1. In the figure, larger fonts indicate greater confidence in the predicted target character. The prediction at t = 7 uses a context consisting of the input and all previous predictions. Since at t = 5 the prediction is incorrect, i.e. it should be ‘R’ (the green character in the reference) instead of ‘L’, it leads to an incorrect prediction at t = 7. In this way, an LSTM is more likely to generate an unbalanced sequence deteriorating in quality as the target sequence is generated. A statistical analysis on the real prediction results from an LSTM was performed in order to motivate the work reported here. The analysis supports our hypothesis, and found that on test examples longer than 10 characters, the precision of predictions for the first two characters was higher than 77%, Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-16)