Title: Character-aware Attention Residual Network for Sentence Representation
Authors: Xin Zheng
Zhenzhou Wu
Venue: ICLR 2017
Keywords: Deep learning

Abstract: Text classification in general is a well studied area. However, classifying short and noisy text remains challenging. Feature sparsity is a major issue. The quality of document representation here has a great impact on the classification accuracy. Existing methods represent text using bag-of-word model, with TFIDF or other weighting schemes. Recently word embedding and even document embedding are proposed to represent text. The purpose is to capture features at both word level and sentence level. However, the character level information are usually ignored. In this paper, we take word morphology and word semantic meaning into consideration, which are represented by character-aware embedding and word distributed embedding. By concatenating both character-level and word distributed embedding together and arranging words in order, a sentence representation matrix could be obtained. To overcome data sparsity problem of short text, sentence representation vector is then derived based on different views from sentence representation matrix. The various views contributes to the construction of an enriched sentence embedding. We employ a residual network on the sentence embedding to get a consistent and refined sentence representation. Evaluated on a few short text datasets, our model outperforms state-of-the-art models.

TL;DR: We propose a character-aware attention residual network for short text representation.

Review discussion:
Sentiment analysis is one of most popular tasks on text classification. It would be interesting to see the evaluation of the proposed model on some sentence-level datasets, for example, the Stanford Sentiment Treebank. Is there any reason that kind of experiment was not included?
Thanks for the feedback. We have tried on a tweeter dataset that is almost sentence level and similar to stanford sentiment treebank. It's definitely interesting to test on treebank as well, we will improve on that.
In the meantime, for short text, you can refer to other dataset's results. Thanks.
This paper proposes a character-aware attention residual network for sentence embedding. Several text classification tasks are used to evaluate the effectiveness of the proposed model. On two of the three tasks, the residual network outforms a few baselines, but couldn't beat the simple TFIDF-SVM on the last one.
This work is not novel enough. Character information has been applied in many previously published work, as cited by the authors. Residual network is also not new.
Why not testing the model on a few more widely used datasets for short text classification, such as TREC? More competitive baselines can be compared to.
Also, it's not clear how the "Question" dataset was created and which domain it is.
Last, it is surprising that the format of citations throughout the paper is all wrong.
For example:
like Word2Vec Mikolov et al. (2013)
like Word2Vec (Mikolov et al., 2013)
The citations can't just mix with the normal text. Please refer to other published papers.
Thanks for your comments!
1.	Character-level embedding has been utilized by many works before as we mentioned in related work.
However we find that by combining character-level embedding and word-level embedding could capture more information for short noisy text and would improve the classification performance. The contribution in this paper is more on the two types of features and the first application of residual network on text representation refinement. There are less information in short noisy text than the other long text and not many works focus on short noisy text classification. The two types of features proposed in our paper could capture different aspect of information in the text and leads to good performance. Residual network could further help refine the short text representation and give better result.
2.
In fact, TFIDF-SVM (linear kernel) performs quite well on text classification and it is hard to beat it. The last dataset (AG_news) is typical well-formatted relatively long documents and TFIDF-SVM has great advantage to achieve good performance. Experiment results from (Zhang et al. 2015) also suggest that the TFIDF weighting perform best on three of their testing datasets. Compared with results on ag_news, our model achieves better performance on short and noisy tweets and question datasets than all the other baselines.
This paper proposes a new neural network model for sentence representation. This new model is inspired by the success of residual network in Computer Vision and some observation of word morphology in Natural Language Processing.
Although this paper shows that this new model could give the best results on several datasets, it lacks a strong evidence/intuition/motivation to support the network architecture.
To be specific:
- I was confused by the contribution of this paper: character-aware word embedding or residual network or both?
- The claim of using residual network in section 3.3 seems pretty thin, since it ignores some fundamental difference between image representation and sentence representation.
Even though the results show that adding residual network could help, I was still not be convinced. Is there any explanation about what is captured in the residual component from the perspective of sentence modeling?
- This paper combines several components in the classification framework, including character-aware model for word embedding, residual network and attention weight in Type 1 feature. I would like to see the contribution from each of them to the final performance, while in Table 3 I only saw one of them. Is it possible to add more results on the ablation test?
- In equation (5), what is the meaning of $i$ in $G_i$?
- The citation format is impropriate
Thanks for your comments!
1.	Character-aware word embedding has been utilized by many works before as we mentioned in related work.
However we find that by combining character-level embedding and word-level embedding could capture more information for short noisy text and would improve the classification performance. The contribution in this paper is more on the two types of features, and the first application of residual network on text representation refinement. There are less information in short noisy text than the other long text and not many works focus on short noisy text classification. The two types of features proposed in our paper could capture different aspects of information in the text and lead to good performance. Residual network could further help refine the short text representation and give better result.
2.	As stated in paper, the short text final representation is the concatenation of two types of features which capture different aspects of information and of different scale. To make the representation more consistent, we apply residual network to refine short text final representation.
3.	We add the evaluation for the character-level word embedding and the attention weight for Type 1 feature. Experiment results suggest either part could contribute to better performance.
4.	Sorry for the typo. It should be $G$.
5.	Thanks for the kind reminding on the citation format. We make it the appropriate way.
This paper proposes a new model for sentence classification.
Pros:
- Some interesting architecture choices in the network.
Cons:
- No evaluation of the architecture choices. An ablation study is critical here to understand what is important and what is not.
- No evaluation on standard datasets. On the only pre-existing dataset evaluated on a simple TFIDF-SVM method is state-of-the-art, so results are unconvincing.
Thanks for your comments!
1.	We add the evaluation of weight assigned by attention model for type 1 feature and the character embedding part.
Thus, each part of the architecture has an evaluation. Experiment results suggest that removing each part will lead to worse performance, especially for type 1 and type 2 feature.
Thus, each part of the architecture do contribute to the final performance, and type 1 and type 2 feature are more important.
2.	Twitter data is typical short noisy text and we crawl tweets by ourselves as evaluation data. For text classification, TFIDF weighted feature with SVM (linear kernel) performs quite well and it is hard to beat it. Experiment results from (Zhang et al. 2015) also suggest that the TFIDF weighting perform best on three of their testing datasets.
Besides, we also compare with the work from (Zhang et al. 2015) which is one of the state-of-the-art.

Meta-review:
role: PC
data: comment: The paper introduces some interesting architectural ideas for character-aware sequence modelling. However, as pointed out by reviewers and from my own reading of the paper, this paper fails badly on the evaluation front. First, some of the evaluation tasks are poorly defined (e.g. question task). Second, the tasks look fairly simple, whereas there are "standard" tasks such as language modelling datasets (one of the reviewers suggests TREC, but other datasets such as NANT, PTB, or even the Billion Word Corpus) which could be used here. Finally, the benchmarks presented against are weak. There are several character-aware language models which obtain robust results on LM data which could readily be adapted to sentence representation learning, eg. Ling et al. 2016, or Chung et al. 2016, which should have been compared against. The authors should look at the evaluations in these papers and consider them for a future version of this paper. As it stands, I cannot recommend acceptance in its current form.

Decision: Reject