Venue: ICLR 2018 conference

Reviews:

final_score: rating: null
review_content: How are you training the local context units? Which loss are you optimizing to get the embeddings E and context parameters U?
review_title: Please clarify loss

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a mechanism for learning task-specific region embeddings for use in text classification. Specifically, this comprises a standard word embedding an accompanying local context embedding. 

The key idea here is the introduction of a (h x c x v) tensor K, where h is the embedding dim (same as the word embedding size), c is a fixed window size around a target word, and v is the vocabulary size. Each word in v is then associated with an (h x c) matrix that is meant to encode how it affects nearby words, in particular this may be viewed as parameterizing a projection to be applied to surrounding word embeddings. The authors propose two specific variants of this approach, which combine the K matrix and constituent word embeddings (in a given region) in different ways. Region embeddings are then composed (summed) and fed through a standard model. 

Strong points
---
+ The proposed approach is simple and largely intuitive: essentially the context matrix allows word-specific contextualization. Further, the work is clearly presented.

+ At the very least the model does seem comparable in performance to various recent methods (as per Table 2), however as noted below the gains are marginal and I have some questions on the setup.

+ The authors perform ablation experiments, which are always nice to see. 

Weak points
---
- I have a critical question for clarification in the experiments. The authors write 'Optimal hyperparameters are tuned with 10% of the training set on Yelp Review Full dataset, and identical hyperparameters are applied to all datasets' -- is this true for *all* models, or only the proposed approach? 

- The gains here appear to be consistent, but they seem marginal. The biggest gain achieved over all datasets is apparently .7, and most of the time the model very narrowly performs better (.2-.4 range). Moreoever, it is not clear if these results are averaged over multiple runs of SGD or not (variation due to initialization and stochastic estimation can account for up to 1 point in variance -- see "A sensitivity analysis of (and practitioners guide to) CNNs..." Zhang and Wallace, 2015.)

- The related work section seems light. For instance, there is no discussion at all of LSTMs and their application to text classificatio (e.g., Tang et al., EMNLP 2015) -- although it is noted that the authors do compare against D-LSTM,  or char-level CNNs for the same (see Zhang et al., NIPs 2015). Other relevant work not discussed includes Iyyer et al. (ACL 2015). In their respective ways, these papers address some of the same issues the authors consider here. 

- The two approaches to inducing the final region embedding (word-context and then context-word in sections 3.2 and 3.3, respectively) feel a bit ad-hoc. I would have appreciated more intuition behind these approaches. 

Small comments
---
There is a typo in Figure 4 -- "Howerver" should be "However"

*** Update after author response ***

Thanks to the authors for their responses. My score is unchanged.
review_title: Review of "Bag of region embeddings ..."

final_score: rating: 6: Marginally above acceptance threshold
review_content: () Summary
In this paper, the authors introduced a new simple model for text classification, which obtains state of the art results on several benchmark. The main contribution of the paper is to propose a new technique to learn vector representation of fixed-size text regions of up to a few words. In addition to learning a vector for each word of the vocabulary, the authors propose to also learn a "context unit" of size d x K, where d is the embedding size and K the region size. Thus, the model also have a vector representation for pair of word and position in the region. Then, given a region of K words, its vector representation is obtained by taking the elementwise product of the "context unit" of the middle word and the matrix obtained by concatenating the K vectors of words appearing in the region (the authors also propose a second model where the role of word vectors and "context" vectors are exchanged). The max-pooling operation is then used to obtain a vector representation of size d. Then a linear classifier is applied on top of the sum of the region embeddings. The authors then compare their approach to previous work on the 8 datasets introduced by Zhang et al. (2015). They obtain state of the art results on most of the datasets. They also perform some analysis of their models, such as the influence of the region size, embedding size, or replacing the "context units" vector by a scalar. The authors also provide some visualisation of the parameters of their model.

() Discussion
Overall, I think that the proposed method is sound and well justified. The empirical evaluations, analysis and comparisons to existing methods are well executed. I liked the fact that the proposed model is very simple, yet very competitive compared to the state-of-the-art. I suspect that the model is also computationally efficient: can the authors report training time for different datasets? I think that it would make the paper stronger. One of the main limitations of the model, as stated by the authors, is its number of parameters. Could the authors also report these?

While the paper is fairly easy to read (because the method is simple and Figure 1 helps understanding the model), I think that copy editing is needed. Indeed, the papers contains many typos (I have listed a few), as well as ungrammatical sentences. I also think that a discussion of the "attention is all you need" paper by Vaswani et al. is needed, as both articles seem strongly related.

As a minor comment, I advise the authors to use a different letter for "word embeddings" and the "projected word embeddings" (equation at the bottom of page 3). It would also make the paper more clear.

() Pros / Cons:
+ simple yet powerful method for text classification
+ strong experimental results
+ ablation study / analysis of influence of parameters
- writing of the paper
- missing discussion to the "attention is all you need paper", which seems highly relevant

() Typos:
Page 1
"a support vectors machineS" -> "a support vector machine"
"performs good" -> "performs well"
"the n-grams was widely" -> "n-grams were widely"
"to apply large region size" -> "to apply to large region size"
"are trained separately" -> "do not share parameters"

Page 2
"convolutional neural networks(CNN)" -> "convolutional neural networks (CNN)"
"related works" -> "related work"
"effective in Wang and Manning" -> "effective by Wang and Manning"
"applied on text classification" -> "applied to text classification"
"shard(word independent)" -> "shard (word independent)"

Page 3
"can be treat" -> "can be treated"
"fixed length continues subsequence" -> "fixed length contiguous subsequence"
"w_i stands for the" -> "w_i standing for the"
"which both the unit" -> "where both the unit"
"in vocabulary" -> "in the vocabulary"

etc...
review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present a model for text classification. The parameters of the model are an embedding for each word and a local context unit. The local context unit can be seen as a filter for a convolutional layer, but which filter is used at location i depends on the word at location i (i.e. there is one filter per vocabulary word). After the filter is applied to the embeddings and after max pooling, the word-context region embeddings are summed and fed into a neural network for the classification task. The embeddings, the context units and the neural net parameters are trained jointly on a supervised text classification task. The authors also offer an alternative model, which changes the role of the embedding an the context unit, and results in context-word region embeddings. Here the embedding of word i is combined with the elements of the context units of words in the context. To get the region embeddings both model (word-context and context-word) combine attributes of the words (embeddings) with how their attributes should be emphasized or deemphasized based on nearby words (local context units and max pooling) while taking into account the relative position of the words in the context (columns of the context units). 

The method beats existing methods for text classification including d-LSTMs , BoWs, and ngram TFIDFs on held out classification accuracy. the choice of baselines is convincing. What is the performance of the proposed method if the embeddings are initialized to pretrained word embeddings and a) trained for the classification task together with randomly initialized context units b) frozen to pretrained embeddings and only the context units are trained for the classification task?

The introduction was fine. Until page 3 the authors refer to the context units a couple of times without giving some simple explanation of what it could be. A simple explanation in the introduction would improve the writing.
The related work section only makes sense *after* there is at least a minimal explanation of what the local context units do. A simple explanation of the method, for example in the introduction, would then make the connections to CNNs more clear. Also, in the related work, the authors could include more citations (e.g. the d-LSTM and the CNN based methods from Table 2) and explain the qualitative differences between their method and existing ones.

The authors should consider adding equation numbers. The equation on the bottom of page 3 is fine, but the expressions in 3.2 and 3.3 are weird. A more concise explanation of the context-word region embeddings and the word-context region embeddings would be to instead give the equation for r_{i,c}.  

The included baselines are extensive and the proposed method outperforms existing methods on most datasets. In section 4.5 the authors analyze region and embedding size, which are good analyses to include in the paper. Figure 2 and 3 could be next to each other to save space. 
I found the idea of multi region sizes interesting, but no description is given on how exactly they are combined. Since it works so well, maybe it could be promoted into the method section? Also, for each data set, which region size worked best?

Qualitative analysis: It would have been nice to see some analysis of whether the learned embeddings capture semantic similarities, both at the embedding level and at the region level. It would also be interesting to investigate the columns of the context units, with different columns somehow capturing the importance of relative position. Are there some words for which all columns are similar meaning that their position is less relevant in how they affect nearby words? And then for other words with variation along the columns of the context units, do their context units modulate the embedding more when they are closer or further away? 

Pros:
 + simple model
 + strong quantitative results

Cons:
 - notation (i.e. precise definition of r_{i,c})
 - qualitative analysis could be extended
 - writing could be improved  
review_title: New model for text classification

Meta-review: despite not amazing scores, this is a solid paper.
it created a lot of discussion and was found to be reproducible.
we should accept it to let the iclr community partake in the discussion and learn about this method of n-gram embeddings

Decision: Accept (Poster)