Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a method for learning word embeddings with data-dependent dimensionality. Different words have different numbers of non-zero dimensions, which allows for more or less information to be stored in a given word embedding. The authors argue that this is important to reduce various kinds of over-fitting, which seems to make sense, at least intuitively.

I like the general motivation and idea, but I don't entirely understand how the setup achieves the desired goal. If I understand correctly, the dot product between two embeddings has non-zero contributions for the first l_min components of the two embeddings, where l_min is the smaller of the two vector dimensionalities. This means that *both* vectors must be high-dimensional in order for the increased capacity of a one of the vectors to manifest. This property seems at odds with the information-storage argument given in the paper, which seems to imply that the dimensionality of a *single* embedding is all that is important. 

I also think that the title and much of the wording in the paper is misleading insofar as they suggest that the embeddings may be infinite-dimensional. Restricting "the model to grow only one dimension at a time" necessarily enforces that the embeddings remain finite-dimensional (given that there are a finite-number of time steps), so it seems like the entire analysis should be able to be recast in a finite-dimensional setting without loss of generality. In doing so, I think this would eliminate the need to consider how to account for divergences in the infinite-dimensional setting, and largely reduce the novelty of the proposed algorithm.

I don't find the empirical evidence particularly convincing of the usefulness of the proposed approach.  All-in-all, while the idea is quite interesting, I think this paper lacks the theoretical justification and empirical basis to merit its acceptance as a workshop paper. 
review_title: Interesting idea, but not exactly infinite-dimensional

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a method to learn infinite dimensional word embeddings, based on the infinite RMB method of Cote and Larochelle (2016). 
The paper applies the technique to learn infinite dimension Continuous Bag of Words and Skip Gram models, and show results on word similarity datasets.

I think this is not a trivial extension of the original infinite RMB method.
While the quantitative results are not very convincing (much worse than 200D CBOW and SG), it is a reasonably good workshop paper that offers interesting insights on how a word embedding model uses its dimensions to represent words with multiple senses.
review_title: review

Decision: Reject