Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 
The paper proposes a novel method for embedding numerals which can be learned by using neural word embedding learning techniques. The paper motivates the work by reviewing the difficulty of embedding components to represent numerals: OOV in most cases. Their main contribution is the introduction of a method composes numeral embedding by a weighted average of prototype embeddings based on the similarities between the numeral and prototypes. There are two proposed prototypes: SOM and GMM and the similarity functions are an absolute difference and the density function respectively. During the training, the numerals have the proposed embeddings while the others have normal word embeddings. The paper slightly modifies the negative sampling to ensure numerals being sampled. A series of 4 empirical studies have been presented. First, the paper confirms that the proposed method does not negatively affect non-numeral embeddings. And then, the quality of the numeral embeddings are evaluated and compared. The experiments show that the proposed method has better performance on numerical property tests, numeral prediction, and a sequence labeling task.

Overall, this paper has a novel contribution. The proposed method is well motivated and quite justified by the experiments abliet lacking comparison with previous published results. However, it has some weaknesses.

For the method part, one of the limitations is that it is not an end-to-end method and requires a regular expression to identify the numeral. Second, the weighted average of the prototypes is reasonable, but the similarity function only relies on the magnitude. I think there are other aspects of numerical tokens that it might not be able to capture (e.g. “2019” is similar to “19” in some context). In terms of training, I think it is not hard to extend the method to full language model training (using softmax). However, adding all numerals to the vocabulary would add significant overhead.

For the experiments, some design decisions are left unjustified. For example, a simple ablation experiment on the squashing function will be helpful. Furthermore, guidelines or empirical results on the effect of the number of prototypes can increase the impact of the paper. Finally, I think an analysis of the performance of numerical types will be helpful for future works (e.g., dates, phone numbers, currencies, etc, or discrete vs continuous). 

Minor comments and questions:
1. The log sigmoid in equation 6 is a bit strange, isn’t it log sum exp(). https://arxiv.org/pdf/1402.3722v1.pdf
2. Why do you create a new dataset for the experiment in section 4.3?
3. In section 4.4, you rank only numerals in the test set, but the scores are computed based on all numerals in the vocab. Do you have the performance of ranking all numerals?
4. Just to confirm the “training” in section 4.4 refers to learning the embedding using the skip-gram model, right?

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: I have read the author response.  Thank you for responding to my concerns.

Original review:
This paper presents a word embedding approach for numbers.  The method is based on finding prototype numbers, and then representing numbers as a weighted average of the prototype embeddings, where the weights are based on numeric proximity.  The approach provides some gains over baselines on number similarity, number prediction, and sequence tagging tasks.  While modeling numbers is an interesting task, several aspects of the paper needed more clarification, and the paper’s focus may be somewhat narrow for the ICLR audience.

I think the paper could use a better motivation for the prototype-based approach.  In particular, the fact that the approach uses only quantity (and not the form of numbers) to represent similarity is contrary to my intuition.  For example, I would expect 1960 and 1960.1 to behave very differently in text, because one is a year and the other isn’t.  But the proposed method, if I understand it correctly, would give them similar embeddings because they are very close numerically.

I was not able to understand the SOM portion of the method, it is not self-contained within this paper.

On the Numeracy-600K data set, Chen et al. (2019) shows much higher F1 results than those shown here.  What explains this difference?

The proposed method seems relatively similar to that of (Spithourakis & Riedel, 2018), in that it exposes numeric quantity to the language/embedding model, and uses GMMs to represent numeral distributions.  More clarity about how this approach compares to that one (and others) would be helpful.  Also, how does the (Spithourakis & Riedel, 2018) approach fare on e.g. the number prediction tasks in the submission?  It is true that the submission's approach produces general-purpose embeddings that can be re-used, unlike (Spithourakis & Riedel, 2018).  But we would like to know whether that generality comes at the cost of performance on tasks, and if so how much of a cost.

Minor:
The Lund and Burgess reference seems incorrect.  I don’t see that those authors published a paper by that title in Brain and Cognition in 1996.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
Summary:
The paper talks about a recently highlighted problem in word embeddings which is their incapability to represent numerals, especially the out-of-vocabulary numerals. For addressing the problem, they propose a method that induces a finite set of prototype numerals using either self-organizing map or Gaussian Mixture model. Then, each numeral is represented as a weighted average of prototype numeral embeddings. The method also involves squashing large quantities using log function. Finally, the training is performed similar to Skip-gram in word2vec but with the embedding of numerals computed using prototype numerals. 

Questions:
1. There are two basic motivations of the paper: (1) Most of the word embedding methods do not embed numerals correctly. (2) There is no mechanism of handing OOV numerals. The second one is well addressed but for the former one, it has been shown in recent work [1] that most of the embedding methods do have numerical reasoning capabilities. As stated in [1], the results of [2] demonstrate the opposite conclusion because their analysis is based on cosine distance and nearest neighbor which are not capable of capturing non-linear dependencies between embeddings. 

So, it would be great if, for the results in Section 4.3 instead of using cosine distance, some neural models could be utilized for evaluation (3 layer MLP similar to [1]).

2. In Section 4.4, the task is to predict the target numeral given its context words (similar to CBOW) while the embeddings are trained with a modified skip-gram model. Can one expect superior results if one uses modified CBOW for training embeddings rather than skip-gram?

3. In Table 5, with 100% training data, the performance of all the methods is very close. It would be better if mean and variance across multiple runs are reported. 
review_title: Official Blind Review #2

Meta-review: This paper proposes better methods to handle numerals within word embeddings.

Overall, my impression is that this paper is solid, but not super-exciting. The scope is a little bit limited (to only numbers), and it is not by any means the first paper to handle understanding numbers within word embeddings. A more thorough theoretical and empirical comparison to other methods, e.g. Spithourakis & Riedel (2018) and Chen et al. (2019), could bring the paper a long way.

I think this paper is somewhat borderline, but am recommending not to accept because I feel that the paper could be greatly improved by making the above-mentioned comparisons more complete, and thus this could find a better place as a better paper in a new venue.
Decision: Reject