Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors proposed FIRE (FIeld REepresentation), in which each word is represented by a pair of points in the embedding space (word sense) and scalar fields in the embedding space (contextual information). Polysemy is naturally represented, and sentence representations can be computed directly from the representations of the constituent words (Eq. 4).

Experiments were conducted on the tasks of word similarity, surrounding context similarity, and estimation of the number of word senses. The proposed method was found to be effective to a certain degree.
strengths_and_weaknesses: Strengths

* The attempt to express the meaning of a word as a continuous function on support is both novel and fascinating.

Weaknesses

* Unfortunately, this paper is ill-organized and costly to read.
* The experimental setup is limited, and it is difficult to say that the value of the proposed method has been evaluated. The task used in sentence similarity is not standard (the standard one is STS), and the baselines used are only raw word2vec and BERT. The same problem could be seen in word sense experiments.
* The references and allusions of related work are inconsistent. For example, [21] is one of the most directly related studies because it represents words as a point cloud (polysemy) and proposes a compositional method for sentence representation using barycenter (compositionality; line 64). But this method is not compared in the paper (line 73). In contrast, word2vec [13] also computes sentence representations in terms of barycenter of word vectors and is always compared with the proposed method.
questions: I found both the statement that polysemy is represented by a point cloud (line 34) and the statement that polysemy can be represented by a potential function (e.g., Figure 1). Which is the intent of the authors?
limitations: This would be one of the limitations if the proposed method does not scale to practical dimensions such as 300 or 768. This is also mentioned by the authors in the checklist.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to model polysemy by modeling a word as a tuple
consisting of a set of locations in a field and a nonlinear function.
Compared to other methods for modeling polysemy, this approach also
provides a mechanism for composing word representations. The authors
show that the representations can be meaningfully composed, and that
the method outperforms baselines on word-sense prediction tasks.
strengths_and_weaknesses: **Strengths**
1. This paper proposes an interesting and unusual new perspective to
   the task of modeling polysemy. I am not aware of prior work using
   this kind of approach, and I find it to be thought-provoking and
   think it could be of general interest.

2. The paper is written in a straightforward style and is easy to
   read.

3. The experiments illustrate that the resulting embeddings capture
   word similarity, polysemy, and compositionality. In particular, I
   find Figures 1 and 3 to be interesting, illustrative examples of
   the type of representation provided by this approach.

**Weaknesses**
1. I am not entirely convinced by the motivation for this approach.
   The authors write that "The two most important semantic
   requirements for a word representation are compositionality [...]
   and polysemy" (line 15). I think many readers would dispute this
   claim: the most important requirement for a word representation is
   that similar words are mapped to similar embeddings, which is the
   property that enables generalization from small datasets.
   (The method is also evaluated on semantic similarity in section 6.)
   
   More to the point, it is unclear to me why we would want to compose
   polysemous word representations. While I find the problem to be
   interesting, it is hard for me to imagine a practical setting where
   it would be desirable to compose the representations for "river"
   and "mortgage," or to compose the word "bank" without first disambiguating the intended sense.
   
2. Evaluating polysemy (section 8): Is there any reason that the
   method is not compared to prior method for polysemous word
   representations?
   
3. DBSCAN algorithm: The DBSCAN algorithm clusters words using the
   Euclidean distance. This seems somewhat inconsistent with the
   similarity function proposed in this paper. It might be interesting
   to use a clustering method that is based on the proposed similarity
   function, e.g. a spectral clustering algorithm.
   
4. How is the number of word locations chosen? Compared to
   probabilistic methods, it seems less straightforward to determine
   the number of locations per-word from the data.

**Minor comments**
1. It would be helpful to include additional references
   on fields and potentials, including a general reference and a
   references to any prior work applying these ideas to
   representation learning.
2. The variable L is used in the caption of Figure 1 but not
   introduced in the text until section 4, so that part of the caption
   is difficult to understand on the first reading.
3. Motivation for using the Jacobian: The authors write that using the
   Jacobian of the neural network is a "more efficient way to produce
   the potential function" (line 142). In what sense is it more
   efficient? Is it possible to provide another intuitive
   justification for this choice of similarity function?

questions: 1. Is there any reason that the method is not compared to prior method for polysemous word representations in section 8?
2. Is it possible to cluster words using the proposed similarity function, rather than Euclidean distance, in the DBSCAN algorithm?
3. How is the number of word locations chosen?
4. Can you provide more intuition for the choice of the Jacobian as the similarity function?
limitations: The authors largely address the limitations of this method. It would be good to add a reference for the number of hours required to train the equivalent baseline methods (lines 184-186).
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors presented FIRE,  a semantic field representation that is defined on a D-dimensional space, with a field of word interactions. I find the method novel and innovative. Authors also conducted relevant analyzes that solidify their theories. In my opinion, the paper is well written, sounds and flows.
strengths_and_weaknesses: - The paper is well written
- The formulas are well explained, simple, and easy to follow
- The implementation of FIRE is also explicitly described.
- Relevant experiments have been conducted and are convincing of the approach. FIRE captures polysemy (better than BERT and Word2Vec) which is also a good thing.
questions: - I was wondering if the authors have a more strong explanation of table 3. I would really like them to bring more structural explanation rather than the # of parameters
- It would have also been great for the authors to discuss the limitation of their method. For instance, they used (D, K, L) = (2, 4, 1). How well will FIRE perform in higher dimensions and with larger Ks (number of positions/locations) and Ls? This would have made the paper stronger
- Has the authors explored other types of functions for f()? Dirac function is great but what about for instance the Dirichlet function with is a `non-step` function?
limitations: See Questions
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes an alternative to the standard linear vector spaces that encode word embeddings. The authors propose FIRE (FIeld REpresentation), in which words are represented as a set of locations and a nonlinear function defined on a field. This enables word representations to model the linear nature of compositionality and the nonlinear nature of polysemy. Following existing work on word and sentence representations, the authors propose methods to calculate word/sentence similarity and to compose word/sentence representations. They also present an efficient way to implement the potential function of a word, using the Jacobian of an MLP, and modify the skip-gram negative sampling optimization function to train their model. FIRE is evaluated on word similarity, word disambiguation, and polysemy prediction. Its performance is compared to existing methods, including Word2Vec and contextual BERT embeddings.  The results show that on word similarity and disambiguation tasks, FIRE performs similar to existing methods. On polysemy prediction, FIRE outperforms the baselines, showing that its nonlinear nature allows it to better model polysemy.
strengths_and_weaknesses: FIRE is a nice addition to the word and sentence representation literature. The technique is well motivated from a mathematical perspective and quite distinct from existing methods. However, I do not think the results are strong enough to support the introduction of such model complexity, when compared to the simplicity of Word2Vec and the strength of BERT. Furthermore, I think there are stronger baselines that the model should be compared to, especially for sentence encodings. If more baselines are added and more sophisticated analyses of results are performed, I believe this paper could be a strong submission to a computational linguistics venue.

**Strengths:**

1. FIRE is quite novel and an interesting divergence from the standard approach to encoding words and sentences. As the authors point out, there are many advantages to using locations and nonlinear functions to represent words. 

2. The authors build their work on mathematically solid foundations, which they explain and lay out well. The similarity and compositionality calculations are not obvious extensions of FIRE - these are good ideas and they are presented clearly. Furthermore, training such a model is challenging. The efficient Jacobian-based training algorithm is another valuable contribution of this paper.

3. The whole paper is well structured and clearly written. The authors manage to concisely present ideas that are sometimes mathematically quite dense.

**Weaknesses:**

1. The results are not very strong and require further analyses. Achieving performance that is similar to Word2Vec and BERT is impressive, but not sufficient given the complexity of FIRE. If a new model is proposed that is much more complex than existing methods, that complexity must be justified through convincing results. The results seem to suggest little gain over existing methods. I would suggest more experiments that focus on polysemy, since this seems to be the strength of FIRE.

2. More analysis should be done to interpret and investigate the results. If performance metrics alone are not convincing, a good case can still be made for FIRE based on analyses that demonstrate how the model is effectively modeling certain aspects of language. For example, Figure 2 shows that FIRE models polysemy well. More such results should be reported. I suspect FIRE is also advantageous from an interpretability perspective e.g. perhaps it is possible to extract different senses from FIRE.

3. More baselines, and stronger baselines, should be included in the results. On the word-level tasks, techniques like GloVe and FastText would be useful for comparison. On the sentence-level tasks, FIRE should be compared to state-of-the-art sentence embedders like InferSent, SentenceBERT, and Universal Sentence Encoder, as well as different word embedding composition techniques, besides weighted summation (e.g. element wise multiplication, averaging, matrix multiplication). I understand that these methods are contextual, but they are still the benchmark against which new sentence embedding methods should be compared.

questions: 1. Much of your paper is motivated by the claim that compositionality is a linear quality. What is this claim based on? Transformers are currently state-of-the-art for sentence composition and they are nonlinear.

2.  In Table 1, why do you mark Word2Vec as not being effective with small D? One of the persistent advantages of Word2Vec is its low dimensionality.

3. For the experiments with FIRE in Section 8, why do you have to extract and cluster the representations of 1000 similar words? Would it not be possible to obtain a prediction of the number of senses by just using the potential function of a word? To me this would be more convincing proof of FIRE’s value - if the degree of polysemy is encoded in the word representation itself.
limitations: The authors should consider adding more discussion on the limitations of their model. It should be made more clear in which contexts FIRE could be useful and in which in couldn't.
review_title: null

Meta-review: metareview: Two reviewers who suggested that we accept the paper had significant engagement with the authors and that improved the paper quite a bit.  The fourth reviewer also made several suggestions regarding further analysis, which also was incorporated by the authors to a large extent, and it improved the paper.  I disagree with the fourth reviewer that Neurips is not the right audience for this paper, while a CL conference is, and don't agree that it is a reason for rejection.  The other review suggestion a strong reject does not hold too much ground since they are suggesting that the authors compare with more baselines (and they do compare with several reasonable baselines) but ignoring the interesting method that the authors have presented.  Overall, I think the paper is quite interesting even though the results are not extraordinarily SOTA and hence we should give it an audience at Neurips.
confidence: Less certain
Decision: Accept