FIRE: Semantic Field of Words Represented as Non-Linear Functions
Authors: Xin Du
Kumiko Tanaka-Ishii
Venue: NeurIPS 2022 Accept (2022)
Keywords: natural language
nonlinear word representation
field representation
word polysemy
semantic compositionality

Abstract: State-of-the-art word embeddings presume a linear vector space, but this approach does not easily incorporate the nonlinearity that is necessary to represent polysemy. We thus propose a novel semantic FIeld REepresentation, called FIRE, which is a $D$-dimensional field in which every word is represented as a set of its locations and a nonlinear function covering the field. The strength of a word's relation to another word at a certain location is measured as the function value at that location. With FIRE, compositionality is represented via functional additivity, whereas polysemy is represented via the set of points and the function's multimodality. By implementing FIRE for English and comparing it with previous representation methods via word and sentence similarity tasks, we show that FIRE produces comparable or even better results. In an evaluation of polysemy to predict the number of word senses, FIRE greatly outperformed BERT and Word2vec, providing evidence of how FIRE represents polysemy. The code is available at https://github.com/kduxin/firelang.

FIRE: Semantic Field of Words Represented as Nonlinear Functions
Xin Du The University of Tokyo duxin@cl reast. u-tokyo. ac. jp
Kumiko Tamaka-Ishi The University of Tokyo kumiko@cl. rcast u- u-tokyo... ac jp
Abstract
State-of-the-art word embeddings presume a linear vector space, but this approach does not easily incorporate the nonlinearity that is necessary to represent poly- semy. We thus propose a novel semantic Fleld REepresentation, called FIRE, which is D-dimensional field in which every word is represented locations and a nonlinear function covering the field. The strength of a word's s relation to another word certain location is measured the function value at that location. With FIRE, compositionality tivity, whereas polysemy is represented via the of points and the function's 's multimodality. By implementing FIRE for English and comparing it with previous representation methods via word and sentence similarity tasks, show that FIRE produces comparable or even better results. In an evaluation of polysemy to predict the number of word senses, FIRE greatly outperformed BERT and Word2vec, providing evidence of how FIRE represents polysemy. The code is available at bttts::/githuu com//kduxin/ffrelang
set of its
represented via functional addi-
Introduction
Word representations have become an important basic technique in NLP. The most important se- mantic requirement for word representation, above all, that similar words are mapped to similar embeddings. Furthermore, the next two most important properties the representation of two composed words matches the composition the two representations, and polysemy, the representation of semantic ambiguity. Intuitively, compositionality is linear quality [27 whereas polysemy nonlinear and thus requires a representation via mathematical multimodality, which is difficult to represent within linear space.
compositionaliiy [27] i.e.,
State-of-the-ar -frt word representations are based on linear vector space. The linearity of vectors naturally accommodates compositionality, but the question of how to incorporate polysemy has been an issue. Indeed, recent random-variable representations at the cost of losing compositionality.
20
represent polysemy but
The most important previous work that seems incorporate polysemy in addition to compositionality could be BERT 3 however, it is a contextual embedding that represents one vector only under given context that disambiguates the meaning. In contrast, polysemous word has multiple meanings even without context. Moreover, disambiguation (e.g., with BERT) is also hard, because semantics is often continuous: the word "bank" has meanings in WordNet [77, which are hard disambiguate even for humans. Our idea was thus to devise an entirely new representation that is nonlinear but naturally represents linearity as well.
Hence, Fleld REpresentation, which we call FIRE, is a non-contextudl representation of words. is realized in D-dimensional space in which every point has a quantity, called semantic strength, that is generated by interactions among words. Every word is represented by a pair consisting of
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

set of the word's 's locations and nonlinear function f representing the word's context, where the function outputs the semantic strength at a location the field. Two functions can be linearly added, which provides the basis of compositionality. Polysemy is represented through the set of a word's locations and the multimodality of the function f. Through this representation, the similarity of two words is obtained by adding the function values at the word locations.
Our experimental results confirmed that FIRE showed similar or better results in comparison with previous Word2XX representations under settings with the same number of parameters. Furthermore, by representing multimodality over the entire field, FIRE has a much wider expressiveness for polysemy, In an evaluation to predict the number of word senses, FIRE greatly outperformed BERT.
2 Related Works
Vectoral word representations. The most widely used word representations are vectoral forms, such as Word2Vec [T5.. [12 showed that Verd2Vec is essentially decomposition of the point-wise mutual information matrix. Other vectoral representation methods extract different information from corpora. Important works include GloVe [19], fas/Texi (6), ELMo [20). and BERT
It has been attempted represent polysemy via multiple vectors per word 26 but such "mixture" of vectors lost the quality of compositionallty.
22 proposed Word2Gauss, which represents each word by a Gaussian distribution. They exploited the covariance matrix to model the semantic ambiguity of words. Asymmetric similarity (via the Kullback-Leibler divergence) was further introduced to represent hypernyms and hyponyms. Word2GM 3 uses a mixture of (two) Gaussian distributions.
Gaussian word representations.
It is able to represent polysemy by assigning each Gaussian component represent separate meaning of a word. [25] proposed to use the Wasserstein distance between two Gaussian distributions to measure word similarity, and that work was generalized by [[]] use elliptical distributions. Between Gaussian or elliptical distributions, the Wasserstein distance has a closed form and can be computed efficiently. We compare these Word2XX methods in Table !I
Multinoulli distributions on point clouds. 10 proposed Word2Cloud, in which each word is represented by (discrete) Multinoulli distribution defined cloud of 64 points in low- dimensional space (e.g.. RR²). Between Multinoulli distributions, the Wasserstein distance has no closed form, but is approximated efficiently by the Sinkhorn distance 77. In the word clouds of Word2Cloud, each point is a Dirac delta function, and we adopt that notion here.
The context mover's 's distance (CMD)
[2]] used pretrained GloVe vectors 19 in 300 to construct a point cloud that represents a "context." " In this paper, we do not show quantitative results for CMD, because we were unable to reproduce the original paper's results with the published code. Instead, we report results for Word2Cloud and GloVe: Word2Cloud is theoretically close CMD, and 23 reported that GloVe showed comparable performance with CMD.
also uses the Sinkhorn distance.
Many of these random-variable variable word representations incorporate polysemy. However, none except CMD 2 incorporate compositionality. Therefore, they have not been extended to sentence represen- tations. CMD [ can produce sentence representation by computing the Wasserstein barycenter of the words, which gives a new Multinoulli distribution on the point cloud. However, that computation not additive, whereas FIRE is additive.
Sentence representations. A typical sentence representation is a vector obtained as a weighted additive composition of vectoral word representations. Various weighting heuristics have been studied, such as the term frequency- inverse document frequency (TF-IDF) 14] [ proposed the smooth inverse frequency (SIF), simple weighting scheme that works exceptionally well. In this paper, too, we evaluate the capability of the sentence field representation produced by applying the SIF to FIRE word representation.
State-of-the-art vectoral sentence representation methods adopt an adaptive weighting scheme by using a self-attention mechanism (8); one implementation includes the sentence representation of BERT g Such methods can also be applied to our word field representation, which remains future work.

---

Inscrrance, morigngs loan, (en, salery, tax, peysite, pest.. south, treland, northern telands, aealand.
2iVeer Sinnscial
2.0
32
15
1.0
0.0
0.0
-24
-.8
33
-3. -3.0 -2.5
-2.0 -1.0
-15
-30
-2.5
-1.0 -0.5
Figure Left: Semantic field of bank for 2, 4, K 1, where is the dimension of this representation space, L is the number of transformation layers (see Formula (7). and K is the number of polysemy locations (Formula 2 With K 1, each word has only one location in the field, which is denoted by a scaled Dirae function, fr mm(s). The triangle shows the location of bunk. The circles represent the locations of 1000 words that were frequent and similar to bank. The words were clustered and colored by using the DBSCAN algorithm (9. Polysemy is visible via two clusters of words, finance and geography. The white circles represent the locations of words that DBSCAN considered outliers.
Right: Semantic field generated for = 2, L = K by superposing the functions of river (blue) and financial (red). The two points show the words' locations. The color intensities indicate the values the functions at a point, and the arrows indicate the gradients.
3 Semantic Field: FIRE
3.1 Word Representation
FIRE is semantic field generated by words. In physics, field is space in which every point has some physical quantity, of which a common type is the strength, e.g., electric field strength. FIRE uses the semantic strength generated by words. While particles in physical field are often identical, the particularity of FIRE is that every word produces a different semantic strength and is thus allocated within the field in association with an individual function.
Let W set of words and be a D-dimensional real space. Our semantic field defined on S. A word W is represented as a 2-tuple [u, f()]]. where μ represents the word's 's locations. In previous linear space model [5]] a word was usually located at a single point. However, a word can be polysemous, and in such cases, it requires a representation with multiple locations, suggested in (26]. Let K denote the number of locations.
f(s) S + R a continuous realevalued function covering 8, realized as neural network. f(s) represents the context generated by the surrounding words of w, as will be explained in Section 4.1
Figure (left) shows an example portion of S computed for D = with 5 = [-4, and K = 1, for the term bank. The location j for bank is shown by the orange triangle. There are two clusters related to bank, one for financial bank and the other for a river bank. The circles show the locations z for 1000 words (among the most frequent 50,000 words) that were frequent and similar bank, with similarity defined in the following section.
3.2 Similarity of Two Words
In our approach, two words w, jus, (s)) and wy = Iuy, [s)] in field interact. Given the location sy of wy. the interaction with w; is measured by f(())). As mentioned above, polysemy means that wy has multiple locations, (a)
1, I,..., K: thus, the ((() must added. The location here means that an entity is scattered in field: it can be multiple and noncontinuous, and it may have different mass at different locations. This notion mathematically expressed as a measure, and integral with measure is formulated via the Lebesgue integral.
3

---

The simplest measure is the Dirac delta function 8(s), which indicates that a location is point-wise within field. For K and μj ((s)), we have fidui f((s)). By using the Dirac delta function, we can compute the integral exactly without any approximation.
FIRE incorporates polysemy via (1) the multimodality of f(s) and (2) the number K of word locations. As mentioned above, the multimodaliity of f(s) is implemented via a neural network.
As also have the symmetric quantity of wy's 's location in the field of (s) and wj's wy's location in the field of [()) the similarity of the two words is naturally expressed as follows:
+ f f3 dp Yus, wy E W.
sim wi, wz)) = fs day fiduy
(1)
Using a measure for word's location facilitates simple representation. For K 1, p Dirac mixture with weights (A) as follows:
K m (2) 8()(4))
(2)
By using this y for each word, Formula
reduces the following (because f fdō(s) f(s)):
sim w,)
Σ [100
(k) my
m
(b)
r,(0!*),
(3)
Jmm
which is simply a weighted sum the two functions fs() and (s) at all the locations of wy and Wi, respectively.
The similarity this work between words and sentences (which we define later in Section 3.3 via measure is different from the standard notion of distance in Euclidean space and is related to metric learning methods [ΑI] Like metric learning, FIRE is nonlinear with / realized as a multi-layer neural network. Whereas metric learning methods share large neural network for all inputs, FIRE assigns an individual small neural network each word.
3.3 Composition of Words and Similarity of Two Sentences
set of words, r Wi...., wp), including sentences, can also be represented in [M, f(s) notation when defined as a weighted average of word representations {14 f1)s),...... [ f(((s))))) (s)]}. as follows:
RP4 ((8) = Σ nf{(s),
(4)
where yv are the weights. In this work, used SIF weights (2] as the 7/. Note that this f(s) is multimodal and thus polysemous, because all the fi(s) are multimodal.
Figure α (right) shows such a superpasition of the words river (blue) and financial (red). Here, river the lower mid, with (s) forming a diagonal river-like flow from upper right to lower left, whereas financial is at the upper left, with (s forming reddish area. The color intensities and arrows represent the function values and gradients, respectively. The field superposing river and financial is similar to the representation of bunk on the left, demonstrating the effect of FIRE's linear composition.
The similarity of two sets of words or sentences. r = [M, fl and I' = j f']. is measured by applying the similarity function (.. Let n and n' denote the respective numbers of words in the two sentences. The words in I and are assigned weights = [m...., ,.." and y hi..... % respectively. Then, as shown Appendix Formula ① is simply calculated as follows:
sim ( = TTT'
(5)
where E is the word similarity matrix, defined as Σig sim (w; w)) with and wy being words in sentences and r', respectively. In other words, the similarity between two sentences can be obtained from the pairwise word similarities between the sentences.

---

Implementation of FIRE
4.1 Nonlinear Functions As Planar Transformations
The simplest way implement the function f(s) would be construct a neural network NN with D-dimensional input and 1-dimensional output, i.e., f(s) = NN(s; where 0 represents the parameters of the neural network. A usual choice for the NN a multi-layer perceptron (MLP).
more elaborative way to produce the function would to use the Jacobian of such an NN: - S. The trace of the Jacobian gives the divergence of the vector field generated by the NN, which is scalar in IR:
f(s) tr aNN(s;
As the Jacobian (NN(s;; 05/ can be decomposed into multiplication of the Jacobians of all the NN's layers, it can be computed efficiently. In fact, it was empirically found to work better than the previous simplest MLP.
In particular, a normalizing flow 22 i..., a reversible neural network, can be used here as NN, as naturally generates vector field 5 S. For our implementation,
adopt negative multi-layer planar (MLPlanar) transformation 221 which is a simple kind of normalizing flow. Each layer of
MLPlanar is determined by two vectors and with scalar b, as follows:
Planar(2; 0) = x + tanh(v" x + b)u,
(6)
where 0 = (v, b, u) is the parameter set. The Jacobian of the I-th layer with input 8()) is
@Planar(s(o) I Ag).
as())
where Ag) = tanh' (18)(())))00 + bu)) wcy is rank-one matrix, with the derivative tanh'
1 - tanh?. Thus.
((s) tr OPlanar (s; tr II Ac)), 1=1
Ös
where s(z) (1) and (()) Formula 7 uses the negative trace, so that f has modes that are mostly "peaks" as in Figure I rather than "valleys." Because similar words are located where has higher values, the negative trace drives similar words toward "peak," which facilitates the separation of a word's multiple meanings. This enables FIRE to discover polysemy. Formula ( expands to matrix polynomial of degree L, which can represent the multimodality of f(s). For the above case with D 2, evaluating (s) takes (L)) time.
4.2 FIRE Training by Skip-Gram with Negative Sampling
For training FIRE, we adopt the Skip-gram algorithm with negative sampling (SGNS) [6], an unsupervised method. More advanced training methods (e.g.. GloVe) can also be applied, which remains a future work.
In SGNS (one negative word per time), three- word tuple (w,, up, w.) is sampled from corpus. Here, W; called the center word, Wp is a positive word that co-occurs with the center word somewhere in the corpus, and w,, is an arbitrary word, called the negative word. We use only one negative word per positive, following the suggestion in [12].
The idea of SGNS is to optimize the parameters in word representations such that sim (wi, w.) is high and sim (w,, w.) is low, following the assumption that words co-occurring in the same neighborhood are more similar. For example, the words bank and morígage should have a high similarity because they co-occur frequently.
'The original planar transformation had additional constraints on the parameter values to preserve reversibility. Because FIRE does not require reversibility, we skip those constraints to speed up our implementation. We thus call Formula (6) pseudo- planar transformation.
5

---

Following the implementation in [5], the optimization function with respect to similarity is
min Σ a--sim(;;, wp)) + a (sim(w;, w.)).
(8)
where (xr) = log(1 + exp(xc) is the softplus function. Evaluation of the similarity in Formula between two words has (KL)) time complexity when =
Note that Formula (8) is nor the tripler loss in metric learning. A triplet loss is typically used to evaluate large neural network shared by all words, whereas in FIRE, each word is represented as an individual small neural network defined by Formula 7
4.3 Experimental Settings for Training
We obtained FIRE representations two datasets: the large Wacky dataset, which contain about billion and 17 million tokens, respectively.
4 dataset and the small tex18
For both datasets, we converted the text to lowercase and tokenized it with the NLTK toolkit [5 We followed the settings used in previous works and replaced infrequent words with <unk>. For Wacky, infrequent words were those occurring less than 100 times, forming vocabulary of around 270,000 words; for tex18, the threshold was times, forming a vocabulary of about 70,000 words.
As mentioned in Section 3.1 we used 2-dimensional space S = [14,44]
For the SGNS hyperparameters, negative sample per sample, and the subsample rate for ut was le-5 for Wacky and lc-4 for tex18. The negative sampling probability
followed the settings recommended by 15
We used one
adjusted by power of 0.75. For training FIRE representations, we adopted the Adam W optimizer [33. The OneCycle learning rate scheduler 24 was applied, with the maximum learning rate set 0.005.
As noted above, the time complexity for training is O)L)). For the Wacky dataset, all methods were trained for three epochs, which took about 10 hours for FIRE with 50 parameters per word single GPU. For text8, FIRE was trained for 15 epochs.
In similarity benchmarks, conventional vectoral representations use the cosine similarity function, where sim(w;, wx) Vi, to provide a natural regularization. For fair comparison of FIRE with baselines, we used the following regularization for word similarity in our evaluation:
sim(w; Wz)) exp (iim (W3, w))) / f: did - f f3 dps))
For sentences, substituting Formula (5) with the regularized version word similarity gives the regularized sentence similarity. Because our similarity function is point-wise, smoothing is also required. Appendix C details this process.
Baselines
We qualitatively compared FIRE with previous semantic representations, summarized in Table I For fair comparison such different representations, the most important factor the number of parameters (sixth column). For FIRE (bottom of the table), requires K locations with D dimensions, plus the number of weights (k), i.c., K, giving (D + 1)K. As for f(s), v and u (Section 4.1 are D D-dimensional, are implemented L layers, and have intercepts giving (2D + 1)L. The total is thus (2D + I)L + (D + 1) parameters per word.
The second column shows whether the method must process word with context, and only BERT- large has this requirement. The third and fourth columns in Table D iindicate whether a method has the qualities of additive compositionality and polysemy; FIRE is the only non-contextual method that theoretically accommodates both. Furthermore, as shown by the fifth column, FIRE and Word2Cloud are the only two that represent polysemy an interpretable way, i.c., that are visualizable on 2D plane without dimensionality reduction. However, as will be explained in the next section, Word2Cloud is not comparable with FIRE on word similarity benchmarks.
(k) in For an ablation study, we considered variant of FIRE, denoted as FIRE/m, that fixes m Formulas 2 and FIRE/m shares the properties of FIRE, but with fewer parameters per word.

---

Table I: Qualitative comparison of word representation methods. The second to fifth columns indicate the methods' qualities. The sixth column gives the number of parameters assigned per word: D denotes the dimension of the representation, K' is the maximum number of polysemy components, and L is the number of neural layers. The rightmost column gives the time complexity for evaluating the word similarity function, where we consider D = 2 for FIRE as a constant.
Non- Contextual
Composi- tionality
Interpre- tability
N (# of parameters)
Complexity sim(w1, w2)
method
Polysemy
Vectoral representation
Word2Vee (2013)
D
O(N)
GloVe (2014)
O(N)
BERT-large (2019)
1024
high
Random variable representations
Word2Gauss/S (2014)
x
x
D+1 1
O(N)
Word2Gauss/D (2014)
2D
O(N)
Word2GM/S (2017)
(D 2
O(KN)
Word2GM/D (2017)
X
(2D 11K+
O(KN)
Word2Cloud (2019)
x nonlinear
V
K = 64
O(N2)
CMD (2020)
K 200, 400
O(N²)
Our semantic-fiele field representations
FIRE (2022)
V
(2D+1) ()=+)))+
O(KL) O(KL)
FIRE/m (2022)
V
(2D - 1)L DK
We adopted the baselines listed in Table [ in this work, as explained in Appendix D Briefly, Word2Gauss and Word2GM represent words Gaussian mixtures, whereas Word2Cloud represents words as a Multinoulli distribution of K discrete points in a low-dimensional space. For Word2Gauss and Word2GM, "/S" indicates spherical covariance matrix r/ requiring one parameter (i.e., r), whereas "/D" indicates diagonal covariance matrix diag(r1, ... .TD) requiring D parameters. For Word2Cloud, refer the results reported in the original paper, because the original code is not available. As Word2Cloud was only implemented on the relatively small text8 dataset, for comparison, constructed FIRE models with the same number of parameters and trained them the text8 dataset.
6
Evaluation on Basic Word Semanties
First, FIRE was compared with the baselines on 12 word similarity benchmarks to test the basic word semantic representational capability. As these benchmarks for non-contextual methods, BERT-large was not included. For each benchmark, we computed the similarity of a word to other words and generated ranking of those words. The gold standard was compared via Spearman's correlation coefficient with a higher indicating better method.
The results listed in Table 2 Each block groups methods with the same number of parameters. Generally speaking, our method produced comparable or often even better results than the baselines.
For either #par-50 or 100 (third fourth block, respectively). FIRE was the only method whose performance was comparable to that of vectoral methods. For example, achieved an average of 47.9% and 49.8%, respectively, for #par=40 and 50; these values were only 1.1 % and 0.6% less than those of Word2 Vec (second and third blocks). These results show that our functions and measures have sufficiently large capacity
with semantic field small [-4, 4 In Appendix we included further experimental results about K' effect. Briefly, our method is computationally efficient even for larger K, because the time complexity of FIRE is O(K), rather than O O(K2) as in Word2GM and Word2Cloud.
Another perspective is to specifically compare the www-dimensional representations, i.e., those with small D (last block). Word2Cloud is such representation and is comparable to FIRE with D Although Word2Cloud was computed on the smaller tex18 dataset, it produced better similarity scores than the 2-dimensional Word2Vec listed the top row Table 2 When trained on tex18, however, FIRE greatly outperformed Word2Cloud, even with smaller number of parameters.

---

Table 2: Spearman's 's correlation coefficient (%) between the gold standard and the similarity rankings produced by the various word representations. The results are grouped by methods with the same number of parameters. The best and second-best results in each group in bold and underlined, respectively. The results for the previous methods were obtained by replicating them with the published code, except for Word2Cloud in the bottom block, for which the code was unavailable. The values the row for Word2Cloud were taken from the original paper 100
datuset
MC
MEN
RG
WS
WS.
MT
MT
RW
Verb
YP
SV
Ave.
# word pairs
30
3000
65
203
252
287
771
2034
143
130
999
3500
#pur.
D
L
K
Dutaset: Wacky (3B lokens)
Werd2Vee
22.2
17.4
5.5
Word2Vee
40
40
67.8
69.1
66.1
70.1
52.1
65.0
60.7
27.8
32.6
39.6
18.4
19.3
49.0
FIRE/m (ours)
10
72.0
65.2
71.9
64.6
47.7
63.5
60.1
25.9
32,9
31.8
20.1
19.8
47.9
Word2Vec
50
72.7
69.2
54.2
62.1
27.5
30.2
19.7
20.3
GloVe
50
66.8
68.7
73.4
61.3
43.2
58.6
62.2
28.0
34.3
42.1
19.4
15.8
47.8
Word2Gauss/DD
25
51.7
50.2
65.5
51.1
48.0
56.9
46.0
23.3
23.9
26.0
16.4
39.0
Word2Gauss/S
49
63.7
53.7
69.8
54.3
48.6
57.4
48.9
25599
19.5
32.8
36.9
18.6
43.3
Word2GMSS
23
58.4
57.7
63.7
58.9
37.4
$1.0
53.6
24.1
10.0
28.3
16.0
18.4
39.8
FIRE
74.2
52.1
59.2
34.5
18.7
Word2Vee
100
100
71.4
73.6
75.2
74.2
57.7
66.4
66.3
29.1
39,8
47.1
21.5
22.9
53.8
GloVe
100
69.6
72.3
77.0
49.0
61.9
65.8
30.2
35.0
49.2
23.3
18.5
51.4
Word2GaussDD
50
68.8
55.8
70.0
55.3
48.1
61.5
50.8
25.6
9.1
37.4
27.9
18.5
44.
Word2Gouss/S
99
64.4
71.0
48.0
50.3
15.2
27.4
19.3
Word2GMM/
48
63.1
60.7
68.6
61.0
45.5
56.4
57.5
25.7
23.1
36.8
18.6
20.1
44.8
FIRE (ours)
20
75.9
69.4
76.4
70.4
54.4
62.7
61.1
27.2
33.7
38.9
20.0
21.0
Word2GM/S
104
50
79.1
73.6
74.5
75.5
59.5
66.6
60.8
28.6
37.4
45.1
19.7
22.6
53.6
Dataset: text8 (17M tokens)
Word2Cloud
32.0
FIRE (ours)
50
10
42.6
43.2
44.8
52.4
38.3
55.6
43.0
12.3
36.5
22.8
14.3
9.4
34.6
(ours)
56.2
50.6
The results Wonf2Cloud were taken from the original paper.
The results with #par==044 was acquired with best peetrained model available at slightly larger vocabulary and trained with different scheme.
which has
7 Evaluation of Compositionality via Sentence Similarity Benchmark
FIRE's linear compositional capability was evaluated using a sentence similarity benchmark. sen- tence representation is produced by composing words in conventional vectoral word representations, making full use of linear compositionality. Similarly, FIRE represents a sentence by composing word
representations described in Section 3 The similarity of two sentences is given by Formulas 5
and (see Appendix C). with respect to the SIF weights [2]
We set = 10-3 for SIF.
We compared FIRE with vectoral word repre- sentations on the Word-in-Contee
(WiC) [2] and STS (2012-2016) benchmarks. WiC is bi- nary classification task asking whether a word's meaning differs in two given context sentences. The semantic similarity scores between sentence pairs were computed, with scores below thresh- old indicating that the word's meaning differs in the two sentences. STS evaluates sentence similarity by the Pearson correlation coefficient between the predicted similarity and the gold standard (see Appendix
Table 3: Performance the WIC and STS (2012- 2016) benchmarks. See Appendix G for the de- tailed STS results.
N
total par.
WiC
STS
Word2Vec
50
13M
61.8
59.7
100
27M
62.0
61.9
FIRE
50
13M
61.9
57.5
100
27M
62.7
61.1
BERT-large
1024
340M
66.5
63.3
Table S lists the results. Note again that Word2Vec and FIRE are non-contextual, whereas BERT-large is contextual. The second and third columns give the model's number of param- eters per word and total parameters, respectively. The fourth and fifth columns give the accuracy and correlation scores, respectively.

---

(a) PCA
(b) Spectral Embedding
"bank" (poly)
"rock"
(poly.
"glacier"
(mono.)
FIRE
Word2Vec
Word2Gauss
Word2GM
BERT
FIRE
BERT
65
Word2Vec
Pearson's correlation
30
Word2Vec
BERT
Word2Gauss
FIRE
Word2Gauss
FIRE
60
Word2GM
WordZGM
Accuracy
20
55
50
0.2
0.4
0.6 0.8
0.2
0.4
0.6 0.8
Parameter eps of the DBSCAN
Parameter eps of the DBSCAN
Figure 2: Upper: Geometric distributions of word embeddings computed with five methods for the polysemous words bank (top row) and rock (middle row), and the monosemous word glacier (bottom row). Lower Left:
Accuracy of polysemouusmoonosemuuu detection with respect to the DBSCAN threshold parameter "eps." Lower Right: Pearson correlation between the detected number of senses and the gold standard, with respect "eps."
BERT-large, a contextual method, has 20 times more parameters than the non-contextual Word2Vec and FIRE, and its accuracy and correlation were higher because of the context stored in this large memory. Between the non-contextual methods, FIRE performed better than Word2Vec on WiC. BERT's performance was naturally the best because it has context; more importantly, this result demonstrates that FIRE's compositionality comparable to Word2Vec.
Evaluation of Polysemy
The quality of FIRE lies in its capability to capture polysemy. Different meanings (i.e., senses) of word are automatically represented as separate modes nonlinear function, as seen for bank in Figure with finance- and geography-relate modes. Appendix A gives more examples.
We quantitatively evaluated the polysemy of FIRE via the task of word sense detection, where the number of word senses is predicted by each method and compared with a gold standard via WordNer 177. From the Core WordNet dataset Wiihh with 5000 frequent senses of 3739 different words, we selected words annotated with at least three senses (266 words that are strongly polysemous). From Core WordNet and the whole WordNet, we also selected 276 words that are strongly monosemous. We thus used 542 words, with various numbers of word senses.
FIRE was compared with four previous word representations: Word2Vec, Word2Gauss, Word2GM, and BERT. All methods except BERT are non non-contextual. With the non non-contextual methods, including FIRE, for each target word, the most similar 1,000 words were acquired from the most frequent 50,000 words, and their locations were plotted in the 2D plane. For Word2Gauss and Word2GM, we used the mean vectors of the Gaussian distributions, See Appendix for the details, Figure
http:: httpa//tww...p....ee......eaeaaa
The word list is available http:s/s//gtttuooo.oooo/u/////at

---

(upper), we show the results, with dimensionality reduction by (a) principal component analysis (PCA) and (b) spectral embeddings, for three words; "bank" and "rock" are polysemous, while "glacier" is monosemous.
In contrast, BERT is contextual model requiring context to count the number of senses. For each target word, 1,000 context sentences were collected from the Wacky dataset, and corresponding BERT vectors for the target word were generated. For a target word that was split into subwords by BERT's default tokenizer, we averaged the subword vectors to represent the target word. The rightmost columns in Figure (upper, a and b) show BERT's results for the three sample words.
As mentioned above, for plotting on the 2D plane, we used two dimension reduction methods, PCA and spectral embeddings, to acquire 2D points from the vectors for BERT (1024-d) and the other methods. For FIRE, the application of PCA is equivalent to performing rotation and normalization, but nevertheless applied it for fair comparison. For the details, see Appendix H
Figure 2 (upper) clearly shows how well FIRE captured the different senses. The term "bank" produced three senses, with those the right corresponding river bank and a "large pile," listed in many dictionaries. On the other hand, such clusters for Word2Vec and BERT are difficult to see. As Word2 is monosemous, it is not expected to produce any clusters, while clear clusters are barely visible for BERT. Although BERT performed well in contextual processing, this shows how context and polysemy are not necessarily the same. As for Word2GM, it uses two Gaussian components for each word and is biased to show two clusters, even for the monosemous word "glacier."
For all 542 words, the number of clusters, I, was counted with DBSCAN with different thresholds eps. For all methods the number was counted on the PCA-reduced 2D planes. For BERT, this PCA result was better than direct application of DBSCAN to the original high-dimensional spaces. The number of clusters was then compared with the gold standard. As the gold standard consisted of 49% (= (=266/35 266/542) polysemous and 51 % monosemous words, the result was judged accurate by = 1 for monosemous words and / for polysemous words.
Figure 2 (lower) shows the results for the five methods, with the horizontal axes indicating eps. The vertical axes of the lower-left and lower-right graphs show the binary accuracy and Pearson correlation coefficient, respectively. Word2Vec, Word2Gauss, and Word2GM performed at chance level for any eps value. BERT was slightly better but could not detect the number of senses. In contrast, FIRE showed clear capability of identifying polysemy.
Conclusion
We proposed FIRE, a semantic field representation that is defined on a D-dimensional space, with a field of word interactions. A word is represented by a pair that consists of a set of multiple locations for the word and a function covering the semantic field that represents the word's context. The interaction of words is measured via similarity function, which is naturally defined the sum of the functions at the word locations. The multiple possible word locations and the function's multimodality account for the quality of polysemy, and the functional addition formally guarantees the quality of additive compositionality.
FIRE is implemented via planar transformation of neural network. Multiple images of 2D field were shown to demonstrate the capability of polysemy representation. FIRE was also evaluated through large-scale analysis of word and sentence similarity benchmarks, in which it showed comparable or better results than conventional Word2XX representations. Moreover, FIRE greatly outperformed BERT and Word2Vec in evaluating polysemy by well predicting the number of word senses. These results provide evidence of FIRE's 's capability to incorporate polysemy as a nonlincar quality with compositionality
linear quality.
Our future work will entail extension of this framework to a more general representation, which will further improve the capability of FIRE.
Acknowledgements
This work was supported by JST, CREST Grant Number JPMJCR2114, Japan, and by ISPS, KAK- ENHI Grant Numbers JP20K20492 and 1721J11781.
10

---

References
[1] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algehraic structure of word senses, with applications to polysemy. Trans, Assoc. Comput. Linguistics, 6:483-495, 2018.
[2] Sanjeev Arora, Yingyu Liang, and Tengyu Ma. simple but tough-to-bet be-ttt baseline for sentence embeddings. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[3] Ben Athiwaratkun and Andrew Wilson. Multimodal word distributions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), pages 1645- 1656, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[4] Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. The wacky wide web: a collection of very large linguistically processed web-crawled corpora. Language resources and evaluation, 43(3):209-226, 2009.
[5] Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. " "O'Reilly Media, Inc.", 2009.
[6] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomás Mikolov. Enriching word vectors with subword information. Trans. Assoc. Compul. Linguistics, 5:135-146, 2017.
[7] Marco Cuturi. Sinkhorn distances:
Lightspeed computation of optimal transport. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[9] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density- algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226-231, 1996.
[10] Charlie Frogner, Farzaneh Mirzazadeh, and Justin Solomon. Learning embeddings into entropic wasserstein spaces. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9. 2019. OpenReview.net, 2019.
[11] Mahmut Kaya and Hasan Sakir Bilge. Deep metric learning: survey. Symmetry, 11(9):1066, 2019.
[12] Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. Advances neural information processing systems, 27:217772185, 2014.
[13] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018
[14] Christopher Manning and Hinrich Schutze. Foundations of statistical natural language process- ing. MIT press, 1999.
[15] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. Yoshua Bengio and Yann LeCun, editors, 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013.
[16] Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 31 3111-3119, 2013.
11

---

[17] George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11):39-44 1995.
[18] Boris Muzellec and Marco Cuturi. Generalizing point embeddings using the wasserstein space of elliptical distributions. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 10258-10269, 2018.
[19] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Alessandro Moschitti, Bo Pang. and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543. ACL,, 2014.
[20] Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
[21] Mohammad Taher Pilehvar and José Camacho-Collados. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 1267-1273. Association for Computational Linguistics, 2019.
[22] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning. pages 1530-1538. PMLR, 2015.
[23] Sidak Pal Singh, Andreas Hug, Aymeric Dieuleveut, and Martin Jaggi. Context mover's distance & barycenters: Optimal transport of contexts for building representations. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Muchine Learning Research, pages 3437-3449. PMLR, 2020.
[24] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Operations Applications, volume 11006, page 1100612. International Society for Optics and Photonics, 2019.
[25] Chi Sun, Hang Yan, Xipeng Qiu, and Xuanjing Huang. Gaussian word embedding with wasserstein distance loss. CoRR, abss/808..n 8080070066, 2018.
[26] Fei Tian, Hanjun Dai, Jiang Bian, Bin Gao, Rui Zhang, Enhong Chen, and Tie- Yan Liu. A probabilistic model for learning pulti-prototype word embeddings. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 151-160, 2014.
[27] Ran Tian, Naoaki Okazaki, and Kentaro Inui. The mechanism of additive composition. Mach. Learn., 106(7): 108371)0033-1130 2017.
(28) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 1998-6008 2017.
[29] Luke Vilnis and Andrew McCallum. Word representations via gaussian embedding. In Yoshua Bengio and Yann LeCun, editors. 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] All claims are supported by experimental results.
(b) Did you describe the limitations of your work? [Yes] The longer training time (~10
hours) could be a limitation, which we have described in Section 4.3
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We did not recognize any negative societal impacts of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] have.
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Code, data, and pretrained models are in the supplementary materials.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.3
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [N/A] We did not report error bars because would be too computationally expensive.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] The datasets wacky and WordNet were cited. As for text8, we were unable to find a proper way to cite the creators.
(b) Did you mention the license of the assets? [Yes] BERT-large Word2Vec: Apache License 2.0; Word2Gauss: MIT; Word2GM: BSD 3; Wacky: CC Attribution- Noncommercial-S
Alike 4.0 International; text8: unavailable; WordNet 3.0 license.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We included subset of the Core WordNet dataset, which we used evaluate polysemy.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] The new dataset was a subset of WordNet, which is open.
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [N/A] The datasets do not contain personally identifiable information.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "ZZDQENA8Xv", "reviewer": "Reviewer_J2V2", "summary": "This paper proposes to model polysemy by modeling a word as a tuple\nconsisting of a set of locations in a field and a nonlinear function.\nCompared to other methods for modeling polysemy, this approach also\nprovides a mechanism for composing word representations. The authors\nshow that the representations can be meaningfully composed, and that\nthe method outperforms baselines on word-sense prediction tasks.", "questions": "1. Is there any reason that the method is not compared to prior method for polysemous word representations in section 8?\n2. Is it possible to cluster words using the proposed similarity function, rather than Euclidean distance, in the DBSCAN algorithm?\n3. How is the number of word locations chosen?\n4. Can you provide more intuition for the choice of the Jacobian as the similarity function?", "limitations": "The authors largely address the limitations of this method. It would be good to add a reference for the number of hours required to train the equivalent baseline methods (lines 184-186).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. This paper proposes an interesting and unusual new perspective to\n   the task of modeling polysemy. I am not aware of prior work using\n   this kind of approach, and I find it to be thought-provoking and\n   think it could be of general interest.\n\n2. The paper is written in a straightforward style and is easy to\n   read.\n\n3. The experiments illustrate that the resulting embeddings capture\n   word similarity, polysemy, and compositionality. In particular, I\n   find Figures 1 and 3 to be interesting, illustrative examples of\n   the type of representation provided by this approach.\n\n**Weaknesses**\n1. I am not entirely convinced by the motivation for this approach.\n   The authors write that \"The two most important semantic\n   requirements for a word representation are compositionality [...]\n   and polysemy\" (line 15). I think many readers would dispute this\n   claim: the most important requirement for a word representation is\n   that similar words are mapped to similar embeddings, which is the\n   property that enables generalization from small datasets.\n   (The method is also evaluated on semantic similarity in section 6.)\n   \n   More to the point, it is unclear to me why we would want to compose\n   polysemous word representations. While I find the problem to be\n   interesting, it is hard for me to imagine a practical setting where\n   it would be desirable to compose the representations for \"river\"\n   and \"mortgage,\" or to compose the word \"bank\" without first disambiguating the intended sense.\n   \n2. Evaluating polysemy (section 8): Is there any reason that the\n   method is not compared to prior method for polysemous word\n   representations?\n   \n3. DBSCAN algorithm: The DBSCAN algorithm clusters words using the\n   Euclidean distance. This seems somewhat inconsistent with the\n   similarity function proposed in this paper. It might be interesting\n   to use a clustering method that is based on the proposed similarity\n   function, e.g. a spectral clustering algorithm.\n   \n4. How is the number of word locations chosen? Compared to\n   probabilistic methods, it seems less straightforward to determine\n   the number of locations per-word from the data.\n\n**Minor comments**\n1. It would be helpful to include additional references\n   on fields and potentials, including a general reference and a\n   references to any prior work applying these ideas to\n   representation learning.\n2. The variable L is used in the caption of Figure 1 but not\n   introduced in the text until section 4, so that part of the caption\n   is difficult to understand on the first reading.\n3. Motivation for using the Jacobian: The authors write that using the\n   Jacobian of the neural network is a \"more efficient way to produce\n   the potential function\" (line 142). In what sense is it more\n   efficient? Is it possible to provide another intuitive\n   justification for this choice of similarity function?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NCgvh0R9tNo", "reviewer": "Reviewer_awZz", "summary": "The authors presented FIRE,  a semantic field representation that is defined on a D-dimensional space, with a field of word interactions. I find the method novel and innovative. Authors also conducted relevant analyzes that solidify their theories. In my opinion, the paper is well written, sounds and flows.", "questions": "- I was wondering if the authors have a more strong explanation of table 3. I would really like them to bring more structural explanation rather than the # of parameters\n- It would have also been great for the authors to discuss the limitation of their method. For instance, they used (D, K, L) = (2, 4, 1). How well will FIRE perform in higher dimensions and with larger Ks (number of positions/locations) and Ls? This would have made the paper stronger\n- Has the authors explored other types of functions for f()? Dirac function is great but what about for instance the Dirichlet function with is a `non-step` function?", "limitations": "See Questions", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is well written\n- The formulas are well explained, simple, and easy to follow\n- The implementation of FIRE is also explicitly described.\n- Relevant experiments have been conducted and are convincing of the approach. FIRE captures polysemy (better than BERT and Word2Vec) which is also a good thing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DsLbbSzgaAg", "reviewer": "Reviewer_A5Nv", "summary": "The authors proposed FIRE (FIeld REepresentation), in which each word is represented by a pair of points in the embedding space (word sense) and scalar fields in the embedding space (contextual information). Polysemy is naturally represented, and sentence representations can be computed directly from the representations of the constituent words (Eq. 4).\n\nExperiments were conducted on the tasks of word similarity, surrounding context similarity, and estimation of the number of word senses. The proposed method was found to be effective to a certain degree.", "questions": "I found both the statement that polysemy is represented by a point cloud (line 34) and the statement that polysemy can be represented by a potential function (e.g., Figure 1). Which is the intent of the authors?", "limitations": "This would be one of the limitations if the proposed method does not scale to practical dimensions such as 300 or 768. This is also mentioned by the authors in the checklist.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* The attempt to express the meaning of a word as a continuous function on support is both novel and fascinating.\n\nWeaknesses\n\n* Unfortunately, this paper is ill-organized and costly to read.\n* The experimental setup is limited, and it is difficult to say that the value of the proposed method has been evaluated. The task used in sentence similarity is not standard (the standard one is STS), and the baselines used are only raw word2vec and BERT. The same problem could be seen in word sense experiments.\n* The references and allusions of related work are inconsistent. For example, [21] is one of the most directly related studies because it represents words as a point cloud (polysemy) and proposes a compositional method for sentence representation using barycenter (compositionality; line 64). But this method is not compared in the paper (line 73). In contrast, word2vec [13] also computes sentence representations in terms of barycenter of word vectors and is always compared with the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9IRAEgEk1E6", "reviewer": "Reviewer_GMJr", "summary": "This paper proposes an alternative to the standard linear vector spaces that encode word embeddings. The authors propose FIRE (FIeld REpresentation), in which words are represented as a set of locations and a nonlinear function defined on a field. This enables word representations to model the linear nature of compositionality and the nonlinear nature of polysemy. Following existing work on word and sentence representations, the authors propose methods to calculate word/sentence similarity and to compose word/sentence representations. They also present an efficient way to implement the potential function of a word, using the Jacobian of an MLP, and modify the skip-gram negative sampling optimization function to train their model. FIRE is evaluated on word similarity, word disambiguation, and polysemy prediction. Its performance is compared to existing methods, including Word2Vec and contextual BERT embeddings.  The results show that on word similarity and disambiguation tasks, FIRE performs similar to existing methods. On polysemy prediction, FIRE outperforms the baselines, showing that its nonlinear nature allows it to better model polysemy.", "questions": "1. Much of your paper is motivated by the claim that compositionality is a linear quality. What is this claim based on? Transformers are currently state-of-the-art for sentence composition and they are nonlinear.\n\n2.  In Table 1, why do you mark Word2Vec as not being effective with small D? One of the persistent advantages of Word2Vec is its low dimensionality.\n\n3. For the experiments with FIRE in Section 8, why do you have to extract and cluster the representations of 1000 similar words? Would it not be possible to obtain a prediction of the number of senses by just using the potential function of a word? To me this would be more convincing proof of FIRE’s value - if the degree of polysemy is encoded in the word representation itself.", "limitations": "The authors should consider adding more discussion on the limitations of their model. It should be made more clear in which contexts FIRE could be useful and in which in couldn't.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "FIRE is a nice addition to the word and sentence representation literature. The technique is well motivated from a mathematical perspective and quite distinct from existing methods. However, I do not think the results are strong enough to support the introduction of such model complexity, when compared to the simplicity of Word2Vec and the strength of BERT. Furthermore, I think there are stronger baselines that the model should be compared to, especially for sentence encodings. If more baselines are added and more sophisticated analyses of results are performed, I believe this paper could be a strong submission to a computational linguistics venue.\n\n**Strengths:**\n\n1. FIRE is quite novel and an interesting divergence from the standard approach to encoding words and sentences. As the authors point out, there are many advantages to using locations and nonlinear functions to represent words. \n\n2. The authors build their work on mathematically solid foundations, which they explain and lay out well. The similarity and compositionality calculations are not obvious extensions of FIRE - these are good ideas and they are presented clearly. Furthermore, training such a model is challenging. The efficient Jacobian-based training algorithm is another valuable contribution of this paper.\n\n3. The whole paper is well structured and clearly written. The authors manage to concisely present ideas that are sometimes mathematically quite dense.\n\n**Weaknesses:**\n\n1. The results are not very strong and require further analyses. Achieving performance that is similar to Word2Vec and BERT is impressive, but not sufficient given the complexity of FIRE. If a new model is proposed that is much more complex than existing methods, that complexity must be justified through convincing results. The results seem to suggest little gain over existing methods. I would suggest more experiments that focus on polysemy, since this seems to be the strength of FIRE.\n\n2. More analysis should be done to interpret and investigate the results. If performance metrics alone are not convincing, a good case can still be made for FIRE based on analyses that demonstrate how the model is effectively modeling certain aspects of language. For example, Figure 2 shows that FIRE models polysemy well. More such results should be reported. I suspect FIRE is also advantageous from an interpretability perspective e.g. perhaps it is possible to extract different senses from FIRE.\n\n3. More baselines, and stronger baselines, should be included in the results. On the word-level tasks, techniques like GloVe and FastText would be useful for comparison. On the sentence-level tasks, FIRE should be compared to state-of-the-art sentence embedders like InferSent, SentenceBERT, and Universal Sentence Encoder, as well as different word embedding composition techniques, besides weighted summation (e.g. element wise multiplication, averaging, matrix multiplication). I understand that these methods are contextual, but they are still the benchmark against which new sentence embedding methods should be compared.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept