Neural Embeddings for Text
Authors: Oleg Vasilyev
John Bohannon
Venue: Submitted to ICLR 2023 (2023)
Keywords: text embedding
semantic embedding
neural embedding
neural text representation

Abstract: We propose a new kind of embedding for natural language text that deeply represents semantic meaning. Standard text embeddings use the vector output of a pretrained language model. In our method, we let a language model learn from the text and then literally pick its brain, taking the actual weights of the model's neurons to generate a vector. We call this representation of the text a neural embedding. With analysis of its behavior on several datasets, we confirm the ability of this representation to reflect semantics of the text. We also compare neural embeddings with GPT sentence (SGPT) embeddings. We observe that neural embeddings achieve comparable performance with a far smaller model, and that the embeddings respond to semantics differently.

Under review as a conference paper at ICLR 2023
NEURAL EMBEDDINGS FOR TEXT
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a new kind embedding for natural language text that deeply repre- sents semantic meaning. Standard text embeddings layers of a pretrained language model. In our method, we let a language model learn from the text and then literally pick its brain, taking the actual weights of the model's s neurons to generate a vector. We call this representation of the text a neural embedding. We confirm the ability of this representation reflect semantics of the text by an analysis of its behavior on several datasets, and by a comparison of neural embedding with state of the art sentence embeddings.
the outputs from hidden
INTRODUCTION
Capturing the semantic meaning of text as vector is fundamental challenge for natural language
processing (NLP) and an area active research (Giorgi et al. 2021 Zhang al., 2020 Gao et al.
2021
Huang et al. 2021 Yan et al. 2021 Zhang et al., 2021 Muennighoft 2022; Alexander Liu
2022
Chuang et al. (002)). Recent work has focused on fine-tuning pretrained language models
with contrastive learning, either supervised (e.g. Reimers & Gurevych 2019); Zhang et al. (2021);
Yan et al. (202T) or unsupervised (e.g. Giorgi et (2021)) et al. (02TT). The embedding is
generated by pooling the outputs of certain layers of the model processes a text.
Motivated by the need for deeper semantic representations of text, we propose a different kind of embedding. We allow language model to fine-tune text input, and then measure the resulting changes to the model's own neuronal weights as a neural embedding. We demonstrate that neural embeddings do indeed represent the semantic differences between samples of text. We evaluate neural embeddings on several datasets and compare them with several state of the art sentence embeddings. We observe that neural embeddings correlate better specifically with semantics, while being comparable other evaluations. We find that neural embeddings behave differently from the known embeddings we considered. Our contribution:
1. We propose new kind of text representation: neural embeddings (Section 2
2. We evaluate embeddings by using several datasets and several criteria (Section We show that by these criteria the neural embeddings are (1) better correlated with semantic similarity and consistency, and (2) strongly differ by the errors they do and by how they represent the qualities of the text.
NEURAL EMBEDDING METHOD
The technique for generating neural embeddings is using micro-tuning, first introduced for the BLANC-tume method of document summary quality evaluation Vasilyev et al. (2020). It is tuning on one sample only, and the tuned model is used for the sample only. Tuning a pretrained model on a specific narrow domain is common practice to improve performance. Micro-tuning takes this to extreme, narrowing down the 'domain' to 'dataset' consisting of just one sample.
For each text sample, start with the original language model and fine-tune only few selected layers Lo, ..,,, Iy while keeping all other layers frozen. Once the fne-tuning on the text sample complete, we measure the difference between the new weights W and the original weights W;
githhb url will be provided here. The code is in supplementary material. thtpp://github..co fhtps:/gituu..om//Pieerr wwt../cimmm//lllllaa...

---

Under review as a conference paper at ICLR 2023
embedding
neural embedding
neural network
neural network
text
text
subtract original
inference
micro-tuning
Illustration comparing the usual output embeddings (left) and neural embeddings (right). The output embeddings are taken from aggregated outputs of certain layers inference, shown the left. Neural embeddings are taken from weights of certain layers at micro-tuning.
Figure 1:
of each layer Z.j and normalize the resulting vector. We obtain the neural embedding of the text by concatenating the normalized vectors:
Ec Eel
E
E. 1==o:: WWJ//w Wyl
(I)
Here the symbol means concatenation, e.g. | is a concatenation of do, 01, ... Om- Schemat- ically, illustration is in Figure For clarity, the algorithm is shown in Appendix A g
Figure
For example, if we select three layers from the standard BERT base model, and if each selected layer has 768 weights, then the resulting embedding will have size 768 * 3 = 2304. Before entering Equation| 1 the weights of each layer flattened from their possibly multidimensional tensor form.
Through this paper we use the pretrained transformer Devlin et al. (2019) model bert-base-uncaser from the transformers library Wolf' et al. (2020). We found that layers either from the top of the model, or from the last transformer block of the model perform best. the next section we use the following selection (see Appendix A by the notations of the huggingface transformers library
1. Lo cls. dss.predictioos.o trans transform frrm. Lorm.Layer
N N weight
2. L1 cls. dds.predictions. irtiiiiiiiios..nnss....... trans orrm.Layer N Norm.bias
3. L2 = cls. dls.predictions..
wtw.iiooss..s.s.......... trans orm.dense.hin bias
The micro-tuning task is similar to its pretraining objective, the masked token task. In order to keep the tuning to few epochs with few masking combinations, and to avoid the randomness of the masking, chose periodic masking strategy, wherein the masking (or absence of masking) repeats at every Pth token. Consider the masking blueprint (ks m). P = k + m. To obtain an input from the text, we keep the first tokens of the text and mask next tokens, repeating this pattern to the end of the text. We then generate our second input by shifting the pattern by I token, followed by shifting tokens, and so on up to l: m tokens. Moreover, we can create inputs from not one several blueprints: [(K], m1).... ..., (kn, m). For clarity, this algorithm for creating inputs is shown in Appendix A Figure 10
All the inputs, randomly shuffled, constitute a 'dataset' for the micro-tuning, In our evaluations use 10 epochs micro-tuning with learning rate 0.01 and a mix of the simplest masking blueprints: [22, 1), (1, 1), 21, 3),, which results in single batch of no more than 12 inputs for any text fitting into model maximal input size. See Appendix B about the processing time and the factors to reduce it.
Our ablation study, by removing a layer (Appendix C.1 and by removing masking blueprint (Appendix C.2 shows relative importance of the layers and the blueprints.
2

---

Under review as a conference paper at ICLR 2023
3 EVALUATIONS
The goal of this section is to demonstrate that neural embeddings do capture the semantic differences between texts, and that average the texts with similar meanings get close to each other in the neural embedding space. No embedding can be ideal for all purposes; here we attempt to observe the behavior of neural embeddings different data and from different points of view.
We are also comparing neural embedding with the following existing sentence embeddings:
1. all-mppet-hase-v2 and all-Minil.M-L.6vv2 4
2. LaBSE Feng et al. (2022 F
3. SGPT- 125M and SGPT-5.88
Muennighofi 2022 9
10
3.1 CORRELATIONS WITH SEMANTIC, LEXICAL AND SYNTACTIC SIMILARITIES
In this section we review how neural embeddings correlate with semantic, lexical and syntactic similarities of paraphrases. The separation of these three qualities is possible by utilizing the quality- controlled paraphrase generation 12 introduced in Bandel et al. (2022). We generate paraphrases with different degree of similarity to the original phrase. By changing one generation control parameter (through values 0.05, 0.10, 0.15, .... 0.95) and keeping two other fixed (with values selected from 0.1, 0.3, 0.5, 0.7, 0.9), we vary one specific similarity (semantic lexical or syntactic). while keeping two others fixed. As our original phrases we take first sentences from the first 100 pairs of phrases from MRPC train dataset- Microsoft Research Paraphrase Corpus Dolan & Brockett (2005
Dolan
Our own embedding based similarity score of a generated phrase is simply a scalar product of the neural embeddings of the generated phrase and of the original phrase. For clarity Appendix D contains more detail. The correlations obtained for the generated phrases are shown in Figure 2 Each cell value in the figure represents a correlation between the embedding- based score and the generation 'score" (control parameter value of the generation). For example, the top right corner of the first (left) heatmap in Figure 2 has a dark-blue (high) correlation value; in this case the semantic and syntactic similarities were kept constant high value 0.9, while the lexical similarity has been varying.
All the correlations are positive, and are especially strong for lexical and semantic qualities. The cor- relations are also positive for the existing usual kinds of embeddings that selected for comparison. Since all the correlations are positive, we can visualize and compare the goodness of neural embed- dings usual embeddings by the ratio of neural embeddings correlations to the usual embeddings correlations. Figures] 3 5 6 and 7 show such comparisons with several known embeddings, in scale intentionally cut to the same interval (0.5, 1.5).
Accordingly to these figures, the embeddings-basee scores that use neural embeddings correlate better with semantic similarity and with lexical similarity. Other embeddings work better for syntactic similarity. Stronger correlation with semantic similarity is unambiguously good for an embedding. Lexical and syntactic similarities are supposedly (at least approximately) untangled from semantics, so a stronger correlation with them may be considered as a desired or not desired feature, depending on downstream goals.
MRPC sentences are typically long. In appendix E we repeat our evaluation on short sentences, observing even better performance of neural embeddings in comparison to the other embeddings.
htng::/www..bbrtte/dddcssptttte models.html
"utpss//uuggingiace../eeeeeeeeee wtwp:///wwwcoooceeeeetttannnnnne
" futtss//tttbbbeee///ogg//////122
10 htapss///tthbbbcmmmmmmmmmiiiih/p
3

---

Under review as a conference paper at ICLR 2023
lexical
syntactic
semantic
09
09
syntactic
lexical
lexical
05
0.1
01
0.00
0.3
0.7
0.3
0.7
0.1
semantic
semantic
syntactic
Figure Evaluation of neural embeddings on generated phrases with controlled similarity. The original phrases from MRPC. Each cell represents Kendall Tau (variant c) correlation of the embedding-beee score with a selected kind of similarity: lexical similarity on the left heatmap, syntactic similarity the middle heatmap, and semantic similarity the right heatmap. The X and axes show the similarity by the fixed qualities.
lexical
syntactic
semantic
0.
07
07
-22
syntactic 17
lexical
03
03
0.8
01
01
semantic
semantic
syntactic
Figure 3: Ratio of correlations of neural embeddings correlations of all-mpnet-base-v2 embeddings.
3.2 TRIPLETS ANCHOR- POEIT-PI--IIAIAAIIIII
If we compare the semantic similarity between texts and C' with regard text A. and if B is semantically closer to A, then we would want the scalar product of the embeddings of B and A to be higher than the product of the embeddings of and A:
sim (A, B) sim(A,C)
(EA* En) (Ea Ec)
In this section will take triplets of texts with the similarity sim satisfying sim (A, B) > sim(A, C*) from several datasets, and we will count the fraction of triplets for which Equation D not satisfied.
We can get the triplets (A, B, C) from any dataset of texts grouped by semantic similarity, so that semantically similar texts belong to the same group Such selection of triplets was used for grouped images in Wang al. (2014); Hoffer & Ailon (2015) and in related computer vision works. Selecting 'anchorr and "positive' from the same group, and "negative" C from any other group provides (FA Ec). With dataset in hand, we create all possible
test: We should verify (FE EB) > triplets and then count the fraction of the triplets for which this condition is not satisfied. The datasets we use, listed with the short notations of Tables and 2
1. mrpe: Composed of all sentence pairs that were annotated as similar in the 'train' dataset of MRPC- Microsoft Research Paraphrase Corpus Dolan & Brockett (2005 [44 There 2474 such pairs, hence we have 2474 groups, each group consisting of two texts (sentences).
2. sts: Composed of all sentence pairs that were annotated with similarity score at least (on the scale 1-5) in STS 'test" (Englsh language) dataset (Cer et al. 2017 May 2021 15 There are 338 such pairs, hence we jave 338 group, each group consisting two texts (sentences).
"https//hugginngfac. hhtnss//hngigggacccooooaassssses 1 hhtps://uuginggface http:://uuuginnnnnec..ouuuussssh co/daaasets/ssth multi
4

---

Under review as a conference paper at ICLR 2023
lexical
syntactic
semantic
09
03
1.4
syntactic
lexical
0
0.
0.3
1.0
03
01
0.6
0.1
0.3
0.5
0.7
09
0.1
0.3
0.5
07
0.
0.1
0.3
0.5
07
0.9
semantic
semantic
syntactic
Figure 4: Ratio of correlations of neural embeddings to correlations of all-MiniLM-L6-12 embed- dings.
lexical
syntactic
semantic
09
0.
07
-12
syntactic
lexical
lexical
05
05
03
0.8
01
01
0.3
0.1
0.
0.3
077
semantic
semantic
syntactic
Figure 5: Ratio of correlations of neural embeddings to correlations of LaBSE embeddings.
3. catasaurus: First 5000 pairs of
paraphrases
from the dataset
catasaurus wwtpsrsspaappppaaaaataa
4. asset/r: Subset 'test' of the dataset asse simplified sentences.
Alva-Manchego et al. 2020 the dataset of
5. cestwe/t: 5000 samples from the subset 'tapaco" of the dataset cestwc/paraphras dataset. The samples selected as 500 classes (by labels), with 10 first texts selected for each class.
6. cestwely: Composed as cestwe/t, but using the subset "quora'.
7. cestwe/c: Composed as cestwc/t, but using the subset 'coco'.
8. sumingg: 1700 generated summaries of the dataset SummEva Fabbri al. (2021). Each summary is labeled by the document for which was generated; there are 100 documents, generated summaries for each document.
9. summ-r: 1100 human-written summaries of the dataset SummEval. Each summary is labeled by the document for which it was generated; there are reference summaries for each document.
10. text-summ-g: This dataset composed by adding the texts of the documents to the dataset summ-g, but with restriction that only the document texts serve A, and only the summaries serve as and C.
11. text-summ-r: Composed lext-summ--g, but with human-written reference summaries instead of the generated summaries.
The above datasets diverse: besides paraphrased sentences, there are paraphrased questions (cestwc/quora), sentence simplifications (asset), related summaries created from the same text (summ- g and summ-r), and related text-summary pairs (text-summ-g and text-summ-r). In order to have
17 "happs//giihuu..oo//cceeeee http://hhhbbbcommmecccooeeeeeees
"httpss/hugggingfcce.oo https:/hhhgggggfaaccc.o/aassssss htt.://ngw......ooooo//c///////n
19 http:://gituuh..ooaalllliiini www.oo/mmaallll//.ss/....

---

Under review as a conference paper at ICLR 2023
lexical
syntactic
semantic
09
03
1.4
Q7
syntactic
lexical
0
0.
0.3
1.0
0.3
03
01
0.6
0.1
0.3
0.5
0.7
0.
0.1
0.3
0.5
0.
0.9
0.1
0.3
0.5
07
0.9
semantic
semantic
syntactic
Figure 6: Ratio of correlations of neural embeddings to correlations of SGPT-125M embeddings.
lexical
syntactic
semantic
09
09
1.4
syntactic
-12
lexical
0.5
1.0
03
0.8
0.1
011
0.1
0.3
0.7
0.1
0.5
0.9
0.3
0.7
semantic
semantic
syntactic
Figure 7: Ratio of correlations of neural embeddings to correlations of SGPT-5.8B embeddings.
manageable numbers of triplets, for some known datasets took subsets only parts of them, as specified above.
Tables and the results of our evaluation. The result for 'text-summ-g" is uniquely sensitive to random seed and the count wrong = 44 can shift up to 10 units up or down.
Here considered normalized embeddings (or, equivalently, cosine measure of similarity). Having unnormalized embeddings from the models all-mpnet-base-v2, all-Minii and LaBSE does not make meaningful difference (Appendix
The embedding column shows the kind of embeddings used. Neural embeddings are obtained using the bert-bese-uneased model, the layers listed Section and micro-tuning with the blueprints (2, (1, 1), (1, 2), (1, 3)], 10 epochs and learning rate 0.0T, as described in Section 2
The error column shows the fraction of 'broken' triplets (A, B, C) the triplets for which Eq 2 is not satisfied. The intersect column I shows the fraction of the broken triplets that the considered embeddings have in common with neural embeddings:
I Temb N Tneural
min Tembl | (nnerrat)
Here is intersect, Temb is set of the triplets broken by the considered embeddings, and Tneura! is set of the triplets broken by the neural embeddings. The intersect is not listed for the neural embeddings, for which it equals 1.
The same column shows the average value of the scalar product (EA EB) over all the triplets, ie. the average value of the product between embeddings of the texts which were selected as semantically close (same group). Similarly, the diff column shows the average value of the scalar product (EA+ * Ec) over all the triplets, i.e. the average value of the product between embeddings of the texts which were selected semantically different (different groups).
In most datasets the neural embeddings and the known embeddings behave differently. Their errors largely disjoint, with the overlap below 50% for many datasets in Tables and Another

---

Under review as a conference paper at ICLR 2023
Table 1: Performance of embeddings, measured the ability conform the ABC-relation by 2q Column error is the fraction of broken ABC-relations. Column total is the number of the triplets, and column wrong is the number of broken triplets; error wrong/ trrong/total. /toall. is the intersection with Neural errors, by Eq S Column same is the average value of embeddings product
for semantically similar AB texts; column diff is the product for semantically different AC. The best
and the next-best embeddings marked cyan and green correspondingly.
dataset mrpc
embedding
error
total
wrong
same
diff
Neural
6.1e-5
24472808
1491
0.54
0.02
all-mpnet-besd -2
6.6e-5
1621
0.72
0.85
0.06
all-MiniLM-L6--22
8.3e-5
2024
0.78
0.83
0.06
LaBSE
6.4e-5
1574
0.74
0.83
0.17
SGPT-125M
7.9e-5
1945
0.78
0.86
0.18
SGPT-5.8B
5.6e-5
1365
0.77
0.87
0.13
sts
Neural
1.0e-3
455624
455
0.53
0.03
all-mpnet-base-v2
1.8e-3
815
0.64
0.84
0.03
M-L6--22
2. lel3
945
0.67
0.84
0.03
LaBSE
1.0e-3
462
0.69
0.85
0.15
SGPT-125M
1.0e-3
460
0.69
0.85
0.09
SGPT-5.8B
9.1c-4
414
0.71
0.86
0.06
catasaurus
Neural
7.0e-5
99980000
6998
0.79
0.02
all-mpnet-base-y
6.3e-5
6325
0.88
0.97
0.05
all-MiniLM-L6-v2
6.4e-5
6356
0.89
0.97
0.04
LaBSE
6.7e-5
6655
0.88
0.98
0.21
SGPT-125M
6.3e-5
6308
0.88
0.97
0.19
SGPT-5.8B
6.1e-5
6083
0.89
0.97
0.15
asset/t
Neural
4.8e-4
155511620
74731
0.51
0.01
oll-mpnet-base-vy
1.8e-5
2838
0.05
0.87
0.04
all-MiniLM-L6-v2
1.4e-5
2110
0.44
0.87
0.03
LaBSE
1.3e-4
20305
0.41
0.84
0.14
SGPT-125M
9.7e-5
15028
0.31
0.86
0.13
SGPT-5.8B
2.0e-5
3051
0.57
0.88
0.10
cestwe/t
Neural
4.2e-2
224550000
9509234
0.31
0.03
all-mpnet-base-v2
1.6e-2
3683569
0.37
0.64
0.12
al--MiniLM-L6--22
2.1e-2
4607873
0.43
0.63
0.14
LaBSE
1.3e-2
2972922
0.54
0.76
0.22
SGPT-125M
1.9e-2
4282170
0.42
0.68
0.15
SGPT-5.8B
9.0e-3
2011732
0.46
0.69
0.10
cestwe/q
Neural
1.0e-2
224550000
2280159
0.38
0.07
all-mpnet-base-v2
3.1e-4
69480
0.62
0.81
0.06
all-Minil-M-L6-v22
4.4e-4
98295
0.65
0.79
0.06
LaBSE
2.9c-3
648334
0.44
0.75
0.30
SGPT-125M
2.1e-3
479279
0.43
0.77
0.22
SGPT-5.8B
9,2c-4
205408
0.53
0.79
0.19
cestwe/c
Neural
I.le-1
224550000
24563942
0.23
0.08
all-mpnet-base-v2
3,6e-2
8085033
0.60
0.56
0.10
all-MiniLM-L6-22
3.8c-2
8500814
0.61
0.55
0.09
LaBSE
8.3e-2
18588451
0.53
0.52
0.27
SGPT-125M
4.4e-2
9920519
0.57
0.60
0.17
SGPT-5.8B
3.3e-2
7300297
0.61
0.62
0.15
difference between neural and other embeddings is how they spread out in the embedding space.
The average product of embeddings is higher for all known embeddings than for neural embeddings. This is true both for semantically similar and for semantically different texts see the same and diff columns.
7

---

Under review as a conference paper at ICLR 2023
Table 2: Performance of embeddings, the ability conform the ABC-relation by Eq
dataset
embedding
error
total
wrong
same
diff
lext-summ-g
Neural
1.5e-5
2861100
44
0.42
0.03
all-mpnet-base-wa
3.7e-5
107
0.04
0.83
0.11
all-MimiLM-L6--2
1.le-4
321
0.06
0.81
0.11
LaBSE
2.7e-3
7803
0.55
0.75
0.29
SGPT-125M
8.0e-3
22771
0.47
0.70
0.22
SGPT-5.8B
8.7e-4
2495
0.34
0.77
0.20
text-summ-r
Neural
6.7e-3
1197900
7988
0.16
0.27
0.03
all-mpnet-basee base-v2
2.3e-3
2779
0.27
0.73
0.10
all-MiniLM-L6-v2
2.6e-3
3075
0.27
0.69
0.10
LaBSE
2.3e-2
27884
0.46
0.62
0.22
SGPT-125M
1.1e-2
12967
0.16
0.71
0.21
SGPT-5.8B
4.2e-3
4987
0.16
0.72
0.17
summ-g
Neural
4.4e-3
45777600
203275
0.20
0,45
0.03
all-mpnet-base-v2
2.9c-4
13330
0.31
0.82
0.10
all-MiniL.M-L6-v2
6.3e-4
28986
0.31
0.81
0.10
LaBSE
2.3e-3
106435
0.27
0.78
0.24
SGPT-125M
6.3e-3
287977
0.20
0.78
0.25
SGPT-5.8B
1.Se-3
70610
0.22
0.80
0.19
summ-r
Neural
3.7e-2
11979000
440827
0.34
0.22
0.02
all-mpnet-base-v2
6.4e-3
76309
0.45
0.67
0.09
all-MimiL.M-L6-v2
8.1e-3
97028
0.44
0.64
0.09
LaBSE
1.8e-2
211836
0.39
0.61
0.18
SGPT-125M
1.6e-2
190809
0.34
0.66
0.18
SGPT-5.8B
7.1e-3
85433
0.44
0.67
0.15
3.3 AGREEMENT WITH POSITIVELY AND NEGATIVELY ANNOTATED PAIRS
The datasets containing the pairs of sentences deemed not similar provide another way of evaluation: any similar pair must have higher product of embeddings than any non-similar pair:
sim [A, B) > sim = (EA * EB) > (Ec * ED)
Table JableeCootann contains the results of this evaluation.
Table 3: Performance of embeddings, measured the ability to grade pair of sentences labeled similar higher than pair of sentences labeled non-similar. Column total is the total number of tuples (similar pair, non-similar pair). Column wrong is the number of errors, so that error wrong/ prrong/total. total. Column same is the average value of embeddings product for pairs labeled as similar; column diff is the average product for pairs labeled as different.
dataset mrpc
embedding
total
wrong
diff
Neural all (ll-mpnne--barr base-v2
2.6e-1
2953956
772375
0.62
0.54
0.42
2.2e-
651299
0.58
0.85
0.71
all-MiniLM--L--22
2.5e-1
736308
0.60
0.83
0.71
LaBSE
2.2e-1
637497
0.67
0.83
0.70
SGPT-125M
2.4e-1
695606
0.62
0.86
0.76
SGPT-5.8B
2.1c-1
607056
0.61
0.87
0.75
sts
Neural
9.9e-2
180492
17893
0.56
0.53
0.29
all-mpnet-base-v2
3.3e-2
5889
0.52
0.84
0.33
all-MiniLM-L6-v2
4.0e-2
7216
0.61
0.84
0.37
LaBSE
6.6e-2
11948
0.50
0.85
0.56
SGPT-125M
4.1e-2
7333
0.56
0.85
0.48
SGPT-5.8B
2.0e-2
3626
0.59
0.86
0.42
MRPC 'train' dataset contains 2474 similar and 1194 non-similar pairs. The STS 'test' dataset contains 374 pairs with score 4 or 5 which we used as similar, and 479 pairs with score 2 or 1 which we used non-similar.
8

---

Under review as a conference paper at ICLR 2023
3.4 CORRELATIONS WITH HUMAN SCORES
dataset contains human-scored pairs of texts or sentences, with the score reflecting at least in some sense similarity between the texts, then the product of embeddings should correlate with the score. In Table we show examples of such correlations.
Table 4: Correlations of embedding product with human scores.
dataset
score
embedding
kendall-c
kendall-b
spearman
sts
similarity
Neural
0,478
0,482
0.658
all-mpnet-base-
0.651
0.655
0.834
all-MiniL
0,638
0,642
0,820
LaBSE
0.537
0.541
0.723
SGPT-125M
0.608
0.612
0.795
SGPT-5.8B
0.675
0.680
0.857
summeval
coherence
Neural
0.100
0.097
0.137
all-mpnet-base-v2
0.115
0.112
0.159
all-Minil.M-L.--22
0.096
0.093
0.134
LaBSE
0.113
0.110
0.154
SGPT-125M
0.156
0.151
0.216
SGPT-5.8B
0.169
0.164
0.232
summeval
consistency
Neural
0.098
0.157
0,200
all-mpnet-base-v2
0.075
0.120
0.153
all-MiniLM-LL-v22
0.087
0.138
0.176
LaBSE
0.077
0.122
0.156
SGPT-125M
0.011
0.017
0.022
SGPT-5.8B
0.068
0.108
0.137
summeval fluency
Neural
0.063
0.084
0.109
all-mpnet-basee
0.056
0.075
0.097
all-MiniLM-LL--22
0.064
0.086
0.112
LaBSE
0.055
0.074
0.095
SGPT-125M
-0.004
-0.006
-0.007
SGPT-5.8B
0.023
0.030
0.039
summeval
relevance
Neural
0.190
0.188
0.263
all-mpnet-base-v2
0,200
0.198
0.278
all-MiniLM-LL-v22
0.169
0.167
0.236
LaBSE
0.217
0.215
0.301
SGPT-1251
0.198
0.196
0.275
SGPT-5.8B
0.230
0.228
0.318
The 'sts' (STS test dataset) is scored by similarity of the phrases. The 'summeval' (SummEval dataset Fabbri et al. ))2T)) )) has human-scored, by expert, each of 1700 summaries, which were generated by 17 models for 100 documents. Each summary is scored for coherence, consistency, fluency and relevance (we using the score averaged over the 3 expert scores). Neural embeddings are the best in correlations with consistency, which is a quality that is probably the closest to semantics.
CONCLUSION
In this paper we introduced neural embeddings, a new method for representing text as vectors. To our knowledge, this is the first attempt to obtain representations in this manner. In Appendix G we listed the factors by which the construction neural embeddings differs from the usual embeddings; based on the presented there observations conclude that micro-tuning is the most important factor.
Our observations in Section and and in Appendix show that neural embeddings make more emphasis on semantic and lexical qualities, and less on syntactic qualities. We also observed through Section 3 that neural embeddings behave distinctly differently from all the embeddings we included in our comparisons. As illustration of the difference, in Appendix H show results for simple "ensemble': the concatenation neural and SGPT- 125M embeddings. As expected, in many cases the concatenation outperforms them both.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Samuel Yang Alexander Liu. Masked autoencoders as the unified learners for pre-trained sentence representation. arXiv, arXiv:2208.0022 2022. URL https://arxiy// org/ abs 2208
00231
Fernando Alva-Manchegg, Louis Martin, Antoine Bordes, Carolina Scarton, Benoît Sagot, and Lucia Specia. ASSET: A dataset for tuning and evaluation sentence simplification models with multiple rewriting transformations. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 4668-4679, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v 1/2020. macl-main.424, URL https: /aais//acllannn www.aaoooanntoooo..ooooooooo org/ 2020.
424
Elron Bandel, Ranit Aharonov, Michal Shmueli-Scheuer Ilya Shnayderman, Noam Slonim, and Liat Ein-Dor. Quality controlled paraphrase generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 596-609, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.08866////0022.aclll
URL https: //a//alaatthh ogy. ogg. 2022 acl long. 4.5
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguisties. doi: 10.
URL hatiis::/aclaantoooogyy.oo
Yung-Sung Chuang, Rumen Dangovski. Hongyin Luo, Yang Zhang. Shiyu Chang, Marin Soljacic, Shang- Wen Li, Scott Yih, Yoon Kim, and James Glass. DiffCSE: Difference- contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4207-4218, Seattle, United States, July 2022. Association for Computational Linguistics. URL http:////aaaaaatooooo.oooo////// wtt.:///222.aaaa.....m naacl main.311
Jacob Devlin, Ming- Ming-WWii Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171 4186, Minneapolis, Minnesota, 2019. URL https:/7
Association for Computational Linguistics. doi: (0.18653/w//N191111
ac gellantnology.orr///e wttp
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases.
In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL
https: http::://aaaaatthhho logy. org/ 1000//05550000
Alexander R. Fabbri, Wojciech Kryšciñski,, Bryan McCann, Caiming Xiong, Richard Socher, and
Dragomir Radev. SummEval: Re-evaluating summarization evaluation. Transactions of the
Association for Computational Linguistics, 9:391-409, 2021. URL htttps://boioor///1
1162/tacl a 00033
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostie BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 878-891, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1r653//1/2222.ac-loonggg.
URL
https://aclanthologg
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894 6914,,910, 10, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 211118666/2222111 10023l/1/221.emlppp-p www.ee-mmnnnn....... main.552. URL https //acl logy. o orgg 2021. wwn maalllppaiii n. 552
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. DeCLUTR: Deep contrastive learn-
ing for unsupervised textual representations. In Proceedings of the 59th Annual Meeting of
10

---

Under review as a conference paper at ICLR 2023
the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 879-895, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/11/2011.aclloo
URL
https://acl antho logy. Seggy.orgg22211 .al2111.lllloo.....
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In Aasa Feragen, Marcello Pelillo, and Marco Loog (eds.), Memilarity-Baxedd Pattern Recognition, pp. 84-92, Cham, 2015. Springer International Publishing. ISBN 978-3-319-24261-3
James Y. Huang, Kuan-Hap Hao Huang, and Kai-Wei Chang. Disentangling semantics and syntax in sentence embeddings with pre-trained language models. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1372-1379, Online, June 2021. Association for Computational Lin- guistics. doi: 10. 10.18653/v122011
www.el.nacllmaaa maaeel-mai..05 108. URL https: httpss//accaanttooooo.ooo)
2021 nww.caaccll.mm.m mwt-mmain.11
Philip May. Machine translated multilingual sts benchmark dataset. 2021. URL https://githui. com/ whw.ippaa/sttaa.. mulbii-mttiii. 1mm
Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. arXiv, arXiv:2202.08904, 2022.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP). pp.
Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/D19- 1410. URL htRhstts::/// ac www.aanoooooooooooooooooo..eee..
Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. Fill in the BLANC: Human- quality estimation of document summaries. In Steffen Eger, Yang Gao, Maxime Peyrard, Wei Zhao, and Eduard Hovy (eds.), Proceedings of the First Workshop Evaluation and Comparison of NLP Systems, pp. 10-20. Association for Computational Linguistics, November 2020. doi: 10.18653/ v1/2020.eval4nIpp 1.2. URL http::://aaaaattta/aa /tLpLsss//aclaathoooog//
www.1...........ppp... 1.2
Jiang Wang. Yang Song, Thomas Leung, Chuck Rosenberg. Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. Learning fine-grained image similarity with deep ranking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1386-1393, 2014.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45. Association for Computational Linguistics (2020), 2020. URL http:////acaaattooo..ooo////////
emnlp- demos.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Naturul Language Processing (Volume Long Papers), pp. 5065-5075, Online, August 2021, Association for Computational Linguistics. doi: 10.18653/w1/0211..ac a.cl-oon......
393. URL http::::///aaaathoooggoo org
wwtp:/y....g.200...... 2222 www.222..cc11.. acc-long......... 393
Dejiao Zhang, Shang- Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. Pairwise supervised contrastive learning of sentence representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5786-5798, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18665.... 10.18865/v1/2227 10.18 467. URL https: / ac anthology.........o. org/ 2021 wwn.p-ma 467
Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. An unsupervised sentence embedding method by mutual information maximization. In Proceedings the 2020 Conference on
11

---

Under review as a conference paper at ICLR 2023
Empirical Methods in Natural Language Processing (EMNLP), pp. 1601-1610, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/w1/220.mmmllpl mammmll-mmin... 124. URL
https://aclaathhoo/l wtgp://o...../.2.../ emnlp-main. 124
A
NEURAL EMBEDDINGS AND MICRO-TUNING
In section we defined neural embedding for text. For clarity, the pscudocode is shown in Figure 8
Given:
Model .M
Weights { from selected layers { of M, all other layers always frozen Obtain embedding for text T':
Split To + +...
Each text chunk T; consists of max number of sentences fitting input size of the model M
model layers {Lj} weights to {Wr}
Initialize satputs_and_labell for each chunk T):
Make inputs and labels
Add inputs and labels to inputs and labels
Tune model M inputs Laaddddaabee
Take the tuned weights {Wj} of the layers {Li}.
Calculate changes with respect to original weights: dW, = Wj W
Normalize and concatenate them: E. dw |
Normalize the result: E' Ee/lE.
return
Figure 8: Producing neural embedding for text T. Notice that the weights of layers {45} are always reset to the original values before obtaining neural embedding for text. Here the symbol means concatenation, e.g. | ay concatenation of ag, 01 amn
(1.3)
(1,2)
(1.1)
(2,1)
Figure 9: The inputs for micro-tuning on text with length 6 tokens, obtained by the blueprints [22, 1), (1, 1), (1, 2), (1, 3)]. The tokens kept (not masked) are colored green. Labels are made for predicting the masked tokens. Altogether there are 12 inputs, represented here by 12 rows of length 6.
Our choice of layers for the evaluations presented in Section influenced by several factors. Practically, any 2D layers (for example attention weights) are too large to create an embedding. The decoder bias layer ('els.prent predictiion.decodee bias") 1D but also too large, with a size 30522. reasonable to chose layers from top of the model, when most of the processing is done by the bulk of the transformer layers, and the patterns emerged; this means that the last transformer block and the classification layer the good places pick the layers from. In terms of huggingface BERT, it is the layers bert.encoder.layer I' and 'cls, 'cls, preddctiiion''. Empirically, we found that contiguous set of two three layers the end of these blocks works the best, e.g.
orm. weight 2. L1 cls. predictions.ss.. trans ormm.Layer N Norm.bias
1. Lg = cls prs..pcddiiiooss trans form
3. L2 = cls. d.s.pprdicttion.c wttpiiittiiss.tannsssss fom. bias
1. Lg = ber..encoder.laerr...... wtw.l//yer.....opppuu.. Layer Norm. wiigh
2. L www.encoddrrllyyerr... wow.per......ooot..ooooooooo Layer Norm.iias
12

---

Under review as a conference paper at ICLR 2023
3. L2 = bert.encoder..e www.:encoodr..ayyr.I..... http://oooee...e....ooootttdddee
Taking the same layers from lower blocks worsen the neural embeddings performance. our evaluations used the first choice; in Appendix C.I we present the ablation results with one of the layers excluded.
Figure is an illustration for obtaining the inputs for text of 6 tokens with the blueprints [(2. 1), (1,1), (1, (1,3).. There are only 12 inputs for any text fitting into model maximal in- put size even less than 12 inputs if the text length is less than 4 tokens. This fits into single batch.
For clarity, the pseudocode for obtaining inputs for micro-tuning is shown in Figure 10 In Appendix
we present the ablation results with of the blueprints excluded.
Interestingly, if we do not mask tokens at all (but require prediction for all tokens), we still get reasonable embeddings, albeit with much lower "quality' (if evaluated by any criteria of Section 3) The loss is still defined by cross entropy, no matter how easy the prediction task is, and for the same reason there is always weights difference in Equation
Given:
Blueprints for masking: [(k), mi m1), (kn; mn))
Text chunk tokens T= (1o, l-.-..
Initialize empty list imputs, Landddaabo for each masking blueprint (ki, m4):
Period P = k; + m,
Maximal shift Smaz = min(mi min(P.
for in range(Smaz))
input / deepcopy(T)
initialize labels I,, with length len(L) len(T)
for j in range(len(()) (T)):
offset R = (j 3)P
R >= k;: set label: Líj] = T[j]
mask token: /[j] = mask
Append duple (I, L) impuls annddl_adddddd Indela
return impuls and
Figure 10: Creating inputs and labels from a chunk of text. The given length of the chunk len(T) must not exceed the max allowed size of model input.
B FASTER MICRO-TUNING
For convenience or reproducibility. the neural embeddings can be obtained simply by using the BERT model as it is, with all the layers frozen, except the selected layers, described in Section 2 But there are several simplifications that make obtaining neural embeddings almost as fast as a simple forward run of BERT.
Since the micro-tuning 'dataset' is extremely small, all the inputs can be kept on GPU through all epochs. More importantly, since all the selected (for embeddings) layers are located the top of the model, there is no need to repeat the forward run through all the lower transformer blocks. The hidden states from the last block preceding the layers of interest can be obtained only once, at the first epoch, and then reused in all the next epochs. In result, at each epoch except the first one, the micro-tuning deals only with the top the model (separated from the rest). Again, since the micro-tuning 'dataset' is extremely small, even these hidden states can be kept GPU, unless we dealing with really long text of too many input windows.
For example, using simply the whole BERT model (with 10 epochs), the average time of obtaining neural embedding on GPU P100 for MRPC sentence is 0.24 sec., and for STS sentence 0.16 sec. With hidden states reused, these times become 0.096 sec. and 0.057 sec. Reducing the number of epochs offers another option: Switching from 10 epochs further decreases these times almost twice (with evaluation results in most cases worse only but several percents). However, this is still
13

---

Under review as a conference paper at ICLR 2023
much slower than time around 0.006 0.008 by sall-mpnet-base- 'LaBSE' or 'SGPT-145M' (both for MRPC and STS).
C ABLATION: REMOVING LAYERS AND MASK-BLUBPRINTS
C.1 ABLATION: LAYERS
In this section we present ablation study results for excluding one of the layers from using in constructing the neural embedding (three layers were used in our evaluations, Section 3). We are using the same micro-tuning as in Section 3 but with 5 (not 10) epochs. The Neural embedding is compared in Tables 5 6
and g with the embedding versions Neural-LO, Neural-L1 and Neural-L22 where the suffix indicates the layer that was not used. Reminder, the layers are:
1. Lo = cls. els.preicction trans fransform form. N ormo weight
2. L1 cls pww.ictiioosstteeeee trans form. Layer biass
3. L2 = cls. drs.prddictions..o trans orm.dense.hbe bias
Table 5: Ablation: removal of a layer. Triplets.
dataset
embedding
error
total
wrong
same
diff
mrpc
Neural
7.2e-5
24472808
1757
0.53
0.03
Neural-LO
1.2e-4
2996
0.55
0.01
Neural-L1
7.0e-5
1713
0.51
0.03
Neural-L2
1.6c-4
3905
0.52
0.04
Neural
1.le-3
455624
516
0.53
0.04
Neural-LO
1.2e-3
541
0.56
0.03
Neural-L1
1.le-3
517
0.51
0.04
Neural-L2
1.5e-3
697
0.51
0.05
catasaurus
Neural
7.1e-5
99980000
7090
0.77
0.02
Neural-LO
7.4e-5
7445
0.81
0.01
Neural-L1
7.0e-5
7037
0.75
0.02
Neural-L.2
7.le-5
7123
0.75
0.03
asset/t
Neural
6,4e-4
155511620
99951
0.50
0.02
Neural-LO
6.2e-4
96207
0.52
0.01
Neural-L1
8.2c-4
128230
0.48
0.02
Neural-L2
1.3e-3
196091
0.48
0.03
cestwe/t
Neural
4.4e-2
224550000
9991006
0.30
0.03
Neural-LO
4.6e-2
10253234
0.33
0.03
Neural-L1
4.6e-2
10353332
0.29
0.03
Neural-L2
4.9e-2
10969280
0.29
0.03
cestwe/q
Neural
1.2e-2
224550000
2645701
0.38
0.08
Neural-LO
1.2e-2
2584319
0.41
0.08
Neural-L1
1.3e-2
2835001
0.36
0.07
Neural-L.2
1.7e-2
3711619
0.37
0.09
cestwc/c
Neural
1.2e-1
224550000
25984217
0.23
0.09
Neural-LO
1.1e-1
25249882
0.26
0.09
Neural-LI
1.2c-1
27228191
0.22
0.08
Neural-L2
1.3e-1
30012156
0.23
0.09
and we can conclude that the layer L2 is the most important: the embeddings Neural-L22 never done well in the tables. The layer LO is the least important and maybe can be excluded, depending on task: the embeddings Neural-LO done well in many cases in the tables.
From Tables 5
C.2 ABLATION: MASK BLUEPRINTS
Here in Tables T0 and present results of ablation by excluding masking blueprints from construction of neural embedding. Similar to the ablation study of layers, we compare here Neural embeddings with embedding versions Neural-BO, Neural-B11 Neural-B2 and Neural-B3, where the
14

---

Under review as a conference paper at ICLR 2023
Table 6: Ablation: removal of layer. Triplets. Texts and summaries.
dataset
embedding
error
total
wrong
same
diff
text-summ-g
neural
3.8e-5
2861100
109
0.42
0.03
neural-LO
3.1e-4
897
0.43
0.04
neural-L1
7.7e-5
221
0.39
0.03
neural-L2
3.7e-4
1067
0.38
0.04
text-summ-r
neural
6.9e-3
1197900
8277
0.26
0.03
neural-LO
9.3e-3
11167
0.28
0.03
neural-L.1
9.3e-3
11184
0.24
0.02
neural-L.2
1.5e-2
18101
0.24
0.03
summ-g
neural
5.6e-3
45777600
255665
0.44
0.03
neural-LO
6.9e-3
317828
0.45
0.03
neural-L1
6.6e-3
300512
0.42
0.03
neural-L2
8.2e-3
376727
0.43
0.04
summ-r
neural
3.6e-2
11979000
435033
0.22
0.03
neural-LO)
4.5e-2
540283
0.24
0.02
neural-LI
4.7e-2
565869
0.21
0.03
neural-L2
5,4e-2
652541
0.21
0.04
Table 7: Ablation: removal of a layer. Pairs similar and not similar.
dataset
embedding
error
total
wrong
same
diff
mrpe
Neural
2.6e-1
2953956
772375
0.54
0.42
Neural-LO
2.6e-1
764793
0.55
0.43
Neural-L1
2.7c-1
800874
0.51
0.40
Neural-L2
2.7e-1
804656
0.52
0.41
sts
Neural
1.Ie-2
180492
18975
0.53
0.29
Neural-L
1.0e-2
18742
0.56
0.31
Neural-LI
1.le-2
19569
0.51
0.28
Neural-L.2
1.1e-2
20485
0.51
0.29
Table 8: Ablation: removal of a layer. Correlations of embedding product with human scores.
dataset
score
embedding
kendall-c
kendall-b
spearman
sts
similarity
Neural
0.470
0.473
0.648
Neural-LO
0.471
0.475
0.649
Neural-L.
0.465
0.468
0.642
Neural-1.2
0.453
0.457
0.628
summeval
coherence
Neural
0.104
0.101
0.143
Neural-LO
0.112
0.109
0.154
Neural-L1
0.098
0.095
0.134
Neural-L2
0.098
0.095
0.135
summeval
consistency
Neural
0.092
0.147
0.187
Neural-10
0.087
0.135
0.171
Neural-I.1
0.095
0.152
0.193
Neural-L2
0.084
0.134
0.171
summeval fluency
Neural
0.057
0.076
0.098
Neural-LO
0.053
0.071
0.091
1
0.057
0.078
0.100
Neural-L2
0.050
0.067
0.086
summeval
relevance
Neural
0.194
0.192
0.269
Neural-B
0.202
0.200
0,280
Neural-L1 0.184
0.182
0.254
Neural-L.2
0.178
0.176
0.248
suffix indicates the blueprint excluded from constraction of the embedding. Reminding from Section 2 the blueprints are: [(2, 1), (1, (1,2), (1,3))-
15

---

Under review as a conference paper at ICLR 2023
Table 9: Ablation: removal of mask-blueprint. Triplets.
dataset
embedding
error
total
wrong
same
diff
mrpc
Neural 7.2e-5
24472808
1757
0.53
0.03
Neural-BO
7.9e-5
1932
0.53
0.03
Neural-B1
7.0e-5
1713
0.53
0.03
Neural-B2
7.1e-5
1747
0.52
0.02
Neural-B3
9.5e-5
2334
0.50
0.01
Neural
I.le-3
455624
516
0.53
0.04
Neural-BO
1.2e-3
534
0.53
0.04
Neural-B1
1.0e-3
466
0.53
0.04
Neural-B2
1.4e-3
653
0.51
0.04
Neural-B3
1.3e-3
582
0.50
0.03
catasaurus
Neural
7.1e-5
99980000
7090
0.77
0.02
Neural-BO
7.1e-5
7113
0.77
0.02
Neural-B1
7.le-5
7079
0.77
0.02
Neural-B2
7.2c-5
7170
0.76
0.02
Neural-B3
7.3e-5
7281
0.74
0.01
asset/t
Neural
6.4e-4
155511620
99951
0.50
0.02
Neural-BO
7.3e-4
113772
0.50
0.02
Neural-BI
7.2c-4
111335
0.50
0.02
Neural-B2
7.0e-3
109574
0.49
0.01
Neural-B3
8.2e-4
126789
0.47
0.01
cestwc/t
Neural
4.4c-2
224550000
9991006
0.30
0.03
Neural-BO
4.5e-2
10101417
0.31
0.03
Neural-B1
4.5e-2
10096884
0.31
0.03
Neural-B2
4.7e-2
10530677
0.30
0.03
Neural-B3
5.0e-2
11259016
0.28
0.02
cestwe/q
Neural
1.2e-2
224550000
2645701
0.38
0.08
Neural-BO
1.3e-2
2855655
0.38
0.09
Neural-B1
1.3e-2
2813956
0.38
0.09
Neural-B2
1.2c-2
2626329
0.37
0.07
Neural-B3
1.3e-2
2913397
0.35
0.06
cestwe/c
Neural
1.2e-1
224550000
25984217
0.23
0.09
Neural-BO
1.2c-1
26374181
0.24
0.09
Neural-B1
1.2e-1
26567529
0.24
0.10
Re2
1.2c-1
26059561
0.23
0.08
Neural-B3
1.2e-
26882127
0.20
0.06
The conclusion from Tables 9 10 11 and 12 is simpler than from ablation of the layers. The importance of the selected blueprints not very different, excluding any of them leads worse performance in most cases, yet the blueprint B1 maybe less important than the others. The blueprint B1 is the simplest blueprint (1,1) - one token kept and one token masked.
D CONTROLLED-GENTRATE PHRASES
In Section 3.1
we used control-generated phrases for comparing lexical, syntactic and semantic representation of the phrases by neural embeddings. We reviewed how neural embeddings correlate with semantic, lexical and syntactic similarities of paraphrases. In this appendix we provide more detail. The separation of the three qualities (lexical, syntactic and semantic) is possible by utilizing the quality-controlled paraphrase generation
200 introduced in Bandel et al. (2022). We generated
paraphrases with different degree of similarity to the original phrase. By changing one generation control parameter and keeping two other fixed, we varied one specific similarity (semantic or lexical or syntactic), while keeping two others fixed.
16

---

Under review as a conference paper at ICLR 2023
Table 10: Ablation: removal of mask-blueprint. Triplets. Summaries and texts.
dataset
embedding
error
total
wrong
same
diff
texl-summ-g
neural
3.8e-5
2861100
109
0.42
0.03
neural-BO
3.3e-4
934
0.41
0.04
neural-B1
3.4e-4
971
0.41
0.05
neural-Bi B2
3.3e-4
944
0.39
0.04
neural-B3
6.7e-5
193
0.38
0.01
text-summ-r
neural
6.9e-3
1197900
8277
0.26
0.03
neural-BO
8.0e-3
9535
0.27
0.03
neural-B1
9.1e-3
10894
0.27
0.04
neural-B2
9,4e-3
11283
0.24
0.03
neural-B3
8,9e-3
10602
0.22
0.01
summ-g
neural
5.6e-3
45777600
255665
0.44
0.03
neural-BO
5.9e-3
270434
0.44
0.04
neural-BI
6.1e-3
277742
0.44
0.04
neural-B3
6.2c-3
283341
0.43
0.04
neural-B3
6.2e-3
284366
0.41
0.02
summ-r
neural
3.6e-2
11979000
435033
0.22
0.03
neural-Bi -Bra
4.1e-2
493764
0.23
0.03
neural-Bi -1
4.2e-2
501948
0.23
0.03
neural-Bi B2
4.3e-2
516386
0.21
0.03
neural-B3
5.2e-2
623027
0.19
0.01
Table 11: Ablation: removal of mask-blueprint. Pairs similar and similar.
dataset
embedding
eтror
total
wrong
same
diff
mrpc
Neural
2.6e-1
2953956
772375
0.54
0.42
Neural-BR BO
2.7c-1
793688
0.52
0.42
Neural-B1
2.7e-1
786711
0.53
0.42
Neural-B2
2.7e-1
783147
0.52
0.41
Neural-B3
2.7c-1
796018
0.50
0.39
sts
Neural
1.le-2
180492
18975
0.53
0.29
Neural-BB
1.le-2
19953
0.53
0.30
Neural-B1
1.0e-2
18760
0.53
0.30
Neural B2
1.1e-1
19328
0.51
0.28
Neural-B3 -B3
1.le-2
20024
0.50
0.26
Our own embedding- based similarity of a generated phrase is simply a scalar product of the neural embeddings of the generated phrase and of the original phrase. In Section [3. we calculated correlations between the embedding-basee similarity and the generation control parameter. As original phrases we took first sentences from the first 100 pairs phrases from MRPC train dataset Microsoft Research Paraphrase Corpus Dolan & Brockett (005 following:
Each evaluation defined by the
1. The selected quality (semantic or lexical or syntactic) for which we generate phrases with different degree of similarity, by varying the corresponding control parameter.
2. The fixed control parameters for the remaining two qualities.
We have chosen our fixed control parameters from the values 0.1, 0.3, 0.9. Thus, with 5 x 5 25 choices of the fixed control parameters, and with 3 choices of selecting the quality vary, we have 25 x 3 = 75 evaluations, i.c. 75 correlations to calculate. For each evaluation, we did the following:
1. For each phrase of 100 original phrases, we generated new phrases with the selected control parameter taking the values 0.05, 0.10, 0.15, 0.95 (and with the other two control
17

---

Under review as a conference paper at ICLR 2023
Table 12: Ablation: removal of mask-blueprint. Correlations of embedding product with human scores.
dataset
score
embedding
kendall-c
kendall-b
spearman
sts
similarity
Neural
0.470
0.473
0.648
Neural-BO
0.460
0.463
0.636
Neural-B1
0.467
0,470
0.645
Neural-B2
0,466
0,469
0.643
Neural-B3
0,468
0.471
0.644
summeval
coherence
Neural
0.104
0.101
0.143
Neural-B8 -BO
0,092
0.089
0.126
Neural-B1
0.092
0.089
0.126
Neural-B2 0.097
0.094
0.133
Neural-B3 0.095
0.092
0.130
summeval
consistency
Neural
0.092
0.147
0.187
Neural-BB -BO
0.083
0.132
0.168
Neural-B1
0.082
0.130
0.166
Neural-B2
0.091
0.144
0.183
Neural-B3 0.102
0.162
0.206
summeval
fluency
Neural
0.057
0.076
0.098
Neural-BO
0.047
0.063
0.081
Neural-B1
0.049
0.066
0.085
Neural-B2
0.057
0.077
0.099
Neural-B3
0.066
0.089
0.115
summeval
relevance
Neural
0.194
0.192
0.269
Neural-BO
0.173
0.171
0.240
Neural-B
0.172
0.171
0.240
Neural- -B2
0.175
0.173
0.243
Neural
0.181
0.180
0.251
lexical
syntactic
semantic
0.
0.9
800
0.7
syntacicc
0.5
lodical
0.5
loocall
700
03
0.1
500
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.1
0.3
0.
0.7
09
semantic
semantic
synlactic
Figure 11: Number of control-generatee phrases for lexical similarity (left), syntactic similarity (middle), and semantic similarity (right). The X and axes show the similarity by the fixed qualities. The original phrases are from MRPC.
parameters fixed). increasing of the selected control parameter by 0.05 to the next value does not change the generated phrase, the generated phrase is ignored. The control parameters for the remaining two qualities remain fixed.
2. We obtained the embeddings of the original and of the generated phrases.
3. For each generated phrase, the embedding- -ased score was calculated as a dot-product between its embedding and the the embedding of the corresponding original phrase.
4. A correlation (Kendall Tau, variant e) was calculated between the embedding based score and the generation control parameter.
Since ignored duplicated generation phrases while increasing the control parameter along the values 0.05, 0.10, 0.15, 0.95, there are around 300 900 scored phrases for each evaluation. We show the number of the generated phrases in Figures
and 12
At any choice of the fixed control
18

---

Under review as a conference paper at ICLR 2023
lexical
syntactic
semantic
700
0.
0.9
0.7
600
syntacicc
koxical
looical
0.5
0.5
500
0.3
400
0.1
0.1
0.3
0.5 ssmantic
07
0.9
0.1
0.3
0.5 semantic
07
0.
0.1
0.3
0.5
07
0.9
syntactic
Figure 12: Number of control-generated phrases for lexical similarity (left), syntactic similarity (middle), and semantic similarity (right). The X and axes show the similarity by the fixed qualities. The original phrases are from STS.
lexical
syntactic
semantic
09
03
0.3
07
syntactic
lexical
00
0.
0.5
03
01
0.1
011
01
0.3
0.7
0.1
0.5
0.9
0.3
0.7 syntactic
semantic
semantic
Figure 13: Evaluation neural embeddings on short generated phrases with controlled similarity. The original phrases are from STS. Each cell represents Kendall Tau (variant correlation of the embedding-basse score with a selected kind of similarity: lexical similarity on the left heatmap, syntactic similarity the middle heatmap, and semantic similarity the right heatmap. The X and Y axes show the similarity by the fixed qualities.
parameters, the number of generated phrases is not less than 300. All the correlations shown in Section 3.1 have p-value far below 0.05.
E CORRELATION OF FMBEDDING-BASSE SCORES WITH SIMILARITY OF SHORT CONTROL-GENERATTI SENTENCES.
In Section 3.1 we have considered correlations of embedding- based scores with similarity of sentences taken from MRPC dataset. MRPC sentences are typically long. Here we repeat the same review for shorter sentences.
our original sentences we select from STS test subset (Cer et al., 2017 May 2021 22 100 first sentences of length between 20 and 40 characters, and generate the results in the same manner as was done in Section 3.1 The resulting correlations for neural embeddings are shown
in Figure 13
The corresponding ratios of the correlations of neural embedding-basee score to the correlations based other embeddings shown in Figures [4 neural embeddings have stronger correlations with semantic and lexical similarities, but now also, in comparison with some of the embeddings, stronger correlations with syntactic similarity.
15
16 17 and [8 Again, in Section 3.1
http:://htugggggffccc.oootaaaaae multi
19

---

Under review as a conference paper at ICLR 2023
lexical
syntactic
semantic
09
03
1.4
Q7
syntactic
lexical
0
0.
0.5
1.0
03
01
0.6
0.1
0.3
0.5
0.7
0.
0.1
0.3
0.5
0.
0.9
0.1
0.3
0.5
07
0.9
semantic
semantic
syntactic
Figure 14: Ratio of correlations of neural embeddings to correlations of all-mpnettbase--22 embed- dings.
lexical
syntactic
semantic
09
0.9
144
-12
syntactic
lexical
0.5
0.5
1.0
0.3
0.8
0.1
06
0.1
0.3
0.5
0.7
09
0.1
033
0.5
07
09
033
0.5 syntactic
07
08
semantic
semantic
Figure 15: Ratio of correlations of neural embeddings to correlations of all-MiniL embed- dings.
F UNNORMALIZED SENTENCE EMBEDDINGS
In our evaluations in Section we used normalized embeddings (hence dot-produce equal to cosine). Here we point out that having unnormalized embeddings for the models all-mpnet-base-t all- MiniL-MLL.-v2 and LaBSE does not affect much our observations.
The count of errors (the column "wrong" in Table changes when these embeddings are unnormalized by not more than errors. The only difference for Table that unnormalized embeddings of model all-MiniLMM-L--2 get the count "wrong" 736307 instead of 736308. And the correlations in Table remain the same within the shown precision.
G NEURAL HIDDEN LAYER EMBEDDINGS
Neural embeddings, introduced in Section differ from the commonly used "hidden layer' embed- dings four factors steps:
Micro-tuning: the model gets tuned on extremely small 'dataset', produced from a single sample.
2. Measuring the difference: The embedding is not simply taken from a (micro-tuned) model, but obtained as difference between the embeddings from the micro-tuned model and the original model.
3. Using weights: The embedding is taken not from output of hidden layer, but from the weights of the model.
4. Combining layers: The results from several layers concatenated.
20

---

Under review as a conference paper at ICLR 2023
lexical
syntactic
semantic
09
03
1.4
Q7
syntactic
lexical
0
0.
0.5
1.0
0.3
03
01
0.6
0.1
0.3
0.5
0.7
09
0.1
0.3
0.5
07
0.
0.1
0.3
0.5 syntactic
07
0.9
semantic
semantic
Figure 16: Ratio of correlations of neural embeddings to correlations of LaBSE embeddings.
lexical
syntactic
semantic
09
1.4
0.7
syntactic
lexical
0.
0.5
1.0
03
01
0.6
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
07
0.9
semantic
semantic
syntactic
Figure 17: Ratio of correlations of neural embeddings to correlations of SGPT-125M embeddings.
Do all these three factors necessary? And if yes, how important is each of them? In order to answer these questions, we compare here the neural embeddings with their simplified versions, stripped from one or more of the factors. These are the simplified versions:
1. cls: embedding picked up from CLS token on output of the BERT last transformer block (block 11).
2. avg: average of embeddings picked up from all tokens of the text on output of the last block.
3. cls-tuned: The same "cls', but obtained from micro-tuned model.
4. avg-tuned: The same as "aag', , but obtained from micro-tuned model.
5. cls-diff: Obtained as the difference between 'cls-tuned' and 'cls'.
6. avg-diff: Obtained as the difference between 'avg-tuned" and 'avg'.
neural-1: Neural embedding using just one layer, specifically layer (by huggigface notations) 'bert.encoderrlayer. loutput.dense.ee dentteeeeaees..
transformers
For fair comparison, here the neural embeddings are also taken from block 11 (not from cls layers):
1. Lg bertt.ncooder.lgeer..l....o Layer Norm. weight
2. L1 Bert.encoderrlaeerlll....o Layerr Nore. ormmbias
3. L2 = berl.encoder. wwwp://www.....oouuu..eeeee....
Thus, in our simple collection of embeddings here, the "cls' and "avg' represent the usual embeddings. The "cls-tuned' and 'avg-tuned' represent embeddings that included the first step toward the neural embeddings: micro-tuning. The 'cls-diff' and 'avg-diff' represent embeddings that included the second step toward neural embeddings: measuring the difference. Finally, the 'neural-1" represents the neural embedding with single layer. Notice that selected the most valuable layer, accordingly to the ablation study in Appendix C.I
The results of the evaluations are presented in Tables 13 the same parameters as through the evaluations, Section but here we used 5 (not 10) epochs. The tables show results for normalized versions of embeddings.
14
15
Micro-tuning was done with 16
21

---

Under review as a conference paper at ICLR 2023
lexical
syntactic
semantic
09
03
1.4
Q7
syntactic
lexical
0
0.5
0.5
1.0
03
01
01
0.6
0.1
0.3
0.5
0.7
0.
0.1
0.3
0.5
07
0.9
0.1
0.3
0.5 syntactic
07
0.9
semantic
semantic
Figure 18: Ratio of correlations of neural embeddings to correlations of SGPT-5.8B embeddings.
For evaluations of Table 13 the unnormalized versions of embeddings make less errors only in the following cases, marked yellow: For dataset "catasaurus', the embeddings 'cls-mtuned" make 57149 emors; for dataset 'asset/t', the embeddings 'cls' make 1672558 errors, and embeddings 'cls-mtuned" make 1232707 errors.
15 all unnormalized are worse (make more errors). In Table 16 included unnormalized versions of embeddings (marked by suffix "-u'), in all cases when they do better than the normalized versions.
For evaluations of Tables 14
Overall, from the tables 13 14 15 and 16 we can conclude that all the factors matter, and micro-tuning probably the strongest, most important factor.
H NEURAL AND SGPT EMBEDDINGS CONCATENATED
We observed in Section that neural embeddings and SGPT behave differently: their errors do not overlap greatly, their average products have different ranges, and their correlations with text qualities are different. Here we suggest an additional illustration of this point. We create simple "ensemble" by concatenating neural embeddings and SGPT embeddings into single embedding. An ensemble expected to perform better than either its components if the they perform not too differently but have different properties. Specifically we used here SGPT-125M embeddings.
In Table we observe that this is indeed true. The exception 'text-summ-g' where neural embedding orders of magnitude better, "cestwefe" with almost tie, and, strangely, catasaurus, In [ for 'sts', SGPT was much better to start with. In Table/4 neural and SGPT embeddings almost equally good correlating with relevance, and in that case the concatenation is much stronger than either of them.
Interesting, though evaluation of summary quality not the purpose of the embeddings, neural embeddings correlations for consistency scores exceed correlations of all flavors of ROUGE (ROUGE-L, ROUGB-I, ROUGE-2 and ROUGE-3). Similarly, Neural+SGPT correlations higher than all flavors of ROUGE for relevance, and SGPT correlations are higher for coherence.
22

---

Under review as a conference paper at ICLR 2023
Table 13: Neural embeddings micro-tuned embeddings usual embeddings. Triplets.
dataset
embedding
error
total
wrong
same
diff
mrpc
neural
6.7e-5
24472808
1639
0.54
0.03
neural-1
8.5e-5
2076
0.57
0.04
cls
1.4e-2
333146
0.92
0.75
avg
3.9e-4
9637
0.92
0.62
cls-mtuned
9.0e-3
221189
0.92
0.73
avg-mtuned
2.3e-4
5608
0.90
0.56
cls-diff
3.7c-4
9044
0.56
0.06
avg-diff
2.2c-4
5432
0.58
0.05
sts
neural
1.0e-3
455624
471
0.53
0.04
neural-1
1.le-3
516
0.58
0.05
cls
2.8e-2
12805
0.92
0.73
avg
3.8e-3
1708
0.89
0.55
els-mtuned
2.2e-2
9918
0.91
0.70
avg-mtuned
2.8e-3
1257
0.88
0.50
cls-diff
2.1e-3
975
0.56
0.05
avg-diff
1.7c-3
792
0.58
0.06
catasaurus
neural
7.le-5
99980000
7082
0.78
0.02
neural-1
7.7e-5
7716
0.81
0.02
cls
1.1e-3
107713
0.98
0.63
avg
7.4e-5
7369
0.98
0.51
cls-mtuned
7.6e-4
75988
0.97
0.60
avg-mtuned
7.2e-5
7146
0.97
0.46
cls-diff
7.3e-5
7313
0.80
0.03
avg-diff
7.3e-5
7333
0.82
0.03
asset/t
neural
4.9e-4
155511620
75916
0.50
0.02
neural-1
6.0e-4
93684
0.54
0.02
cls
1.le-2
1745865
0.92
0.69
avg
3.9e-4
60617
0.91
0.52
els-mtuned
8.6e-3
1334770
0.91
0.67
avg-mtuned
2.2c-4
34829
0.89
0.47
cls-diff
3.6e-3
555409
0.52
0.03
avg-diff
2.2e-3
342306
0.54
0.02
cestwc/t
neural
4.1e-2
224550000
9243349
0.31
0.03
neural-1
4.5e-2
10138993
0.34
0.04
cls
6.3e-2
14026873
0.94
0.87
avg
2.8e-2
6174293
0.78
0.53
cls-mtuned
5.3e-2
11884179
0.92
0.84
avg-mtuned
2.4c-2
5370379
0.75
0.48
cls-diff
8.0c-2
17848425
0.33
0.04
avg-diff
5.3e-2
11905894
0.34
0.03
cestwe/q neural
I.le-2
224550000
2389985
0.39
0.08
neural-1
1.2e-2
2582021
0.42
0.09
cls
7.1e-2
15989315
0.94
0.89
avg
1.le-2
2499481
0.85
0.64
cls-mtuned
5.2e-2
11637580
0.93
0.86
avg-mtuned
8.3e-3
1860801
0.83
0.60
cls-diff
3.5e-2
7795495
0.41
0.10
avg-diff
2.1e-2
4623758
0.44
0.11
cestwe/c neural
1.le-1
224550000
25060134
0.24
0.10
neural-1
1.2e-1
224550000
26195509
0.29
0.12
cls
3.1e-1
68860399
0.83
0.79
avg
1.2e-
25818812
0.79
0.67
cls-mtuned
2.8e-1
63727774
0.82
0.77
avg-mtuned
1.0e-
22631189
0.76
0.63
cls-diff
1.6e-1
35766534
0.26
0.11
avg-diff
1.4e-1
32400635
0.30
0.14
23

---

Under review as a conference paper at ICLR 2023
Table 14:
Neural embeddings micro-tuned embeddings usual embeddings. Triplets. Summaries
and texts.
dataset
embedding
error
total
wrong
same
diff
text-summ-g
neural
3.8e-5
2861100
109
0..4
0.03
neural-1
4.1e-
1162
0.42
0.03
cls
5.6e-2
160086
0.81
0.69
avg
1.6e-2
46442
0.91
0.76
cls-mtuned
3.7e-2
106778
0.80
0.65
avg-mtuned
4.7e-3
13322
0.87
0.67
cls-diff
2.1e-3
6051
0,42
0.04
avg-diff
6.5e-4
1854
0.44
0.03
text-summ-r neural
6.9e-3
1197900
8277
0.26
0.03
neural-1
1.le-2
13520
0.27
0.02
cls
9.5e-2
113506
0.78
0.68
avg
5.4e-2
64511
0.86
0.73
cls-mtuned
7.2e-2
85916
0.76
0.64
avg-mtuned
2.7c-2
32740
0.81
0.64
cls-diff
2.9c-2
34862
0.27
0.04
avg-diff
1.9e-2
22154
0.28
0.02
summ-g
neural
5,6e-3
45777600
255665
0.44
0.03
neural-1
8.4e-3
386119
0.46
0.04
cls
4.6e-2
2094610
0.86
0.71
avg
8.2e-3
373988
0.92
0.71
cls-mtuned
3.6e-2
1646157
0.85
0.68
avg-mtuned
5.4e-3
244771
0.89
0.64
cls-diff
1.6e-2
710337
0.46
0.05
avg-diff
1.2e-2
541905
0.47
0.04
summ-r
neural
3.6e-2
11979000
435033
0.22
0.03
neural-1
4.6e-2
548970
0.25
0.04
cls
9.5e-2
1137445
0.83
0.73
avg
3.6e-2
432290
0.85
0.69
cls-mtuned
7.9e-2
942028
0.81
0.70
avg-mtuned
2.7e-2
327481
0.80
0.62
cls-diff
7.9e-2
946548
0.24
0.05
avg-diff
7.2c-2
862239
0.24
0.03
Table 15: Neural embeddings vs micro-tuned embeddings vs usual embeddings. Pairs similar and similar.
dataset mrpp
embedding
сгтоr
total
wrong
same
diff
neural
2.7e-1
2953956
785630
0.54
0.42
neural-1
2.7e-1
784603
0.57
0.46
cls
3.7e-1
1100243
0.92
0.90
avg
2.8c-1
837001
0.92
0.88
cls-mtuned
3.6e-1
1056685
0.92
0.89
avg-mtuned
2.7e-1
788863
0.90
0.86
cls-diff
2.8e-1
830654
0.56
0.45
avg-diff
2.7e-1
796859
0.58
0.46
sts
neural
I.le-1
180492
19194
0.53
0.30
neural-1
1.le-
19686
0.58
0.34
cls
3.5e-1
62899
0.92
0.89
avg
1.9c-1
33342
0.89
0.79
cls-mtuned
3.2c-1
56930
0.91
0.88
avg-mtuned
1.6e-1
28524
0.88
0.77
cls-diff
1.2e-1
21103
0.56
0.33
avg-diff
1.2e-1
21167
0.58
0.34
24

---

Under review as a conference paper at ICLR 2023
Table 16: Neural embeddings micro-tuned embeddings usual embeddings.
dataset
score similarity
embedding
kendall-c
kendall-b
spearman
sts
neural
0.468
0.471
0.646
neural-1
0.469
0.472
0.645
cls
0.137
0.138
0.203
avg
0.346
0.348
0.497
cls-mtuned
0.173
0.174
0,255
avg-mtuned
0.383
0.385
0.544
cls-diff
0.443
0.446
0,618
avg-diff'
0.456
0.459
0.630
summeval
coherence neural-1
neural
0.105
0.102
0.143
0.099
0.096
0.136
cls
0.007
0.007
0.010
avg
0.088
0.085
0.119
cls-mtuned
0.016
0.016
0.022
avg-mtuned
0.086
0.083
0.118
cls-diff
0.069
0.067
0.095
avg-diff'
0.082
0.080
0.113
summeval
consistency neural
0.096
0.153
0.194
neural-1
0.075
0.119
0.151
cls
0.036
0.058
0.073
cls-u
0.047
0.074
0.095
avg
0.079
0.125
0.159
cls-mtuned
0.045
0.072
0.091
cls-mtuned-u
0.052
0.083
0.106
avg-mtuned
0.085
0.135
0.171
cls-diff
0.069
0.109
0.140
avg-diff
0.069
0.111
0.141
summeval fluency
neural
0.058
0.078
0.100
neural-1
0.044
0.059
0.076
cls
0.048
0.064
0.083
cls-u
0.073
0.098
0.127
avg
0.047
0.063
0.081
avg-u
0.063
0.084
0.109
cls-mtuned
0.053
0.071
0.092
cls-mtuned-u
0.076
0.101
0.131
avg-mtuned
0.056
0.075
0.096
avg-mtuned-u
0,070
0.094
0.121
cls-diff
0.034
0.045
0.058
avg-diff
0.042
0.056
0.073
summeval
relevance
neural
0.198
0.196
0.274
neural-1
0.184
0.182
0.255
cls
0.077
0.077
0.109
avg
0.186
0.184
0.258
cls-mtuned
0.088
0.087
0.124
avg- avg-mtuned
0.185
0.183
0.258
cls-diff
0.179
0.177
0.247
avg-diff
0.181
0.180
0.251
25

---

Under review as a conference paper at ICLR 2023
Table 17: Exampe of concatenation, measured against the triplets (Eq 2).
Neural, SGPT-125M, and
the concatenated embedding "Neura+SGPTT' of Neural and SGPT-125M embeddings. A row with
lowest error marked cyan. dataset mrpe
embedding
eгror
total
wrong
same
diff
SGPT
7.9e-5
24472808
1945
0.86
0.18
Neural
6.1e-5
1491
0.78
0.54
0.22
Neural+SGPT
5.7e-5
1394
0.90
0.70
0.10
sts
SGPT
I.0e-3
455624
460
0.85
0.09
Neural
1.0e-3
455
0.69
0.53
0.03
Neural-SGPT
9.0e-4
412
0.86
0.69
0.06
catasaurus
SGPT
6.3e-5
99980000
6308
0.97
0.19
Neural
7.0e-5
6998
0.88
0.79
0.02
NeuraI+SGPT
6.8e-5
6751
0.91
0.88
0.10
asset/t
SGPT
9.7c-5
155511620
15028
0,86
0.13
Neural
4.8e-4
74731
0.31
0.51
0.01
cestwe/t
SGPT
1.9c-2
224550000
4282171
0.68
0.15
Neural
4.2e-2
9509234
0.42
0.31
0.03
Neural+SGPT
1.7e-2
3775736
0.80
0.50
0.09
cestwc/q
SGPT
2.1e-3
224550000
479294
0.77
0.22
Neural
1.0e-2
2280159
0.43
0.38
0.07
Neural+SGPT
2.0e-3
439138
0.66
0.57
0.15
cestwele
SGPT
4.4c-2
224550000
9920527
0.60
0.17
Neural
1.le-1
24563942
0.57
0.23
0.08
NeuraI+SGPT
4.4e-2
9985684
0.82
0.42
0.13
text-summ-g
SGPT
8.0e-3
2861100
22771
0.70
0.22
Neural
1.5e-5
44
0.45
0.42
0.03
Neural+SGPT
3.7e-4
1068
1.00
0.56
0.12
text-summ-r
SGPT
1.le-2
1197900
12967
0.71
0.21
Neural
6.7e-3
7988
0.16
0.27
0.03
Neural+SGPT
4.4e-3
5238
0.94
0.49
0.12
summ-g
SGPT
6.3e-3
45777600
287978
0.78
0.25
Neural
4.4e-3
203275
0.20
0.45
0.03
Neural+SGPT
2.6e-3
118566
0.93
0.61
0.14
summ-r
SGPT
1.6e-2
11979000
190809
0.66
0.18
Neural
3.7e-2
440827
0.34
0.22
0.02
NeuraI+SGPT
1.2e-2
143632
0.86
0.44
0.10
Table 18: Exampe concatenation of Neural and SGPT-125M, Mcasured on pairs (similar vs
similar).
dataset
embedding
error
total
wrong
same
diff
mrpc
SGPT
2.4e-1
2953956
695606
0.86
0.76
Neural
2
772375
0.62
0.54
0.42
SGPP
2.3e-1
685446
0.77
0.70
0.59
sts
SGPT
4.1e-2
180492
73332
0.85
0.48
Neural
9.9e-2
17893
0.56
0.53
0.29
Neural+SGPT
4.8e-2
8735
0,70
0.69
0.38
26

---

Under review as a conference paper at ICLR 2023
Table 19: Exampe of concatenation of Neural and SGPT-125M. Correlations with human scores.
dataset
score
embedding
kendall-c
kendall-b
spearman
sts
similarity
SGPT
0.608
0.612
0.795
Neural
0.478
0.482
0.658
Neural+SGPT
0.592
0.596
0.779
summeval
coherence
SGPT
0.156
0.151
0.216
Neural
0.100
0.097
0.137
Neural+SGPT
0.156
0.151
0.215
summeval
consistency
SGPT
0.011
0.017
0.022
Neural
0.098
0.157
0.200
Neural+SGPT
0.068
0.108
0.138
summeval fluency
SGPT
-0,004
0.006
-0.077
Neural
0.063
0.084
0.109
Neural+SGPT
0.038
0.051
0.065
summeval relevance
SGPT
0.198
0.196
0.275
Neural
0.190
0.188
0.263
Neural+SGPT
0.233
0,231
0.324
27


Reviews:
[{"review_id": "rtYkYhbRr-", "reviewer": "Reviewer_fZyy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on developing a new type of embeddings for texts (sentences or documents, but not words). \nThe proposed method's basic idea is to consider actual weights in the trained models (seem to assume language models).\nThen, the method computes the weight difference between the original and micro-tuned ones (fine-tuning one sample only).\nThe method is evaluated on several benchmark datasets for semantic computing similarity among texts.\nThe experimental results show comparable results with embeddings obtained from GPT-3.\n\n\n\n\n\n", "summary_of_the_review": "Overall, while the originality of the proposed method is high, the advantage of the proposed method seems to be limited.\nThis is because, as shown in the experiments and ponted out by the authors, the performance of the proposed method does not clearly surpass that of SGPT.\nThe authors claim that the proposed method can be compared using a smaller model but seems to require additional computational costs for micro-tuning.\nThis property seems to limit the usefulness of the proposed method.\nTherefore, I lean toward not recommending this paper to be accepted to the conference.", "strength_and_weaknesses": "Strength:\n* The proposed method seems novel.\n* The task this paper tackles is an essential technology in the NLP field.\n\n\nWeaknesses:\n* I agree with the importance of developing a better text representation. However, the paper does not describe the merits of using the proposed method compared with other existing methods, such as SGPT. Therefore, I am wondering what the advantages of the proposed method are.\n* If I do not miss something, there is no explicit discussion of theoretical justification for the proposed method. Therefore, I am not convinced of the effectiveness of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aJAmE4Qvkc", "reviewer": "Reviewer_JM7i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a text embedding that deeply represents semantic meaning,\nand calls it a neural embedding.\nWhile the conventional text embeddings use the vector output of a pre-trained language model,\nthe neural embedding learns the text and literally pick its brain, and takes the actual weights of the model’s neurons to generate a vector.\nMore precisely, it allows to learn model’s neurons from text, and may reflect deeper semantic features of the text.\nAuthors compare GPT sentence (SGPT) embedding over  several datasets and show that 1) the neural embeddings are comparable to SGPT embeddings, and 2) its errors and qualities are often different.\n", "summary_of_the_review": "Although author's motivation is interesting,\nthey do not provide its theoretical basis and show that comparisons are insufficient.\nThis is the reason why I can not appreciate it.", "strength_and_weaknesses": "Strength\\\n*Authors take an empirical and theoretical approach\\\n*Compare the smaller models with the larger model, SGPT\\\n*Human analysis\n\nWeaknesses\\\n*The theoretical background of the proposed approach is weak.\\\n　The theoretical background and reasons why tuning and masking strategies can acquire semantic representation of texts are unclear. \\\n*Authors' improvement is unclear in applying the micro-tuning\\\n　What is the new rationale for the application of a periodic masking strategy in BERT? \\\n　Why you use BERT instead of GPT? \\\n*Insufficient comparison of technical aspects \\\n　Authors apply only the product of the embeddings to compare the semantic similarity  in Eq (2). \\\n　Is the product of embeddings more common than the difference in distance in triplets? \\\n*No ablation analysis \\\n　Is it fair to compare GPT-base (decoder) and Bert-base (encoder) models in the applied task? \\\n    If we don't apply  a periodic masking strategy to GPT-base model,  \\\n    isn't BERT-base models, not the masking strategy, affecting the outcome?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AdBJSdonvf", "reviewer": "Reviewer_gPmr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new way of representing a text as a single fixed-size vector. While most text embedding methods use model activations to represent the input text, the proposed method computes the difference in model parameters that is induced by finetuning on a single text input towards. The finetuning task is a custom masked language modeling objective.\nThe new method is compared to a recent SOTA model that uses activations as embeddings. Through exhaustive text similarity evaluations, the new model is shown to be qualitatively different from the baseline, exceeding it in performance in several regards.", "summary_of_the_review": "The paper presents a worthwhile and novel idea that shows promising initial results against a state-of-the-art baseline. However, only a controlled experiment can support the main claim that neural embeddings learn something qualitatively different from conventional embeddings. Moreover, the reader learns only little about what makes the model perform due to a lack of ablations on e.g. the training objective. I therefore recommend rejection.", "strength_and_weaknesses": "Strengths:\n* sensible and interesting idea, easy to understand\n* comparison on a lot of text similarity benchmarks\n* promising results, as the proposed method seems complementary to the baseline in many regards\n\nWeaknesses:\n* no evaluation of the time complexity in comparison to the baseline. Only the per example time of the proposed method is evaluated (in appendix), but a reference point is missing. This makes it difficult to judge the practicality of the approach, and (probably) hides its high computational cost.\n* no controlled experiment / ablation: BERT is used for creating the neural embeddings, so it would be logical to use averaged BERT embeddings as a baseline, which would constitute a controlled experiment. While comparison to SOTA is nice (as an upper bound), you cannot rule out that the observed qualitative differences between embedding methods are actually due to the underlying model architecture or training data, rather than the way embeddings are created. This weakens the support for the main claim.\n* writing could be improved. There are many places that appear a bit sloppy (but could be fixed easily); details in the Clarity section below.\n* the method includes a masking scheme that deviates from the original BERT masking objective. As no ablations or explanations are provided, it remains unclear how this specific training objective influences the performance of the model. Could it be responsible for the observed qualitative differences?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2yh8i-6Tjk", "reviewer": "Reviewer_SjTe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a new type of embeddings of text. Instead of relying on the output of the neural network, the proposed embedding is generated by the actual weights of the network, when it is tuned with the specific content/text. The work shows that the embeddings generated this way capture semantic differences between text -- texts with similar meanings are closer to each other in the embedding space. \n\n", "summary_of_the_review": "It is an interesting idea to consider the change of neural network weights in the training process as embeddings. Understanding of the semantics represented by the neural network weights is also a fundamental problem of interests to the community.  However the work fails to establish why the proposed method is superior in comparison with conventional embedding methods. ", "strength_and_weaknesses": "Strength:\nIt is an interesting idea to represent text by the actual neural network weights (or the change of it after tuning in specific). \n\nWeakness:\nThe paper fails to establish why this is a desired method, for the following reasons.\n\n1. The benchmarks (ranking triplets) used are quite elementary. Tasks close to real world scenarios should be used (e.g. retrieval on MS MARCO and other tasks in SGPT and related works).\n\n2. It shows that the proposed embeddings method (110M parameters) performs comparably to SGPT (5.8B parameters), but this does not answer the question whether a smaller SGPT model can achieve similar performance, and whether a larger model with the proposed method can outperform SGPT.\n\n3. Conventional embeddings are defined as the output of a trained neural network. So the embeddings can be efficiently generated when running inference. However the proposed embedding requires running training. This can be far more expensive and hard to compute even if the network is much smaller in size. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject