Towards Unsupervised Recognition of Token-level Semantic Differences in Related Documents
Authors: Jannis Vamvas
Rico Sennrich
Venue: EMNLP 2023 Main (2023)
Keywords: recognizing semantic differences
semantic similarity
multilinguality

Abstract: Automatically highlighting words that cause semantic differences between two documents could be useful for a wide range of applications. We formulate recognizing semantic differences (RSD) as a token-level regression task and study three unsupervised approaches that rely on a masked language model. To assess the approaches, we begin with basic English sentences and gradually move to more complex, cross-lingual document pairs. Our results show that an approach based on word alignment and sentence-level contrastive learning has a robust correlation to gold labels. However, all unsupervised approaches still leave a large margin of improvement.

Towards Unsupervised Recognition of Token-level Semantic Differences in Related Documents
Jannis Vamvas and Rico Sennrich Department of Computational Linguistics, University of Zurich {vamvas, sennrich]@cl uzh.ch ch
Abstract
Chinese shares close higher Friday. Syria opposition threatens to quit talks.
L. L'opposition syrienne accepte des pourparlers. Les actions chinoises clôturent en baisse mercredi.
Automatically highlighting words that cause semantic differences between two documents could useful for wide range applica- tions. We formulate recognizing semantic dif- ferences (RSD) as a token-level regression task and study three unsupervised approaches that rely masked language model. To assess the approaches, we begin with basic English sentences and gradually move more complex, cross-lingua document pairs. Our results show an approach based on word alignment and sentence-leve contrastive learning has robust correlation to gold labels. However, all unsu- pervised approaches still leave large margin of improvement. Code to reproduce our experi- ments is available.'
Figure 1: While diff is a common tool for comparing code, highlighting the semantic differences in natural language documents can be more challenging. In this paper, we evaluate on synthetic documents that combine several such challenges, including cross- moss-linguul compar- ison and non-monotonic sentence alignment.
Comparing document similarity with and without a word present the document; 3. Comparing masked language modeling sur- prisal with and without the other document provided as context.
To evaluate these approaches automatically, we convert data from the SemEval-2016 Task for Inter- pretable Semantic Textual Similarity (iSTS; Agirre et al., 2016) into token-level regression task, re- labeling some words better fit the goal of RSD.2 ? We then programmatically create increasingly com- plex variations of this test set in order to study the robustness of the metrics: We add more negative examples, concatenate the sentences into synthetic documents, permute the order of sentences within the documents, and finally add cross-lingual di- mension by translating one side of the test set.
1 Introduction
A pair of documents can have semantic differences for a variety of reasons: For example, one docu- ment might be a revised version of the second one, or it might be a noisy translation. Highlighting words that contribute to semantic difference is a challenging task (Figure 1). Previous work has studied word-level predictions in the context of in- terpretable textual similarity Lopez--azpio et al., 2017) or evaluation of generated text (Fomicheva al., 2022), but did not necessarily focus on se- mantic differences as the main target.
Our experiments show that the first metric cor- relates best to the gold labels, since measuring the alignability of words has a relatively consistent ac- curacy across complexity levels. However, while unsupervised approaches have the advantage of not requiring manual annotations, find that there is considerable gap perfect accuracy, especially for cross-lingual document pairs. Future work could tackle the task by developing supervised models. Besides providing a baseline, unsupervised metrics could also serve as features for such models.
In this paper, we conceptualize the task of rec- ognizing semantic differences (RSD) as a semantic diff operation. We assume that there are relatively few semantic differences and that many words are negative examples. Our goal is to label words us- ing ceff-supervised encoders such as XLM-R (Con- neau al., 2020) without additional training data. Specifically,, we investigate three simple metrics:
1. Performing word alignment and highlighting words that cannot be aligned;
PI iSTS, opposites such as 'higher' and 'lower' are con- sidered similar to cach other, whereas we consider this to he semantic difference worth highlighting.
hhttps://githhi cwt wtt.cccco/eeeceeeeec chhLPP/ recognizing wwwnnniiinn-eenntii-diiiii cemaaniicdiifeeeee dirrencceeeessssee

---

2 Task Formulation
The change in similarity when deleting a, can then serve prediction for a,, which we normalize the range [0, 1]:
The goal of RSD is to analyze two word sequences = 01,-...; On and B = bi,...,bm and to es- timate, individually for cach word, the degree to which the word causes a semantic difference be- tween A and B. For example, given the sentences 'Nice sweater!' and 'Great news!", the correct la- bels would be close to for 'Nice' and 'Great' and close to I for 'sweater' and 'news'.
diff = sim as B) sim(A, B) +
2
Cross-entropy of a Word Encoders such as XLM-R BERT (Devlin et al., 2019) have been trained using masked language modeling, which can be leveraged for our task. Let (ag be the cross-entropy under a masked language model that predicts the token d; given a context A', where as has been masked. By concatenating B and A' into an augmented context BA', we can test whether the additional context helps the language model predict ag. If the inclusion of B does not reduce the cross-entropy, this could indicate that B does not contain any information related a;:
Transformer-based encoders usually process doc- uments as sequences of subword tokens. Our strat- egy is to predict labels for individual subwords and to average the labels of the subword tokens that make up word. To make the notation more readable, use A and B to refer to the tokenized sequences well.
3 Recognition Approaches
Alignability Word The final hidden states of a Transformer encoder (Vaswani et al., 2017) repre- sent A as sequence of token embeddings h(A) = h(ai),.... h(an). In the same way. B is indepen- dently encoded into h(B) h(bi)...., h(bm).
H H(a:) (() - Haa) max H (a) H BAA)
Apmi(a;|AA' BA')
diff (a:) = meaxi0,,mmmiq/ (a:miiaa//'' A'; BA")).
We base our score normalized pointwise mutual information (npmi), with simple transformation to turn it into diff mask. semantic difference score between 0 and 1.
A simple approach RSD is calculate soft token alignment between ((A) and h(B) and to identify tokens that are aligned with low confi- dence. A greedy alignment is usually calculated using the pairwise cosine similarity between hid- den states (Jalili Sabet et al., 2020; Zhang et al., 2020). The prediction for a token ag is then given by:
Evaluation Design
4.1 Annotations
We build on annotated data from the SemEval-2016 Task for Interpretable Semantic Textual Similar- ity (iSTS; Agirre et al., 2016). These data consist of English sentence pairs that are related but usu- ally do not have the same meaning. The iSTS task is to group the tokens into chunks, to compute a chunk alignment and to label the chunk alignments with a type and similarity score.
diffalign(as)) 1 max cos (h(s),, h(b;)). b,CB
(bb)))))),
Deletability of Word Previous work has shown that encoders such as XLM-R may be fine-tuned such that the averages of their hidden states serve useful sentence representations (Reimers and Gurevych, 2019; Gao et al., 2021). The similarity of two sentences can be estimated using the cosine similarity between these averages:
The iSTS annotations can be re-used for our task formulation by labeling each word with the inverse of the similarity score of the corresponding chunk alignment. two chunks are aligned and have a high similarity, the words of the chunks receive a label close to O. In contrast, if two aligned chunks have low similarity, or if a chunk is not aligned to any chunk in the other sentence, the words receive label close Spearman correlation between the gold labels and the predicted labels across all words the dataset. Following iSTS, we do no consider punctuation, i.e., we exclude punctuation when calculating the
sim(A, B) cos(avg(AA) aB)(
cos(
Σ h(a:):
1
h(6,)).
A
|B]
0,EA
We approximate the similarity of a partial sequence
Our evaluation metric is the
A \ where a; is deleted, by excluding the token from the average:
1 h(aa)) avg(B))
sim(A'a B) cos(av)(A)
[A]

---

correlation. We deviate from the original iSTS annotations with regard to chunks with opposite meaning, marking them as differences. Further details are provided in Appendix A.
ing the evaluation metric, only the predictions on the English documents are considered.
Experimental Setup
We concatenate the 'headlines' and 'images' sub- sets of the iSTS dataset into a single dataset. We create a validation split by combining the iSTS training split with the PAWS-X validation split. Similarly, we create test split by combining the iSTS test split with the PAWS-X X test split. Ap- pendix D reports data statistics.
4.2 Negative Examples
Most sentence pairs in iSTS have a major seman- tic difference. To simulate a scenario with fewer such positive examples, we add additional negative examples to the test set. We human-verified paraphrase pairs from PAWS (Zhang et al., 2019), where we label each word in the two sentences with O.
We perform our experiments on the multilin- gual XLM-R model of size 'base', In addition to the standard model, we also evaluate a model fine- tuned on SimCSE (Gao et al., 2021). SimCSE a self-super contrastive learning objective that is commonly used for adapting a masked language model to the task of embedding sentences. To train XLM-R with SimCSE, English Wikipedia and calculate sentence embed- dings by averaging the final hidden states of the encoder. We train 10 checkpoints with different random seeds and report average metrics across the checkpoints. Details are provided in Appendix B.
4.3 Synthetic Documents with Permutations
In addition, we experiment with concatenating batches of sentences into documents. The docu- ments should be considered synthetic because the individual sentences arbitrary and there is document-level coherence. Despite this limitation, synthetic documents should allow us test the ap- proaches on longer sequences and even sequences where the information is presented in slightly dif- ferent order.
IM sentences from
Specifically, we keep document A in place and randomly permute the order of the sentences in B receive B', where is the count of unordered sentences (inversion number). We control the de- gree of permutation via i, and sample a permuted document B for each B in the dataset.
Results
Table I presents validation results for the differ- ent approaches. We observe positive correlations throughout. Adding 50% paraphrases negative examples to the test set leads to a decreased curacy, indicating that imbalanced input a chal- lenge. When moving on to synthetic test documents composed of 5 sentences, the approaches tend to converge: word alignment becomes slightly less accurate, while difffel seems to benefit from the increased sequence length. Furthermore, when one of the two test documents is permuted with 5 inver- Chinese (PAWS-X: Yang et al., 2019). For iSTS, we mrchine- the sentences into these lan- guages using DeepL, a commercial service. A risk of machine translation are accuracy errors that add (or eliminate) semantic differences. Our as- sumption is that any such errors are negligible com- pared to the absolute number of semantic differ- ences the 3
4.4 Cross-lingual Examples
Finally, we evaluate on the recognition of differ- ences between document in English and doc- ument in another language. For sentences from PAWS, we use existing human translations into German, Spanish, French, Japanese, Korean, and
sions, recognizing the differences becomes slightly more difficult for most approaches.
Finally, cross-lingual document pairs clearly present challenge to all approaches, since we observe a consistent decline in terms of the aver- age correlation across the six language pairs. Ap- pendix G provides results for the individual tar- get languages, which show that comparing English documents documents in Japanese, Korean Chinese is particularly challenging. Appendices H and I juxtapose monolingual cross-lingual compar- isons, illustrating that the latter are less accurate.
In order to reduce annotation effort, we limit the evaluation to an English-centric setup where only the English sentence is annotated. When calculat-
We manually analyzed a sample of 100 English-Germmn translations. While five samples had issues with fluency, only one sample contained an accuracy error.
Discussion of diff out further adaptation, the hidden states from the
When using XLM-R with

---

Approach
iSTs
Negatives 50% paraphrases
Documents sentences
Permuted 5 inversions
Cross-lingund 6 language pairs
diff
XLM-R (last layer)
51.6
51.5
49.1
45,9
17.1
-XLM-R (8th layer)
56.9
51.0
49.5
48.1
28.7
-XLM-R + SimCSE
64.4
62.3
57.9
56.9
33.5
diffact (XLM-R + SimCSE)
29.6
9.8
29.3
25.8
4.0
diff mask (XLM-R)
51.2
46.1
49.4
49.7
24.9
Table 1: Comparison of different approaches and encoder models on RSD validation split. The table reports word-level Spearman correlation to the gold labels. The variations cumulative: the last column refers to cross-lingual test set of permuted documents containing negative examples.
8th layer yield more useful word alignment than the last layer, which confirms previous find- ings (Jalili Sabet et al., 2020; Zhang et al., 2020). Interestingly, we find that fine-tuning the model with SimCSE strongly improves these results. Even though SimCSE is an unsupervised sentence-level objective, the results suggest that learning sentence- level representations also improves the quality of word alignment. This is in line with related find- ing of Leiter (2021) that supervised fine-tuning NLI can improve the explainability of BERTScore.
et al., 2017) and the evaluation of generated text (Freitag al., 2021; Fomicheva al., 2022: Zerva al., 2022; Rei et al., 2023). Other applica- tion areas include content synchronization across documents (Mehdad et al., 2012) and the detec- tion of relevant differences in legal texts (Li et al., 2022).
Some previous work has explored unsupervised or indirectly supervised upproaches such tasks. Leiter (2021) measured the alignability of words to identify words that negatively impact the qual- ity of translations. Word alignment has also been used to estimate sentence similarity (Mathur et al., 2019; Zhang et al., 2020), and Lee et al. (2022) fine-tuned such a similarity metric with sentence- level supervision in order to promote word-level interpretability.
Relying on the deletability of a word has a lower accuracy than word align- ment. In Appendix F we test more complex for- mulations of diffael and find that accuracy can be improved by deleting bigrams and trigrams in addi- tion subword unigrams, but does not reach the accuracy of diffalign-
Discussion of diff
Deleting words to measure their effect on sentence similarity is related to occlusion-based feature attribution methods (Robnik-Sikonja and Kononenko, 2008). Yao et al. (2023) used similar method to evaluate sentence representations. Fi- nally, the effect of context cross-entropy (cross- mutual information) has previously been analyzed in the context of machine translation (Bugliarello et al., 2020; Fernandes et al., 2021).
Discussion of diffmask Using the cross-entropy of masked language modeling yields some com- petitive results on longer documents. However, latency mcasurements show that this approach is much slower than the other approaches, since the documents need be re-encoded for each masked word (Appendix C).
For the perforrig approach, diffatign with SimCSE, we report results on the test split Ap- pendix E. The test results confirm the patterns we have observed on the validation set.
Conclusion
We formulated the task of recognizing semantic differences (RSD) between two documents as a token-level regression task and analyzed several unsupervised approaches towards this task. Our experiments use annotations from iSTS, which we programmatically recombined into more challeng- ing variants of the test set. We found that the alignability word the most accurate measure,
7
Related Work
The idea of highlighting semantic differences or similarities on the level of individual words has influenced several research areas, notably inter- pretable semantic textual similarity (Lapez-Gazpio

---

especially when the word alignment is computed with SimCSE-aadateed masked language model.
References
Encko Agiree, Aitor Gonzalez-Agiree, Iñigo Lopez- Gazpio, Montse Maritxalar, German Rigau, and Lar- raitz Uria. 2016. SemEEva-2016 task 2: Interpretable semantic textual similarity. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 512-524, San Diego, Califor- nia. Association for Computational Linguistics.
Limitations
Like many NLP tasks, RSD is difficult to formal- ize. In some edge cases, it is unclear which words 'cause' semantic difference, given that natural language is not entirely compositional. For exam- ple, it is unclear which specific words should be highlighted in the pair 'flights from New York to Florida' and 'flights from Florida to New York'. Since iSTS focuses on the annotation of syntactic chunks, we follow that convention and assign the same label to all words in a chunk.
Emanuele Bugliarello, Sabrina J. Mielke, Antonios Anastasopoulos, Ryan Cotterell, and Naoaki Okazaki. 2020. It's casier translate out of English than into it: Measuring neural translation difficulty by cross- mutual information. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages (640,1649, Online, Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Another challenge is distinguishing between se- mantic and non-semantic differences. This paper re-uses annotations from the iSTS datasets and thus inherits its guidelines (except for phrases with op- posite meaning, where we stress semantic differ- ence, while iSTS stresses semantic relatedness).
Furthermore, we assume that semantics is in- variant to machine translation (MT) into another language. In practice, MT might introduce errors that add or eliminate semantic differences, and hu- man translation might be more reliable. However, we expect there to be little correlation between the gold differences and any accuracy errors that might be introduced by MT. Moreover, there are no low- resource language pairs in the experiment, where the risk of MT accuracy errors would be higher.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-41888 Minneapolis, Minnesota. Association for Computational Linguistics.
Patrick Fernandes, Kayo Yin, Graham Neubig, and An- dré F. T. Martins. 2021. Measuring and increasing context usage in context-aware machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 146776478 Online. Association for Computational Linguistics.
A methodical limitation is that our experiments are based on synthetic documents that we com- piled programmatically from human- annoan-e sen- tences, such as headlines and image captions. Our assumption is that synthetic documents can help learn about the accuracy of different recogni- tion approaches and that the findings will roughly translate to natural documents. Finally, we assume that the gold labels that human annotators origi- nally applied to individual sentence pairs remain valid when the sentence pairs are embedded in an arbitrary context.
Marina Fomicheva, Lucia Specia, and Nikolaos Ale- tras. 2022. Translation error detection as rationale extraction. In Findings of the Association for Com- putational Linguistics: ACL 2022, pages 4 4148-4159, Dublin, Ireland. Association for Computational Lin- guistics.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation. Trans- actions of the Association for Computational Linguis- tics, 9: 14444
Acknowledgements
This work was funded by the Swiss Na- tional Science Foundation (project MUTAMUR: no. 176727). We would like to thank Chantal Am- rhein and Marck Kostrzewa for helpful feedback.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference

---

on Empirical Methods in Natural Language Process- ing, pages 1894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joinl Conference on Natu- ral Language Processing (EMNLLP-JJCT) pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020. SimAlign: High qual- ity word alignments without parallel training data using static and contextualized embeddings. In Find- ings of the Association for Computational Linguisties: EMNLP 2020, pages 1627-1643, Online. Association for Computational Linguistics.
Marko Robnik-Sikonja and Igor Kononenko. 2008. Explaining classifications for individual instances. IEEE Transactions on Knowledge and Data Engi- neering, 2065):88-6600
Seonghyeon Lec, Dongha Lee, Seongbo Jang, and Hwanjo Yu. 2022. Toward interpretable semantic tex- tual similarity via optimal transport-based contrastive sentence learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 5969-5979, Dublin, Ireland. Association for Computational Lin- guistics.
Ashish Vaswami. Neam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, Curran Associates, Inc.
Yinfei Yang. Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversar- ial dataset for paraphrase identification. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNE) pages 3687-3692, Hong Kong, China. Association for Computational Linguis- tics.
Christoph Wolfgang Leiter. 2021. Reference- frree word- and sentence-level translation evaluation with token- matching metrics. In Proceedings of the 2nd Work- shop on Evaluation and Comparison of NLP Systems, pages 157-164, Punta Cana, Dominican Republie. Association for Computational Linguistics.
Xiang Li, Jiaxun Gao, Diana Inkpen, and Wolfgang
Alschner. 2022. Detecting relevant differences be- tween similar legal texts. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 256-264, Abu Dhabi, United Arab Emirates (Hybrid), Association for Computational Linguistics.
Wenlin Yao, Lifeng Jin, Hongming Zhang, Xiaoman Pan, Kaigiang Song, Dian Yu, Dong Yu, and Jianshu Chen. 2023. How do words contribute to sentence semantics? revisiting sentence embeddings with perturbation method. In Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pages 3001- 3010, Dubrovnik, Croatia, Association for Computa- tional Linguistics.
I. Lopez-Gazpio M. Maritxalar, A. Gonzalez- G. Rigau, L. Uria, and E. Agirre. 2017. Interpretable semantic textual similarity: Finding and explaining differences between sentences, Knowledge-Bact Systems, 119:186-199.
Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Oräsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 shared task on quality estimation. In Proceedings of the Seventh Conference on Machine Transiation (WMT), pages 69-99, Abu Dhabi, United Arab Emi- rates (Hybrid). Association for Computational Lin- guistics.
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual em- beddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2799- 2808, Florence, Italy. Association for Computational Linguistics.
Yashar Mehdad, Matteo Negri, and Marcello Federico. 2012. Detecting semantic equivalence and informa- tion disparity in cross-lingual documents. In Proceed- ings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 120-124, 124, Jeju Island, Korea. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversarics from word scrambling, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 1298-1308. Minncapolis. Minnesota. Association for Computa- tional Linguistics.
Ricardo Rei, Nuno M. Guerreiro. Marcos Treviso, Luisa Coheur, Alon Lavie, and André Martins. 2023. The inside story: Towards better understanding of ma- chine translation neural evaluation metrics. In Pro- ceedings of the 61 Annual Meeting of the Associa- tion for Computational Linguistics (Volume 2: Short Pupers). pages 1089-1105, Toronto, Canada, Associ- ation for Computational Linguistics.

---

A
Converting iSTS into an RSD Dataset
B SimCSE Training Details
We derive our test set from the 2016 iSTS dataset described by Agirre et al. (2016). The original URLs of the data are:
Positive examples for SimCSE (Gao et al., 2021) are created by applying two random dropout masks to a sentence S, which results in two encodings h(S:) and h'():). Negative examples are arbitrary sentence pairs in the mini-batch. The training ob- jective is:
Train: http://alt. qcri /rt/.//aa/////1111111 task2/ pata/uppooadsstrrinn.oo.... wtt.2tttfff....... tar.gz .az.gz
Test: datta/pploodd/tess..llldttaaaa tar.gz
t/tkkk
esin(h(S)).
4 = = llg
Examples from the iSTS dataset consist of two tokenized sentences and the corresponding chunk alignments. Every chunk alignment is annotated with an aligmment type and a similarity score be- tween 1 and 5, where 5 means semantic equiva- lence. If no corresponding chunk is found in the other sentence, the chunk has similarity score of NIL, which corresponds to O.
Σ}= (5)
where N is the batch size, is temperature pa- rameter, and sim(h, h') similarity measure. In our experiments, use the cosine similarity of the average hidden states as a similarity measure:
simavg (h, h') - cos(avg(h), avg(h')).
We determine word-level labels for our differ- ence recognition task follows:
For training XLM-R on SimCSE, we use maxi- mum sequence length of 128. Otherwise we use the hyperparameters recommended by Gao et al. (2021), namely a batch size of 512, a learning rate of le-5, and = 0.05, and we train the model for one epoch.
1. The similarity score for chunk is applied to the individual words in the chunk.
2. The label is calculated
score/5.
A model checkpoint is made available at the URL: https: https:///hgggngaaaccco/// wot.:aicccco/rrrrccchcc... unsup sisup-simcseexLmmrreeee robbes--------rrbbb-----b----
A small number of chunks are aligned with the OPPO type, which denotes opposite meaning. The iSTS annotation guidelines encouraged the anno- tators assign relatively high similarity scores to strong opposites. However, the context of RSD, we regard opposite meaning as a difference. To account for this, we re-label the OPPO alignments with the similarity score O. An example are the words 'lower' and 'higher' in Appendix H, which were originally aligned with a score of 4.
C Latency Measurements
When creating cross-lingual examples, late all the sentence pairs (4, B) in the dataset from English to the target languages, resulting in the translations A' and B'. We then combine them into two cross-lingual examples: A compared to B' and B compared to A'. We only consider the predictions on A and B, respectively, when calcu- lating the correlations, since we lack gold labels for the translations.
trans-
Approach
Time per 1000 Tokens
diffalign
0.44
diffael
0.45
diff fiff
97.04
Table 2: Comparison of inference time. We use XLM-R of size 'hase' with batch size on an RTX 3090 GPU. We report the number of seconds needed make predictions for 1 1000 tokens of the iSTS test set.
The converted data are available for down- load at https: https://huggingfacee / ZurichNaP/rsd-ists--22

---

D
Dataset Statisties
Dataset
Document pairs
Tokens
Labels < 0.5
Labels > 0.5
Unlabeled Tokens
Validation split
iSTS
1506
27046
64.5%
28.2%
7.3%
+ Negatives (50% paraphrases)
3012
92443
80.6%
8.2%
11.1%
+ Documents (5 sentences)
602
92389
80.7%
8.2%
11.1%
+ Permuted (5 inversions)
602
92389
80.7%
8.2%
11.1%
+ Cross-lingual (DF)
1204
183609
40.6%
4.1%
55.2%
Test split
iSTS
750
13801
70.0%
22.9%
7.0%
+ Negatives (50% paraphrases)
1500
46671
82.4%
6,8%
10.8%
+ Documents (5 sentences)
300
46671
82.4%
6.8%
10.8%
+ Permuted (5 inversions)
300
46671
82.4%
6,8%
10.8%
Cross-lingual (DF)
600
92649
41.3%
3.4%
55.3%
Table 3: Statistics for the validation and test sets. The variations are cumulative, e.g., the bottom row combines all previous variations.
E
Test Results
Approach
iSTS
Negatives
Documents
Permuted
Cross-lingual
diff align XLM-R + SimCSE
62.1 I
61.1
57.0
55.1
31.7
Table 4: Results on the RSD test split for the best-performing approach. The table reports word-level Spearman correlation to the gold labels.
F
Ablations for diff
Approach
iSTS
Negatives
+ Documents
+ Permuted
Cross-lingual
diffael XLM-R + SimCSE
29.6
9.8
29.3
25.8
4.0
-unigrams and bigrams
35.2
10.5
32.3
28.3
4.3
uniigrams, bigrams and trigrams
38.
10.2
33.5
29.2
4.4
unigrams with re-encoding
42.4
11.5
24.0
22.2
6.2
Table 5: Evaluation of more complex variants of diffaa on the validation split. Mcasuring the deletability of bigrams or trigrams of subword tokens (instead only single tokens) tends improve Spearman correlation. In contrast. encoding the partial sentences from scratch (instead of encoding the full sentence once and then excluding hidden states from the mean) does not consistently improve the metric.

---

G
Additional Results
100
100
diffalign diffget
diffatign diffgel
Spearman correlation
80
diff mask
Spearman correlation
80
diffmask
60
60
40
40
20
40%
60%
80%
2
10
12
14
16
Ratio negative examples
Number of sentences in document
(a) Recognizing differences among an increasing amount of negative examples, which do not have semantic differences.
(b) Recognizing differences between increasingly longer syn- thetic documents (ratio of negative sentence pairs: 50%). diffmast is limited to the maximum sequence length of the masked language model.
100
diffalign
100
diff diffadd diffinask
diffael
80
Spearman correlation
difl mask
Spearmnn correlation
80
60
60
40
40
20
20
0
1
2
3
4
5
8
9
10
EN
Inversion number
Target language
(c) Recognizing differences between increasingly permuted documents (document length: 5 sentences; ratio of negative sentence pairs: 50%).
(d) Recognizing differences between English source sen- tences and target sentences in various languages (ratio of negative examples: 50%)..
Figure 2: Additional results for variants validation set. The diffalign and diff approaches use an XLM-R model fine-tuned with SimCSE; the diffmask approach uses the standard XLM-R model.

---

H
Examples (English)
Gold labels
Chinese shares close higher Friday. Syria opposition threatens to quit talks. Syria opposition agrees to talkss Chinese shares close lower Wednesday.
diffalign XLM-R (last layer)
Chinese shares close higher Friday. Syria opposition threatens to quit talks. Syria opposition agrees to talks. Chinese shares close lower Wednesday.
diff XLM-R + SimCSE
Chinese shares close higher Friday. Syria opposition threatens to quit talks. Syria opposition agrees to talks. Chinese shares close lower Wednesday.
deff XLM-R + SimCSE
Chinese shares close higher Friday. Syria opposition threatens to quit talks. Syria opposition agrees talks. Chinese shares close lower Wednesday.
diffmmakk XLM-R
Chinese shares close higher Friday. Syria opposition threatens to quit talks. Syria opposition agrees to talks. Chinese shares close lower Wednesday.
Figure 3: Predictions for example document pair with two English sentences each. The example contains inversion, since the two sentences in the second document have been swapped. The gold labels are derived from the SemEval-2016 Task 2 for Interpretable Semantic Textual Similarity (Agirre al., 2016).
I
Examples (Cross-lingual)
Gold labels
Chinese shares close higher Friday. Syria opposition threatens to quit talks.
L'opposition syrienne accepte pourparlers. actions chinoises clôturent en baisse mercredi.
diffalign XLM-R (last layer)
Chinese shares close higher Friday. Syria opposition threatens to quit talks. oopposition syrienne accepte des pourparlers. Les actions chinoises clöturent en baisse mercredi.
diff XLM-R + SimCSE
Chinese shares close higher Friday. Syria opposition threatens to quit talks.
L'opposition syrienne accepte des pourparlers. Les actions chinoises clöturent en baisse mercredi.
diff XLM-R + SimCSE
Chinese shares close higher Friday. Syria opposition threatens to quit talks.
L'opposition syrienne accepte des pourparlers. Les actions chinoises clöturent en baisse mercredi
diffmash XLM-R
Chinese shares close higher Friday. Syria opposition threatens to quit talks. d'opposition syrienne accepte des pourparlers. Les actions chinoises clöturent en baisse mercredi.
Figure 4: Predictions the example Figure 1, where second document been machine-translare French. Note that the non-English documents in our test examples do not have gold labels. In experiments, only evaluate the predictions for the English document.


Reviews:
[{"review_id": "hfb8OOvlDc", "reviewer": "Reviewer_uPJb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review. Below we respond to your concern about MT: \n\n> _\"The authors use a commercial machine translation system to translate dissimilar pairs and assume that it makes negligible errors. Without verification, this may be a reason for the low cross-lingual results.\"_\n\nThere are several considerations that give us confidence that MT is acceptable: \n\n* We manually checked a sample of 100 EN–DE translations. While five samples had issues with fluency, only one sample contained an accuracy error. \n\n* There are no low-resource language pairs in the experiment. It is possible that one of the other language pairs has a higher rate of accuracy errors than the one we checked, but we do not expect it to be an order of magnitude higher. \n\n* The difference phenomena in the test set tend to be clear-cut (e.g., \"Friday\" vs. \"Wednesday\" or \"higher share prices\" vs. \"lower share prices\"). They are derived from the combination of thematically related but ultimately arbitrary documents, and not from translation errors. Therefore, we expect there to be little correlation between the gold differences and any MT accuracy errors introduced by our evaluation approach. \n\nWe will add the points outlined above to the Limitations section to make our reasoning more explicit. \n\nThanks again for the positive review, and please consider adjusting your Soundness/Excitement scores if our response addresses your concern about the use of MT.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper includes a nice approach to creating a dataset of similar and dissimilar pairs.\n\nAs an unsupervised approach, the main idea of the paper can be further integrated into a supervised system (as also suggested by the authors), offering scope for further improving the detection of semantic differences.\n\nThe paper is well written with the appendices used to supply additional information such as the technical aspects of the work and additional results.", "reasons_to_reject": "The authors use a commercial machine translation system to translate dissimilar pairs and assume that it makes negligible errors. Without verification, this may be a reason for the low cross-lingual results.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper attempts to find semantic differences in documents using unsupervised techniques. Three different approaches are tried, all based on masked language models. The first looks at token alignment, the second considers deletion and the third on normalized pointwise mutual information. The evaluation corpus is constructed using a combination of iSTS and PAWS. For the cross-lingual component, the iSTS data is automatically translated into the foreign language. The authors find that the best results are found with the alignment based approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "l325: which specific words should highlighted -> words should *be* highlighted"}, "extra_text": {}}, {"review_id": "P609eBMYNS", "reviewer": "Reviewer_4d51", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This sort paper is clear and well presented. The proposal is simple, and results potentially interesting in different perspectives, including in  approaches focusing on the evaluation of information transfer. ", "reasons_to_reject": "Not really exciting, no great originality", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an approach aiming at identifying words introducing semantic differences, by studying the differences between pairs of documents. The method is based on  word alignment and sentence-level contrasts. unsupervised approaches are analyzed. Different metrics are proposed: word alignment (highlighting words that cannot be aligned), document similarity (with and without the studied word), masked language modeling surprisal (with and without the other document as context). A test set is created, introducing different levels of complexity. Alignability, deletability and cross-entropy of words are defined and compared. The word alignability provides the most accurate measure. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tp2lbfTJ5J", "reviewer": "Reviewer_RPjL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Below we respond to your concerns: \n\n> _\"The reviewer can not judge the effectiveness of the proposed methods because there are no baselines in the experiments.\"_\n\nYes! Because we are investigating a novel task, there are no baselines from previous work that we could readily use. However, we strongly believe that novel tasks should have a place at EMNLP, and punishing papers for lack of external baselines would make this type of contribution hard to publish. We report results for three approaches, which can serve as baselines for future work. \n\n> _\"If evaluation data can be created automatically, training data can also be created on the same principle, so the reviewer thinks the supervised methods can also be tried.\"_\n\nThe evaluation data we used are not created fully automatically but depend on human annotations from iSTS and PAWS (word-level semantic similarity / sentence-level paraphrasticity). It would be very costly to annotate a sizeable training set using the same approach. \n\n> _\"The Authors should clearly state their reasons for focusing only on the unsupervised approach.\"_\n\nThanks for this recommendation. We will follow your recommendation and will discuss more explicitly in the text why our short paper focuses on unsupervised approaches – especially the cost of manually annotating supervised training data and possible future work in this direction. \n\nThanks again for your review, and please consider adjusting your Soundness/Excitement scores if this author response has addressed your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper describes the task definition of RSD, which may be novel compared with previous works.\n- Each method is clearly explained.\n- The authors create an evaluation dataset specific to RSD, including several settings (negative examples, permutations, and cross-lingual examples), and use it to evaluate the proposed methods.", "reasons_to_reject": "- The reviewer can not judge the effectiveness of the proposed methods because there are no baselines in the experiments. \n- The Authors should clearly state their reasons for focusing only on the unsupervised approach. If evaluation data can be created automatically, training data can also be created on the same principle, so the reviewer thinks the supervised methods can also be tried.\n- \"Alignability of a Word\" is a reasonable approach, but there is a concern that the technical novelty may be slightly weak.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses a novel task, recognizing semantic differences (RSD), which aims to recognize token-level semantic differences between two documents. The authors propose three unsupervised methods for RSD,  Alignability of a Word, Deletability of a Word, and Cross-entropy of a Word, and then evaluate these methods through experiments using a dataset created from the iSTS task dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agree that the paper presents an interesting method and that it is well written. The reviewers found some of the ideas novel and did not have many criticisms. 

One of the reviewers questioned the use of a commercial machine translation, but without evaluating its performance. In the rebuttal the authors carried out a small scale evaluation showing that the errors introduced by MT are minimal and likely to have limited impact on the results.

The lack of comparison with baselines was  criticised by one reviewer because without them it is difficult to know how good the methods are. The lack of baselines is justified by the authors by the novelty of the task. However, they could have tried at least some naive baselines. 

Overall, the reviews indicate that this is a strong paper which contains some novel ideas especially given that it is a short paper which cannot pack too much information.
Decision: Accept-Main