RETSim: Resilient and Efficient Text Similarity
Authors: Marina Zhang
Owen Skipper Vallis
Aysegul Bumin
Tanay Vakharia
Elie Bursztein
Venue: ICLR 2024 poster (2024)
Keywords: text similarity
text embedding
metric learning
near-duplicate detection
dataset deduplication
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: RETSim is a lightweight, multilingual text embedding designed for robust near-duplicate text retrieval, clustering and dataset deduplication.

Abstract: This paper introduces RETSim (Resilient and Efficient Text Similarity), a lightweight, multilingual deep learning model trained to produce robust metric embeddings for near-duplicate text retrieval, clustering, and dataset deduplication tasks. We demonstrate that RETSim is significantly more robust and accurate than MinHash and neural text embeddings, achieving new state-of-the-art performance on dataset deduplication, adversarial text retrieval benchmarks, and spam clustering tasks. Additionally, we introduce the W4NT3D benchmark (Wiki-40B 4dversarial Near-T3xt Dataset), enabling the evaluation of models on typo-laden near-duplicate text retrieval in a multilingual setting. RETSim and the W4NT3D benchmark are released under the MIT License at https://github.com/google/unisim.

Published as a conference paper at ICLR 2024
RETSIM: RESILIENT AND EFFICIENT TEXT SIMILARITY
Marina Zhang', Owen Aysegul Bumin 2 Tanay Vakharia', Elie Bursztein' Google' University of Florida
ABSTRACT
This paper introduces RETSim (Resilient and Efficient Text Similarity), lightweight, multilingual deep learning model trained to produce robust metric embeddings for near-duplicate text retrieval, clustering, and dataset deduplication tasks. We demonstrate that RETSim is significantly more robust and accurate than MinHash and neural text embeddings, achieving new state-of-the-art perfor- mance on dataset deduplication, ad adversarial text retrieval benchmarks, and spam clustering tasks. We also introduce the W4NT3D benchmark (Wiki-40B 4dver- sarial Near- -3xt Dataset) for evaluating multilingual, near-duplicate text retrieval capabilities under adversarial settings. RETSim and the W4NT3D benchmark are released under the MIT License at https//giituub.com/ooggeee
INTRODUCTION
Robust near-duplicate text detection is an essential component of many tasks, including retriev- ing documents, detecting plagiarism (Sun et al., (003) and blocking adversarial spam cam- paigns (Ahmed et al. (022). Users have come to expect that systems can return accurate results despite their queries exhibiting a 20% to 30% typo rate (Hagen et al. 2017). Furthermore, effi- ciently deduplicating text datasets is critical to training state-of-the- large language models (Lee jet al. 2022 Kandpal et al. 2022).
For more than two decades, MinHash-based (Broder al. 1998) Docality-sensitiie hashing (LSH) has been the most prevalent algorithm used for near-duplicate detection due to its simplicity. robust- ness, and speed. For example, the vast majority of dataset deduplication efforts still rely on MinHash (Lee et al., 2022 Kocetkov et al., (022). However, like all LSH-based techniques, MinHash is not without downsides; chief among them being that it is very parameter-sensitive and requires heavy tuning. Additionally, MinHash lacks resilience to typos due to its reliance on n-grams, leading to poor performance on noisy data and vulnerability hash miss-nusting attacks (Issac et al. 2014).
On the other hand, deep learning models are the dominant way to perform vector-based based semantic text retrieval (Muennigholf' et al. 2022). but so far, no neural embedding has been able to consistently outperform MinHash for robust near-duplicate detection (Silcock et al. (022). This is mostly due to the focus on improving semantic capabilities, which leads models be too large to run extremely
quickly and the use of sub-word level tokenization, which is not resilient to typos and adversarial attacks (Morris et al. 2020 Bursztein et al. 2023).
To fill this gap, we introduce RETSim (Resilient and Efficient Text Similarity). lightweight, mul- tilingual deep learning model trained specifically to produce robust neural embeddings specialized for near-duplicate detection. By combining the state-of-the-a art RETVec text vectorizer, modern transformer block (Hua et al. (2022), a large typo-augmented training corpus, and a metric learn- ing training regime, RETSim is able to achieve new tfate-of-the-ar performance near-duplicate detection benchmarks (Section 4.2) dataset deduplication tasks (Sections and 5.1 and spam
clustering applications (Section 5.2
Furthermore, while datasets and henchmarks exist for corpus deduplication and near-duplicate text retrieval, none of these have focused systematically evaluating near-duplicate retrieval perfor- mance under the presence of typos, word manipulations, and sentence or paragraph-level modifica-
Thiis work done during the author's internship Google.

---

Published as a conference paper at ICLR 2024
tions. To address this need, we additionally introduce the W4NT3D benchmark (Wiki-40B 4dversar- ial Near-T3a Dataset) which enables the evaluation of algorithms on adversarial near-duplicate text retrieval in multilingual setting. We report the performance of RETSim, MinHash, and popular neural embeddings such as Universal Sentence Encoder (Cer et al. 2018) and LaBSE. (Feng et al. 2022) this new benchmark in Section 4.22 highlighting uneven performance across languages and types of adversarial manipulations. The RETSim model and the W4NT3D benchmark are made available at http:///ggiiibccmooooo/o//iiiiii under the MIT License.
2 RELATED WORK
Hear-Duplicate Detection Identifying noisy near-duplicate documents in large corpus is fun- damental task with a wide range of applications, such as detecting plagiarism, finding reproduced content in literature ing training datasets for language models. Previous research has shown that duplicates in training datasets lead to inefficient training (Lee et al. 2022) and privacy concerns for large language models
news articles (Gyawali al. 2020; Silcock et al. (022), and deduplicat-
(LLMs), where models memorize and regenerate duplicated training sequences at a much higher frequency (Kandpal et al. 2022).
Unlike semantic text similarity, the task of identifying textual near-duplicates has been predominated by non-neural, g-gambbaseed algorithms such as MinHash (Broder et al. [1998), which is the most widely used technique for deduplicating large training corpuses (Kocetkov et al. [022] Lee et al., 2022). MinHash is a technique for estimating the Jaceard similarity between two sets. Algorithms such as MinHash or SimHash (Charikar, 2002) can be combined with locality seniitvv-seiivvve hashing (LSH) techniques for fast, approximate nearest neighbor search and data clustering. This allows them to scale and deduplicate corpuses containing terabytes of data such C4 (Lee et al., 2022) and The Stack (Kocetkov et al., 2022). However, n-gram or shingling-based techniques typically require texts to be parsed into standardized form (e.g. by lower-casing stripping punctuation), which makes them susceptible to typos and adversarial attacks and pose a challenge when attempt- ing to differentiate between dissimilar documents and near-duplicate documents with adversarial augmentations.
Semantic Text Similarity The task of computing semantic similarity between text is closely re- lated to near-duplicate detection. Semantic text similarity refers to the assessment of the semantic relatedness of two pieces of text based on their meaning rather than their syntactic structure, as in the case of near-duplicate detection. Recently, transformer- based language models such as Universal Sentence Encoder (Yang et al., 2019)), LaBSE (Feng et al. (022) and LLM-based embeddings (Anil jet al., 2023) which embed text into high-dimensional embedding vectors have been successfully documents using cosine similarity. Modern text retrieval sys-
used to retrieve semantically-reeate tems combine these embeddings with an approximate nearest neighbor (ANN) search algorithm to efficiently retrieve documents matching user queries.
However, language models have been shown to be vulnerable to adversarial attacks and naturally- occurring typos (Alzantot al., 2018) Gao al. 2018 Morris et al. 2020). Furthermore, language models are typically very large and costly to run even with hardware acceleration, which makes them unsuited for large-scale dataset deduplication or identifying near-duplicates in the presence of typos or adversarial text manipulations.
Metric Learning Metric learning aims to learn embedding space where similar items have small distance between their embeddings and dissimilar items are further away. Many state-of- the-art embeddings use metric learning for unsupervised training or fine-tuning including Sentence- BERT (Reimers & Gurevych] 2019) and E5 (Wang et al. 2022).
RETVec is resilient, multilingual embedding and text vectorizer trained to be robust against various forms of character-level typos and adversarial attacks. We extend the RETVec training regime to full text documents for RETSim. We use Multi-Similarity Loss (Wang et al.) 2019) for pair-based metrie learning, where typo-laden and near-duplicate versions of texts are trained to be closer in the embedding space, while other texts pushed further away. Multi-Similarii Loss is based on a general weighting framework for bair- losses and achieves state-of-the. performance, outperforming alternatives such Triplet Loss (Schroff al. 2015).
2

---

Published as a conference paper at ICLR 2024
RecSim Partial Duo (num_ cotuiisss 296)
les Chartrcdd
eeteddin
Danse 12
Imput1ox::
Chunk vestor
ReeSi Near Dup (258)
Chunk vecior
vestorized (Num,
RETSim model
Figure 1: RETSim model architecture diagram. RETSim works arbitrary length text by split- ting texts into chunks of 512 characters during its vectorization phase and encodes them using the RETVec character vectorizer. The RETSim model then embeds each chunk of text into 256-dim partial embeddings and combines them to produce the global embedding.
3 RETSIM
3.1
ARCHITECTURE
The RETSim model is composed of three main components (as depicted in Figure !
The character-level vectorizer splits the input text into chunks of 512 characters, then uses the RETVec chararcter encoder (Bursztein et al.) 2023) to encode each chunk, resulting in a batch of (512, 24) dense inputs. The RETVec character vectorizer encodes each Unicode character compact 24-bit binary representation based on its integer codepoint value. This allows the vectorizer to encode all valid Unicode characters and support all languages. Furthermore, the character-level vectorizer has been shown to be more resilient against typos and adversarial attacks.
small transformer model used to compute 56-dimension embeddings for each chunk the input text. RETSimpanial-aa these embeddings directly to finding documents that have matching chunks of text. Architecturally, the model consists of two Gated Attention Unit (GAU) blocks (Hua al. 2022). followed by Generalized-Mean pooling layer (Radenovié et al. 2018). dense projection layer which projects the embedding into 256 dimensions, and an L2 normalization layer. The model has only 536k parameters, which more than two orders of magnitude smaller than other neural embeddings (Table D 12-normalization allows the embeddings to be compared using cosine similarity. We discuss the impact of key architecture design choices in Section Hyperparameter details are provided in Appendix A.1.I. and additional ablations results in Appendix A.5
An embedding averaging module is then used to combine partial text embeddings into full-text embedding which is used for global near-duplicate matching (RETSimNea-Dun)) Averaging chunked embeddings produce global embedding is standard technique used by many models (Cer ct al. 2018) to support infinite length inputs in a cost-efficient manner. We experimented with other aggregation techniques to produce more accurate global embeddings, including training deep- averaging network (Iyyer et al., 2015), but this did not improve performance and resulted in higher computation cost. RETSim Dup and RETSSmpaatall Dup are computed in single forward pass which makes it computationally efficient. We output both types of embeddings as they have different applications: RETSimNeer-Dun is better-suited for full-text matching and retrieval (Section 4.2), while RETSimpartal is used to find partial text matches where the near-duplicate content appears only part of the document (Section 4.3).
3.2 MODEL TRAINING
Dataset We use the multilingual C4 dataset (mC4) for raw text data and following (Xue et al. 2020), use language sampling exponent of 0.3 balance sampling between low and high-resource languages. We only use text containing at least 16 characters, and we randomly select between I and 8 sentences (roughly 512 characters) for each text chunk. For each example in the
3

---

Published as a conference paper at ICLR 2024
training dataset, generate 5 pairs of augmented examples. We apply three levels of augmentation to each example text chunk (in this order): sentence-level word-level, and character-level. For each level, we randomly select the augmentation to be applied from the following categories: insertion, deletion, substitution, and transposition. We randomly apply between 0 - 25% sentence-level aug- mentation and up to 30% combined character and word-level augmentation. Empirically, we found that increasing the percentage of augmentation beyond this point causes RETSim's 's performance to degrade. The full list of augmentations used can be found in Appendix A.2
Training Procedure We train RETSim using Multi-Similarity Loss (aan et al. 2019) with a 4, 8 40, A 0.5, and 0.1. We hypertuned these parameters and the results shown in Appendix A.3] We train for million steps with batch size = 1024. The similarity loss trains the model embed augmented versions of the same text closer in the embedding space, while dissimilar texts are pushed further apart. We use the LAMB optimizer (You et al. (2019) with a max learning (ppendix/A.1.22
rate of 0.001 and cosine decay. Detailed training hyperparameters are reported
4 EVALUATION
Model/Algorithm
Type
Embed//Hash Size
# Model Parameters
LaBSE
Neural
768
471M
Multilingual USE Multilingual B5-Basc
Neural
512
69M
Neural
768
278M
PaI. 2 (Gecko)
Neural
768
SimHash MinHash
Hashing
b bits hashes
N/A N/A
Hashing
RETSim
Neural
256
536k
Table 1: Embedding models and hashing algorithms benchmarked in the paper.
4.1 MODELS AND ALGORIHTMM EVALUATED
We benchmark RETSim against four multilingual semantic text embeddings well popular n- gram based algorithms primarily used in near-duplicate text detection (Table TT. Our baseline text embeddings include Multilingual Universal Sentence Encoder (Yang et al. 2019), LaBSE (Feng jet al.. 2022). Multilingual E5 (Wang et al. (022), and PaLM 2 Gecko Embeddings (Anil et al.
2023
All text embeddings are T.2-normalized and compared using cosine similarity. We use exact search to index and retrieve nearest neighbors from our vector index for the experiments in Section 4
For non-neural near-duplicate detection and clustering algorithms, we selected the most popular algorithms: MinHash (Broder et al., 1998) and SimHash (Charikar) (002). For MinHash, we use Datasketch's MinHashLSH library. Following the most common practices in the literature (Silcock et al. 2022), we use 10 hash functions for MinHash unless otherwise specified. We use word-level n n-grams where we select the best value out of
{2, 3, 4..., 10). For SimHash, we use 64- bit SimHash and conduct shingling at the character level, where the shingle size is selected from n = (2, 4, ..., 10). For the near-duplicate detection benchmarks (NEWS-COPY and CORE Near- Duplicates datasets), we tune the optimal deduplication threshold (e.g. based on cosine similarity for neural- based embeddings and Jaccard similarity for MinHash). Detailed hyperparameter settings for RETSim and baseline algorithms used in the evaluation can be found in Appendix A.3
4.2 W4NT3D: WIKI-40B 4DVERSARIAL TEXRT DATASET EVALUATION
Dataset Description The vast majority of text retrieval benchmarks are focused on evaluating semantic performance. To the best of our knowledge, there is no multilingual benchmark for sys- tematically measuring adversarial robustness for near-duplicate text retrieval. In an attempt to fill in the gap, create and publish the W4NT3D benchmark (Wiki-40B 4dversarial Near-T3xt Dataset), which contains around 400k pairs of syntactically similar texts to evaluate near-duplicate text re- trieval in the presence of various forms of text manipulations and typos.
W4NT3D is based on the Wiki-40B dataset (Guo et al. 2020). The dataset is split into query exam- ples and target examples, where query examples are synthetically-modifee near-duplicate versions
4

---

Published as a conference paper at ICLR 2024
of target example (e.g. with typos). For each of the 41 language splits in Wiki-40B, randomly select 10,000 texts. The length of the target string is uniformly selected from between 16 and 8192 characters, in order to test performance sponding to a target text, we randomly apply up to 25% word and character augmentations, and up to 25% sentence and paragraph augmentations. For each augmentation, [insert, delete, substitute, and swap] operations. We use @ with k = 1 as the main metric, following the setup commonly found in semantic text retrieval benchmarks.
short and long text. To construct the query text corre-
uniformly select from the
Model/Algoritte
Arablc
Chinese
English
German
French
Spanish
Japanese
Korean
Russian
Thai
Avg (41 Langs)
LaBSE
0.588
0.921
Maltilingual USE
0.915
0.986
0.958
0.942
0.938
0.903
0.990
0.984
0.910
0.888
0.122
Multilingual Base
0.936
0.980
0.959
0.944
0.948
0.896
0.979
0.986
0.911
0.921
0.937
Pal.M (Geeko)
0.497
0.934
0.911
0.578
0.701
0.851
0,823
SimHHash
0.558
0.276
0
0.561
0.519
0.513
0.465
0.593
0.554
0.669
0.550
MinHush
0.633
0.172
0.591
0,558
0.556
0.575
0.223
0.814
0.523
0.416
0.538
RETSimennnbDe
0.928
0.963
RH SSimSaaaDDa Dap
0.971
0.971
0.987
0.978
0.983
0.976
0.986
0.991
0.970
0.946
0.977
Table 2: Per-language retrieval performance for various embedding models and algorithms the W4NT3D benchmark. Results on selected languages are reported alongside the average Recall @+) for all 41 languages. Full results for all languages are reported in Appendix A.4
Multilingual Performance Overall, RETSim Dup achieves average Recall @) of 0.977 across all 41 languages on the W4NT3D benchmark (Table 2}. RETSimpatiaa- is second best with Recall @1 of 0.949 and Multilingual E5, the best-performing baseline, is third with aver- age Recall @1 of 0.932. We expect that RETSim outperforms RET Simpaaaiia because the W4NT3D benchmark requires algorithm to not not just find near-duplicates, but to find the most simi- lar text. DETriimanial--Du optimizes for finding the most similar chunk of text in the corpus, which is always the most similar text overall. Similarly, we hypothesize that MinHash and SimHash per- form poorly on the W4NT3D benchmark due to their lack of ability to distinguish which is the most similar text among the near-duplicates detected, and embedding haaseddigg-ased models and cosine similarity offer a more ene-grained measure of similarity.
RETSim outperforms baseline algorithms on all languages except for Chinese and Japanese. For these languages, we theorize that semantic embeddings may have the slight edge in performance because their significantly larger model sizes (more than 100x larger than RETSim, as shown in Ta- ble(I] allow them have better representation the sub-word level tokenizers used in the baseline embeddings often treat each character in Chinese or Japanese individual tokens, which could offer higher resilience typos.
languages with large character sets. Furthermore,
1.00
LaBSE
Muttiingual USE
0.75
Muttinnual
Pal.M M (Gecko)
Roca 1021
SimHHesh
MinHash
0.25
RETSim (Partal-Dup)
Dup?
0.00
Paragroph
Sentence
Word
Character
Augmentation Level
Figure 2: Recall @1 I performance the W4NT3D benchmark, broken down by augmentation type. Results are averaged across all 41 language splits in W4NT3D.
Adversarial Resilience Delving deeper into the impact of various types of text manipulation re- veals that RETSimNearDom and RETSimparia--Du perform almost equally well regardless of the type of augmentation applied (Figure 2 Semantic text embeddings perform well on paragraph, sentence and word-level manipulations, but expected, exhibit significantly weaker performance towards character-level typos. MinHash and SimHash struggle more with word-level augmentations than deep-learning based embeddings and collapse when character-level typos are introduced. We at-
5

---

Published as a conference paper at ICLR 2024
tribute RETSim's resilience to adversarial manipulations to the RETVec character encoder well as using deep metric learning to train robust embeddings.
Figure 4 reports the Recall@ performance of the algorithms as the amount of augmentation in- creases. All algorithms perform perfectly when no augmentation is applied (exact matching), but as the percentage of augmentation increases, n-gram based approaches exhibit steep drop perfor- mance. Semantic text embeddings are able to sustain a larger degree of augmentation before their retrieval capabilities start to degrade (over 20%). RETSimNear-Dup is the most robust algorithm, with a noticeable drop in performance only after around 40% augmentation. This makes RETSim the most effective approach at clustering and deduplicating text under adversarial settings.
10
LaRSE
Mustinguall
Musiinguall Basc
Resiigg
Rocaligg
Psi (Caddo))
Samiitshh
0.6
MinHaash
RETSim (Partal
2040
4008
RETSin (Near Dup)
0%
10%
20%
30%
40%
00%
Max Tanget Text Length
Augmontation Amount
Figure 3: Recall@) @1 performances the W4NT3D benchmark (English only) for varying max target lengths.
Figure 4: Recall@1 I performances on the W4NT3D benchmark (English only) as the amount of augmenta- tion applied to the query text increases.
reports the Recall @ I performance of RETSim and baseline algorithms as the length of the query and target text varies. We see that RETSimNear-Dup and RETSimpania--Dig outperforms all other methods on short texts with fewer than 128 characters. As the text length increases beyond 512 characters, RETSimNea::Dy remains close to perfect while RETSSmpaaiill Dup performance degrades since it splits the text into multiple embeddings and finds the nearest matching chunk text. MinHash and SimHash also perform poorly on short text lengths longer texts. For neural-based embeddings,
Text Length Impact on Performance Figure
3
and start to degrade performance on longer texts for all models except RETSim Dup and Multilingual USE, the only two embeddings that can handle arbitrary length inputs.
observe slight drop in
4.3 REAL- NEAR-DUPLICATE DETECTION EVALUATION
Setup We benchmark RETSim's ability to identify near-duplicate content on real-world datasets from the literature. The NEWS COPP Deduplication dataset (Silcock et al. (202)) contains 27. 27,210 historical news articles with 122,876 positive duplicate pairs. The dataset consists of noisy near- duplicates due to factors like OCR errors, plagiarism, and news aggregation. We also evaluate the algorithms on the CORE Hear-Duplicstes dataset (Gyawali et al. 2020), which consists of 100k scholarly articles (title + abstract) with 25k exact duplicates, 25k near-duplicates, and 50k non- duplicates. Near-duplicates in this dataset arise from article revisions, versioning and metadata differences, and human typos. A key difference between these two benchmarks and the W4NT3D benchmark is that these two benchmarks are focused on detecting and clustering near-duplicate text, rather than robust text retrieval based on syntactic similarity. For both benchmarks, we follow the experimental setup provided in the papers and report Adjusted Rand Index (ARI) for the NEWS- COPY dataset and report precision/recall/f scores on the CORE Near-Duplicates dataset.
Results On the NEWS-COPY dataset, RETSimpanial Dup outperforms all other approaches by significant margin (4.8% ARI compared to our best MinHash result), as reported in Table In the dataset, there are many near-duplicate pairs where one text is significantly longer than the other, so is expected that RETSimpariil-Depg which can find matching text chunks in documents, is more suited for the task and outperforms RETSim Bucketing the near near-duplicate detection rate of each algorithm by the length ratio between positive pairs (Figure 5), observe that RETSimpanialDuu outperforms MinHash regardless of the length ratio, but MinHash surpasses RETSim per- formance when text is above roughly 1.5x the length of the other text in near-duplicate pair.

---

Published as a conference paper at ICLR 2024
1.0
Mogei/Al
ARI
Nook Dupi-Dultttee Delection Rale
Multilingual USE
0.730
Multilingual E5-Base
0.742
S-BERT*
0.700
0.3
SimHash
0.695
MinHash
0.737
0.0
MinHash (Ours)
0.783
2
4
6
7
9
10
Rartiaipt 0.831
Length Ratio belweon Near- Wear--uplicate Text Pair
RETSimNearDDup 0.704
RETTii (Partst-Dup)
REmSii (Near-Dup)
Minittshh
Table 3:
Performance comparison the NEWS-COPY dataset. Adjusted Rand Index (ARI) values reported. * denotes results from Silcock et al. 2022).
Figure 5: Near-duplicate detection rate of RET- Sim vs MinHash for different length ratios of pos- itive pairs. X-axis is the length of longer divided by shorter text, rounded to the nearest integer.
Additionally, we noticed that the labels in the dataset were occasionally noisy, as a substantial por- tion of the RETSim false positives appear to be mear-duplicates upon inspection (Appendix (A.6).
On the CORE -ear-Duplicates dataset (Table 4. where documents are roughly the same size, RETSSipaaiill Dup and RETSimNear Dup performance is nearly the same. RETSimpanial outper- forms MinHash and semantic text embedding baselines on out of 6 reported metrics, including macro Fl score. We use MinHash + LSH with 256 hash functions for computational efficiency and for better performance than the default setting, as recommended by the datasketch library Dedu- plication thresholds and hyperparameter settings for the algorithms can be found in Appendix A.3 A.3
Precision Duplicates
Recall Duplicates
Precision Non-Duplicates
Recall Non-Duplicates
Macro Fl
Accuracy
Model/ Algorithm
Exact Title Matching*
0.830
0.500
0.709
0.992
0.757
0.746
LaBSE
0.937
0.923
0.930
0.943
0.933
0.919
Multilingual USE
0.917
0.907
0.918
0,927
0.917
0.909
Multllinual ES-Base
0.937
0.940
0.945
0.942
0.941
0,932
MinHash LSH
0.929
0.902
0.915
0.938
0.921
0,918
RETSimppnal--aap
0.945
0.941
0.945
0.949
0.945
0.928
RETSimges Dap
0.928
0.937
0.942
0.934
0.935
0.926
Table 4: Evaluation results on the CORE Near- Sear-Duplicates dataset. Precision/recal//macco F1 and
accuracy numbers reported. denotes results from Gyawali et al.) (2020).
5 APPLICATIONS
5.1 TRAINING DATASET DEDUPLICATION
% train examples with dup train
valid examples with dup in train
Model/Algorithm
MinHash + +LSH
0.47%
0.46%
Exact Substring*
2.76%
0.52%
RETSinNea--ue
3.17%
0.59%
RETSimpynittnne
12.77%
2.66%
Table 5: Deduplication rate Wiki-40B (English). denotes results from Lee al. 2022
Setup We evaluate RETSim's ability to deduplicate text training datasets by deduplicating the English split of Wiki-40B (Guo et al., 2020). We conservatively set the cosine similarity deduplica- tion threshold 0.1 for RETSimNenr-Du and 0.15 for RETSimpurtal-Dpp limit the amount of false
positives, based on the optimal thresholds found in the evaluation (Appendix A.3). We use USe-
arch's default vector index for approximate nearest neighbor search (Vardanian 2023 We compare
ddtaskktth: Big Data Looks Small. http:://gtthuh..cm//////uuuuaaut

---

Published as a conference paper at ICLR 2024
Embedding / Hashing examples/sec
Model'Algorithm
Accelerator
Batch Size
time (sec)
MinHash LSH
CPU AMD 7950 32 cores
234
12544
RETSim
Onnx CPU AMD 7950 32 cores
256
10839
270
RETSim
TensorFlow GPU RIX 4090
4096
720
4062
RETSim
TensorFlow GPU NVIDIA H100
16384
363
8069
Table 6: Embedding/hashing speed of RETSim MinHash LSH on the Wiki-40B dataset.
against MinHash + LSH, where we set the number of hash functions to be 256 following Kocetkov al. 2022) and use Jaccard similarity threshold of 0.8 for deduplication (Lee et al. 2022
Results Overall, as reported in Table 5 ReT finds slightly more duplicates in the Wiki- 40B training and validation splits. This is in-line with our deduplication results (Section 4.3) where RETSimNea-Dup outperforms other algorithms. On the other hand, RETSimpatia--uug finds signifi- cantly more matches than the exact substring matching algorithm used in the previous study (Lee) et al. (022), showcasing the usefulness of performing both near-duplicate and partial-duplicate matching at once. This
than-expected number of partial matches indicate that machine learn- ing practitioners should take extra care to deduplicate Wikipedia at the chunk level to avoid feeding duplicate text to their models.
In terms of embedding speed (Table 6). RETSim is significantly slower than MinHash + LSH on CPU (46x slower), competitive when using a desktop GPU such as the RTX 4090 (3x slower) and almost on-par when using high-end GPU like the NVIDIA H100 (1.5x slower). Our current code is written in Python and not fully optimized, so we expect this performance gap to significantly shrink as we optimize our implementation. Although RETSim is slower than MinHash, RETSim is significantly smaller and faster than other text embedding models, and closes the performance gap between neural and non-neural based methods for near-duplicate text detection and dataset deduplication. Both RETSim Dup and RETSimpanial-aa Dup are returned at the same time so they have the same embedding speed. Indexing and retrieval times will depend on the vector index and search algorithm used. For longer documents, RETSimparial-Dan will produce more embeddings than NitNNer so RETSSmpparial Dup offers a tradeoff between finer-grained matching versus indexing/retrieval speed, which will depend on the specific vector search algorithm and dataset used.
5.2 IN THE WILD: SPAM EMAIL CLUSTERING
In this section, we showcase RETSim's real-world performance on clustering near-duplicate text which has been heavily manipulated by adversarial attacks by performing an evaluation on spam campaigns. Spam constitutes a strong proving ground for near-duplicate clustering algorithms as spammers employ adversarial augmentation techniques in an attempt evade detection. Such aug- mentations typically include appending or prepending unrelated text, interleaving random words and different languages, intentionally introducing typos, abusing extended character sets such as emojis and homoglyphs, and more. These techniques are collectively referred to as hash-busting.
Setup The dataset consists of 5,252 spam emails from 196 spam campaigns, donated by Gmail users who flagged them when they reached their inboxes. Each example contains the email subject concatenated with the message content. The emails were misclassified by spam classifier due to their effective adversarial text manipulation techniques, which makes them a challenging test set for clustering evaluations. Some examples of hash-busting attacks and adversarial manipulations we observe include the use of homoglpyphs, uncommon Unicode character sets, invisible characters, and padding with random words from different languages. To get the ground truth campaign clusters, emails were manually reviewed and assigned to a specific spam campaign based on similarity by human reviewers. We use agglomerative clustering to cluster spam emails, and report homogeneity, completeness, V-Measure, and Adjusted Rand Index (ARI) metrics.
Results Overall, we observed that RETSim is significantly better at clustering near-duplicates with adversarial manipulations, outperforming both SimHash and USE across all metrics considered (Table Z). In particular, we observed that RETSim outperforms USE by 4.6% the V-Measure score which is our main metric. The results reported in this section are in-line with what we observe since we deployed RETSim our main near-duplicate detection algorithm in December 2022.
8

---

Published as a conference paper at ICLR 2024
Model / Algorithm
Homogeneity
Completeness
V-Measure
ARI
USE
0.856
0,955
0.903
0.6
Simn Lash + LSH
0.867
0.876
0.871
0.571
0.937
0.963
0.949
0.747
Table 7: Performance on clustering adversarial spam campaigns in practice.
ABLATION STUDIES
Setup In this section, we summarize the key ablation studies we performed when designing RET- Sim. All the models used in this section are trained using the setup detailed in Appendix A.1.2 ex- cept we only train them for 100k steps to reduce computational costs. We evaluate RETSimnaarDDpp performance for each model 1000 examples from each of the 41 language splits and use Recall@1 as reported metric.
subset of the W4NT3D benchmark, where we randomly select
Block Type
@ecall@1
Chunk Size
Recall @
Embed. Dim
Recall
RET MLP
0.975
128
0.979
64
0.969
ConvNeXt
0.978
256
0,984
128
0,980
BERT
0.973
"512
0.986
255
0,986
TS 0,980
1024
0.983
512
0,986
*GAU
0.986
2048
0.978
768
0,986
Table 8: RETSim ablation study results on architecture block type (left), text chunk size (middle), and embedding dimension (right). *Bold denotes the value selected for the final RETSim model.
Results Cabtai]s RETSim ablation study results on max text chunk size, architecture block type, and embedding size. The most important architectural decision to decide the optimal text chunk size and finding the right balance between having the smallest size possible to maximize RETSimparial Dup efficiency while ensuring RETSim Dup full-text embeddings tively on full documents. We find that chunks of 512 characters offer the best performance.
work effec-
We also tested various model architectures and transformer blocks to find the best balance between efficiency and performance. We find that the more modern GAU block (Hua et al.) (022) outper- forms the vanilla BERT transformer block (Devlin al. 2019) and the T5 block (Xue al. 2020).
We also tried modern CNN architectures such as ConvNeXt Xt (Liu et al.) 2022) and the MLP architec- ture proposed in RETVec (Bursztein al. 2023). but both were worse than GAU block performance. Last but not least, we found that increasing the embedding size past 256 dimensions does not yield any meaningful improvements for RETSimNear-uup Accordingly, we opted use 256-dimension embedding for space-efficiency and to maximize indexing and query speed. Additional ablation studies for other hyperparameters can be found in Appendix A.5
FUTURE WORK
RETSim's 's novel training regime, which combines metric learning and data augmentation, has many other potential applications that we plan to explore in future work. For example. it could be adapted or extended to train robust semantic embeddings or image similarity embeddings. Additionally, we expect that general models become bigger and more expensive to run in the future, smaller, specialized models such as RETSim will emerge as an efficient alternative for wide range of tasks.
CONCLUSION
In this paper, we introduced RETSim, a novel, multilingual text embedding which achieves state- of-the-art performance on near-duplicate text detection, dataset deduplication, and syntactic text similarity benchmarks. RETSim significantly faster than previous neural-based text embeddings and more robust than n-gram based algorithms, which makes it suitable for large-scale text retrieval and dataset deduplication, especially in adversarial settings such as spam detection. Furthermore, introduced the W4NT3D benchmark, the first multilingual dataset designed to measure the ad- versarial robustness of near-duplicate text detection algorithms. We release both RETSim and the W4NT3D benchmark under the MIT License.

---

Published as a conference paper at ICLR 2024
REFERENCES
Nacem Ahmed, Rashid Amin, Hamza Aldabbas, Deepika Koundal, Bader Alouffi, and Tariq Shah. Machine Learning Techniques for Spam Detection in Email and loT Platforms: Analysis and Research Challenges. Security and Communication Networks, 2022:1 19, February 2022. ISSN doi: 10.1 10.1155/002/18862888. URL https: wwww hww.wwi.www.............w...w..w com/
1939-0122, 1939-0114.
journa s/ 2022
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei
Chang. Generating Natural Language Adversarial Examples, September 2018. URL http:
//aaxiv.org/aabbbbb http///as///0..... 07998 arXiv:1804.0799: [cs].
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent EI Shafey, Yanping Huang, Kathy
Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Brad- bury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christo- pher A. Choquette-Choo, Aakanksha Chowdhery. Clément Crepy, Shachi Dave, Mostafa De- hghani, Sunipa Dev. Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg. Fangxiaoyu Feng, Vlad Fienber, Markus Freitag. Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Itycheriihh, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta. Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Mar- cello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter. Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang. Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yun- han Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang. Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaLM 2 Technical Report, September 2023. URL http: org/ abs 2305 10403 arXiv:2305. [cs].
Andrei Z. Broder, Moses Charikar, Alan M. Frieze, and Michael Mitzenmacher. Min-wise inde- pendent permutations. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pp. 327-336, 1998.
Eli Bursztein, Marina Zhang, Owen vallis, Xinyu Jia, and Alexey Kurakin. RetVec: Resilient and Efficient Text Vectorizer. 2023.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Con- stant, Mario Guajardo-Cespedes, Steve Yuan, and Chris Tar. Universal sentence encoder. arXiv preprint arXiv: 11175, 2018
Moses S. Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium Theory of computing, pp. 380-388, 2002.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019. URL http:///axxiv [cs].
erg/abs/1811 04805
Fangxiaoyu Feng. Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language- agnostic BERT Sentence Embedding, March 2022. URL 01852 arXiv:2007.0185. [cs].
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers, May 2018. URL http:://aaxx:t oogg/arga.. arXiv:1801 1ee::1101..050 100454454 [cs].
18C1. 04354
10

---

Published as a conference paper at ICLR 2024
Mandy Guo, Zihang Dai, Denny Vrandečié, and Rami Al-Rfou. Wiki-40b: Multilingual language model dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 2440-2452, 2020.
Bikash Gyawali, Lucas Anastasiou, and Petr Knoth. Deduplication of Scholarly Documents us- ing Locality Sensitive Hashing and Word Embeddings. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 901-910, Marseille, France, May 2020. European Lan- guage Resources Association. ISBN 979-10-95546-3444 URL
oogy
Matthias Hagen, Martin Potthast, Marcel Gohsen, Anja Rathgeber, and Benno Stein. A large- scale query spelling correction corpus. Proceedings the 40th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 1261-1266 2017.
Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In International Conference on Machine Learning, pp. 9099-9117. PML.- 2022.
B. Issac, Chiong, and S. M. Jacob. Analysis of phishing attacks and countermeasures, 2014.
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé lii. Deep Unordered Composition Rivals Syntactic Methods for Text Classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational I Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 1681-1691, Bei- jing, China, 2015, Association for Computational Linguistics. doi: 10.3115//1//15-11122 URL
http:://cc org/ /orgg/ann hwtp//ogoooooooooooe 1162
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating Training Data Mitigates Pri- vacy Risks in Language Models. Proceedings of the 39th International Conference on Machine Learning, pp. 10697-10707. PMLR, June 2022. URL https://proceedingg .ttml ISSN: 2640-3498.
mlr.press
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The Stack: 3 TB of permissively licensed source code, November 2022. URL pattp://arxiie www.
org/ abs 2221.21211111 arXiv:2211.1555 [cs].
Katherine Lee. Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating Training Data Makes Language Models Better, March 2022. BtLL:tt:://arate
org abs/////////// 07. 06499 arXiv:2107.064999 [cs].
Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976-1 1986, 2022.
John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training NLP, Oc- tober 2020. URL URI.atp:::/xx....///////
wwwpxxxii.oogg//o. 220//20050.05 arXiv:2005.05909 [cs].
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive Text Em- bedding Benchmark, October 2022. URL
Filip Radenović, Giorgos Tolias, and Ondřej Chum. Fine-tuning CNN image retrieval with human annotation. IEEE transactions on pattern analysis and machine intelligence, (10 1665):1655- 1668, 2018. Publisher: IEEE.
Nils Reimers and Iryna Gurevych. Sentence-bert Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: 108.999811100 10084, 2019.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A Unified Embedding for Face Recognition and Clustering. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 815 823, June 2015. doi: 10.1 10..119//CVPR.215..728666 URLhttp! //aaxiiv.org/ab//155500 03832 1rXiv:1503.0383. [cs].

---

Published as a conference paper at ICLR 2024
Emily Silcock, Luca D' A'Aicco- Wong, Jinglin Yang, and Melissa Dell. Noise-Robust De- De-Duplication at Scale, October 2022. URL [es].
org/ abs 04261 arXiv:2210.04261
Yifang Sun, Jianbin Qin, and Wei Wang. Near Duplicate Text Detection Using Frequency-Biased Signatures. In David Hutchison, Takeo Kanade, Josef Kittler, Jon M. Kleinberg, Friedemann Mat- tern, John Mitchell, Moni Naor, Oscar Nierstrasz, C. Pandu Rangan, Bernhard Steffen, Madhu Sudan, Demetri Terzopoulos, Doug Tygar, Moshe Y. Vardi, Gerhard Weikum, Xuemin Lin, Yan- nis Manolopoulos, Divesh Srivastava, and Guangyan Huang (eds.). Web Information Systems Engineering WISE 2013, volume 81 8180, pp. 277-291. Springer Berlin Heidelberg, Berlin, Hei- delberg, 2013. ISBN 978-3-642-41122- 978-3-642-41230-1 24. TtRLttt:://iii wwi.nksppinnge..e com 10. 1007000077777 3-2 41230 24 Series
doi: 10. 1007/777 3- 642-41224000 1.
Title: Lecture Notes in Computer Science.
Ash Vardanian. USearch by Unum Cloud, October 2023. URL https:://iith// com/
unum-cloudie cloud/usearch
Liang Wang. Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text Embeddings by Weakly-Supervieed Contrastive Pre-training, De- cember 2022. URL httttt//p//aiii org/abs 2212.03533 arXiv:2212.03533 [es].
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R. Scott. Multi-similaaity loss with general pair weighting for deep metric learning. In Proceedings of the IEEE/CVF conferecc on computer vision and pattern recognition, pp. 5022-5030) 2019.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv: 1934, 2020.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Multilingual Universal Sentence Encoder for Semantic Retrieval, July 2019, URL http://arxiv.org
abs/ 04307 1XXiv:1907.0430 [cs].
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arxiv: wweXivv1904.0000 1904.00962, 2019.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 RETSIM DETAILS
A.1.1 RETSIM MODEL HYPERPARAMETERS
The full list of RETSim model hyperparameters can be found in Table 9
Hyperparameter
Value
Max input length (per chunk)
512
Block type
GAU
# blocks
2
Hidden dim
256
Expansion rate
Activation function
Swish relu?
Attention activation function
Absolute positional encoding
ScaledSin
Relative positional encoding
RoPE
Norm type
ScaleNorm
Pooling type
GeM (p 3)
Dropout rate
0
Embedding dim
256
# Parameters
536k
Table 9: Detailed RETSim model hyperparameters.
A.1.2 RETSIM TRAINING HYPERPARAMETERS
Table 10 details the hyperparameters settings for training configuration, loss, and optimizer used to train the RETSim model.
Hyperparameter
Value
Batch size
1024
Train steps
million
LAMB
le-6
LAMB 31
0.9
LAMB 32
0.999
Max learning rate
0.001
End learning rate
0
Learning rate decay
Cosine
Weight decay
Table 10: RETSim detailed training hyperparameters.
A.2 TRAINING DATASET DETAILS
Below, we provide the full list of augmentations used to generate augmented text for the RETSim training dataset, as described in Section 3.2
SENTENCE-LEVEL AUGMENTATIONS
Deletion:
Random sentence deletion
Random sentence truncation
Insertion:
13

---

Published as a conference paper at ICLR 2024
Random prefix sentence
Random suffix sentence
Random sentence insertion
Repeat sentence
Substitution:
Lowercasee/ppercase sentence
Random sentence substitution
Transposition:
Neighboring Swap
WORD-LEVEL. AUGMENTATIONS
Deletion:
Random word deletion
Insertion:
Random word insertion
Random word insertion per language
Substitution:
3-gram frequency based word substitution
Random word substitution
Random word substitution per language
Repeat word
Transposition:
Neighboring Swap
CHARACTER-LEVEL AUGMENTATIONS
Deletion:
Random character deletion
Substitution:
Case substitution
n-gram based substitution for n = 3, 4,5
QWERTY keyboard typo substitution
Homoglyphs substitution
Random ASCII substitution
Random character from language alphabet substitution
Random punctuation substitution
Random Unicode character substitution
Insertion:
Character repetition
n-grams based insertion for 3, 4,5
Random character from language alphabet insertion
Random punctuation insertion
Random Unicode character insertion
Transposition:
Neighboring swap
A.3 DETAILED EVALUATION HYPERPARAMETERS
Figures and contain information on deduplication thresholds values and hyperparameter settings for each algorithm benchmarked the NEWS-COPY and CORE deduplication datasets.
14

---

Published as a conference paper at ICLR 2024
/ Algorithm
Threshold Type
Threshold Value
Hyperparameters
Multilingual USE
Cosine Similarity
0.88
Multilingual E5-Base
Cosine Similarity
0.96
SimHash
Hamming Distance
64 bits, 5-grams (character-level)
MinHash (Ours)
Jaceard Similarity
0.6
10 hash functions, 2-grams (word-lese -evell
RETSimNes:Depp
Cosine Similarity
0.89
RETSimpania-DDip
Cosine Similarity
0.84
Figure 6: Hyperparameter settings for NEWS-COPY dataset evaluation in Section
Model / Algorithm
Threshold Type
Threshold Value
Hyperparameters
LaBSE
Cosine Similarity
0.88
Multilingual USE
Cosine Similarity
0.87
Multilingual E5-B 35-Base
Cosine Similarity
0.96
Siml LSH
Hamming Distance
64 bits, 3-grams (character-level)
MinHash LSH
Jaccard Similarity
0.5
256 hash functions, 3-grams (word-level)
RETSimNca-Dap
Cosine Similarity
0.86
RETSimparial-DDu
Cosine Similarity
0.82
Figure 7: Hyperparameter settings for CORE Near- Dear-Duplicates dataset evaluation in Section 4.3
A.3.1 DEDUPLICATION THRESHOLD IMPACT
1.0
0.8
0.8
Precision F1 Score
0.6
Matic
Metric
0.4
0.2
Precision Recall
0.0
Score
0.00
0.05
0.10
0.20
0.25
0.05
0.10
0.25
0.30
Deduplication (NearDup)
Deduplication Threshold RETSim (Partial-Dup)
Figure Precision/Recal/F1 scores for different cosine distance deduplication thresholds for RETSimNeatDDay (left) and RETSimpaniat-Dmm (right) on the NEWS-COPY dataset.
A.4 DETAILED W4NT3D BENCHMARK RESULTS
Tables
12 show detailed performance results for RETSim and all baseline algorithms for every language split in the W4NT3D benchmark.
15

---

Fublished as a conference paper at ICLR 2024
0,949
0,984
0.986
0.701
0.593
0.971
0,991
zh-tw
0.985
0.978
0.315
0,200
0.957
0.968
0.931
0.990
0.578
0465
0.963
0.986
Table 11: Full per- language @ecall performance for various embedding models and algorithms on the W4NT3D benchmark (part 1).
Table 12: Full per- language Recall @ performance for various embedding models and algorithms on the W4NT3D benchmark (part 2).
zh-en
0.917
0.986
0.980
0.621
0.276
0.172
0.946
0.971
0.929
0.957
0.933
0.937
0.513
0.944
0.982
0.910
0.945
0.609
0.581
0,951
0.985
0,915
0.928
0.924
0,533
0,950
0.976
vi
M
0.899
0.893
0.872
0,822
0.517
0.520
0.941
0.957
0.898
0.885
0.889
0.876
0,530
0.950
0.962
uk
0.930
0.940
0.951
0.902
0.606
0.583
0.954
0.978
0.912
0 0
0.903
0.557
0.934
0.962
0.949
0.69
0.507
0.570
0,899
0.927
0.648
0.964
0.435
0.563
0,606
0,941
0.975
0,882
0,838
0.921
0.669
0.416
0.941
0.946
0.937
0.841
0.958
0.589
0.651
0.945
0,989
th
0.899
0.925
0.552
0.520
0.953
0.969
0.930
0.938
0.934
0.519
0.556
0.9-17
0.983
SY
0.930
0.905
0.896
0.553
0.525
0.963
0.79
0.926
0.913
0.914
0.632
0.568
0.956
0.981
0. 0.908
0.940
0.580
0.570
0.947
0.977
0.912
0,870
0.929
0.356
0.568
0.595
0,946
0.971
0,922
0.901
0,921
0.575
0,573
0,961
0.971
0.923
0.930
0.926
0.665
0.583
0.953
0.985
0.9
0.91
0.554
0.523
0.700
0.888
0.903
0.91
0.596
0.504
0.958
0.976
ru
0.909
0. 0.869
0.936
0.514
0.520
0.948
0.977
0.944
0.958
0,961
0.591
0.591
0,954
0.987
70
en
0.944
0.952
0.955
0,950
0.548
0,573
0.954
0.985
0.918
0,874
0.589
0.557
0.935
0.974
pt
0.928
0.931
0.928
0.585
0.563
0,953
0.979
0.931
0.942
0.932
0.561
0.558
0.949
0.788
pl
0.928
0.921
0.944
0.577
0.560
0.958
0
0
0.927
0,943
0.592
0.598
0,953
0.988
D00
0.936
0,934
0.527
0,540
0.950
0.979
0.912
0. 8,900
0.911
0,579
0.581
0,950
0.973
0.932
0.949
0.533
0.511
0.961
0,980
0.897
0.383
0.390
0.902
0.485
0.924
0968
115
0.922
0.919
0.935
0.624
0.579
0.945
0.983
0.915
0.909
0.899
0.851
0.510
0.942
0.976
0. 0.902
0.941
0.609
0. 0.568
0.957
0. 0,980
0.915
0.915
0.497
0.558
0.633
0.928
0.971
It
Multlingual ES-Base
Model Algoritho
Model Algorithm
Muthilnggul E3 Base
Multilingual USE
Pal Pal-M (Glekk))
REISimpaat Dip
Muttlingual USE
PaLM 2 (Gecko)
RET RET top
RETSimNanDie
LETSimsas
SimHash
MinHfash
LaBSB
SinHash
Min
LaBSE
16

---

Published as a conference paper at ICLR 2024
A.5 ADDITIONAL ABLATION STUDIES
This section includes ablation studies on additional hyperparameters for the RETSim model, includ- ing the loss function, pooling type, and model capacity.
Recall @
20
0.5
0.982
20
0,948
40
0.5
0.984
40
0.919
20
0.5
0.982
20
0,947
40
0.5
0.986
40
0.923
Table 13: Ablation study on Multi-Similarity Loss hyperparameters for RETSim training. Bold indicates the hyperparameter setting selected for the final model.
Blocks
Hidden Dim
Recall @
64
0.965
128
0.980
256
0.986
512
0.986
64
0.962
128
0.980
256
0.984
512
0,987
4
64
0.966
128
0.980
256
0.985
512
0.986
Table 14: Ablation study for RETSim model capacity and size (number of GAU blocks and hidden dimension for the blocks). Bold indicates the hyperparameter setting selected for the final model.
Pooling Type
Recall @
Average Pooling
0.985
Max Pooling
0.983
Generalized Mean Pooling
0.986
Table 15: Ablation study pooling type for the RETSim model. Bold indicates the hyperparameter setting selected for the final model.
A.6 SELECTED EXAMPLES FROM NEWS-COPE DATASET
In this section, we randomly selected a set of false positives and false negatives for RETSim on the NEWS-COPY deduplication dataset to provide further insight into the results.
17

---

Published as a conference paper at ICLR 2024
Text
Text 'LONDON (AP) lan Ball, 26-year- old unemployed Englishman, was brought into court today and charged with attempted mur- der during an at- tempt to kidnap Princess Anne from her car in the heart of London Wed- nesday night. Ball, lean-faced and bearded, stood stiffly in the dock at the Bow Street Magistrate's 's court, handcuffed to two detectives. He spoke only once during his 60- second appearance, saying iha London accent: "I want to apply for legal aid." The court or- dered him held for another hearing on Ma...
chauffeur, a policeman and a passing jour- nalist who tried to intervene. Beaton and the policeman were reported serious condition. The 23-year-old princess and her husband of five months, Capt. Mark Phillips, were not hurt. But police experts said the holes left by one of the bullets fired into the car indicated it passed between them, missing them by inch- es. A police informant said it was believed 1 shots were fired by the assailant. Experts were studying two revolvers found at the scene. They said fi...
By United Press tnfernational Ay SSAST OR BE FRE NG SG The federal government has proposed new methods of eoustructing federal buildings in a move to save ad- ditional en- ergy and suggested ils elfort could be adapted to all new buildings,
Hy United Press International The federal government has Proposed new methods of constructing federal buildings in move lo save addilional energy and suggested ils effort could be adapted to all new buildings, Arthur F, Sampson, General Services Administration ad- ministrater, said new features for such construction would include the collection of rain waler for cooling and irriga- tion, solar energy collectors and the covering of exterior walls with earth. "Whal we are saying that these design criteri...
Washington, Jan. 27. (P) (P))-m Im- mediate removal of John F. J. Her- bert, prohibition administrator for Montana and Idaho, was de- manded in the senate today by Sen- ators Borah, Idaho, and Wheéeler, Montana, on the ground of charges placed before them by department of justice investigators. Wheeler accompanied his demand (Continued on Page 2)
Washington, Jan. 27 1 1AP).-Immied: aie mmoval of John F. Herbert pro- hi- bition administrator for Montana and 'Idaho was demanded the Seuate to- 'day by Sen- ators Borah. idaho, and Waeeler, Montana. on the ground of charges placed before them by Depart- meat of Justice investigators. Wheeler accompanied his demand nith declaration that prohibition en- foreemen: had brukea down. He blamed the "politicians" and called upon the Law Enforcement Commussion to sum- mon members of the Republican Na- tona...
By RAYMOND CLAPPEA (Dnited Presa Stal Correspandoayy London, Jai, -UP -he Am 'erlenn delegation to the navat confer ence today won Is demand for pre- sentation: of the ense of suxiliary warships limitation first at tho noxt plenary session Thuvaday, "Tho chlet delegates, mec- tittg at St. James palace, also decided that tho plenary sesslon would discuss the Main con- ference questions in alpha betical order of ihe countriea pro- posing. Press ta be Admitted The American delegation woo a second vic- tory whe...
London, Jan. 24, W.P-The Amer- jean dele- gation fo the naval cen- ference teday won its demand for presentation of the case of auxil- jary warships linsitation first at the next ple- trary session "Vhursday, The chief delegates, meeting at Si, James Pelace, also decided that the plenary session would discuss the main confeyence questions in alphabetical order af the cauntries proposing. The American del- egation won a second victory when it was decided to udmil certain representatives of the press at fie plenary...
Table 16: Example false negatives for RETSim the NEWS-COPY dataset (pairs of texts not detected as near-duplicates by RETSim but labeled as near-duplicates in the original dataset). Ex- amples are randomly selected and truncated at 512 characters for display.
18

---

Published as a conference paper at ICLR 2024
Text I
Text
BOZEMAN, Mont. (AP) Chet Huntley, whose resonant voice and rough-hewn face be- came familiar to millions on the nightly television news, died Wednesday in his moun- tain resort home. He was 62. He underwent surgery for lung cancer in January but had remained activesuntil recent weeks. He died at 2:20 a.m, according to his widow, Tippy Hunt.ey. Huntiey was teamed for 14 years with David Brinkley on NBC's Huntley- Brinkley Re- port. He quit in 1970 and re- turned to his native Montana develop the $20-millio.
BOZEMAN, Mont. (AP) Chet Huntley, whose resonant voice and rough-hewn face became familiar to millions on the nightly television news, died Wednesday in his moun- tain resort home. He was 62. He underwent surgery for lung cancer in January but had remained active until recent weeks. He died at 2:20 a.m., according to his widow, Tippy Huntley. Huntley was teamed for 14 years with David Brinkley on NBC's Huntley-
Brinkley Report. He quit in 1970 and returned to his native Montana develop the $20 mil- lion Bi...
By THE ASSOCIATED PRESS Some Amer- icans are paying up to 50 per cent more per month for electricity this year than they did last, an Associ- ated Press survey shows. Con- sumers are beginning to organize to fight the rate hikes. A spot check of monthly elec- tric bills this year and last showed that most in- creases have been about $1 or S2, gen- erally about 10 per cent, with the highest reported boost com- ing in Jacksonville, Fia., where the average tab went from $17.90 last year to $27.70 this year. Utility...
By Louise Cook Acenciaiod Prece Writer Same Americans are paying up io 20 per cent more per month far electricity this year ihan they did last, an -Associ- Press survey shows. onsumers are beginning to ze to fight the rate hikes, A spot check of monthly elec- tre hills this year and Jast showed that most increases ve been about $1 or $2, gen- erally about 10 per cent, with the highest reported boost com- ing Jacksonville, Fla., where the average tab went from $17.90 last year to $27.70 this year...
BOZEMAN, Mont. (AP) Vice President
BOZEMAN, Mont. (AP) Vice President Gerald R. Ford says the world will miss the "unique abilities" of former television news anchorman Chet Huntley. Huntley, 62, died his home Wednesday after a long bout with lung cancer. Family spokesmen said me- morial service would be con- ducted for Hunt- ley Sunday at the Big- Sky of Montana resort and recreation area south of Bozeman. Hunt- ley was chair- man of the Big Sky board of directors. Another memorial service is sched- uled Tuesday in the New York studios of...
Gerald R. Ford says the world will miss the "'unique abilities" of former television news anchorman Chet Huntley. Huntley, 62, died his home Wednesday after a long bout with lung cancer. Family 'spokesmen said a memorial service would be conducted for Huntley Sunday at the Big Sky of Montana resort and recreation area south of Bozeman. Huntley was chairman of the Big Sky board of directors. Another memorial service is scheduled Tuesday in the New York studios of the...
WASHINGTON (AP) The House has passed legislation raising the minimum wage from $1.60 an hour to S2 this year for most workers covered and to $2.30 for all by 1978. The bill, approved Wednesday 375 to 37, also would increase by 7 million to 56.5 million the number of workers covered by the mini- mum wage laws. The bill is modified ver- sion of one President Nixon vetoed last year. However, he is expected to sign this one if it is finally approved after ad- justment with a similar Senate passed measure, altho...
WASHINGTON (AP) The House has passed legislation raising the minimum wage from S1.60 an hour to $2 this year for most workers covered and to $2.30 for all by 1978. The bill, approved Wednes- day 375 to 37, also would in- crease by 7 million to 56.5 mil- lion the number of workers cov- ered by the minimum wage laws. The bill is modified version of one President Nixon vetoed last year. However, he is ex- ted to sign this one if is inally approved after adjust- ment with a similar Senate- passed measu...
Table 17: Example false positives for RETSim on the NEWS-COPY dataset (pairs of texts detected as near-duplicates by RETSim but not labeled as near-duplicates in the original dataset). Examples are randomly selected and truncated at 512 characters for display.
19


Reviews:
[{"review_id": "byh2kqq6cK", "reviewer": "Reviewer_XRFc", "summary": "This work addresses the problem of learning robust metric embeddings for near-duplicate text retrieval, clustering, and dataset deduplication tasks. Towards this end it proposes a lightweight multilingual deep learning model called RETSim. The work argues that previous solutions for near-duplicate text retrieval are not resilient to typos in the text documents and are vulnerable to adversarial attacks. Further, some of the previous solutions are very parameter-sensitive and consequently need heavy tuning and/or are too big to be used effectively in real-world applications of near-duplicate text retrieval. The proposed solution to the problem of learning robust metric embeddings for near-duplicate text retrieval makes use of a large typo-augmented training corpus, RETVec text vectorizer, a lightweight  transformer block (Hua et al., 2022) and  a metric learning training regime. The tokenizer splits the text into multiple chunks of size at most 512 characters and encodes each chunk using a character encoder. Each encoded chunk is fed to a lighweight transformer that computes a 256 dimensional embedding for the chunk. The embeddings from all the chunks of a text input are then combined into an embedding for the full text by averaging. \n\nThe model is trained on a dataset derived from the multilingual C4 dataset by applying sentence, work and character level augmentations on the text. Multi-Similarity Loss is used in training which attempts to make augmented versions of the same text closer to each other in the embedding space. \n\nIn addition to addressing the technical problem of measuring similarity of near-duplicate texts, this work introduces a new benchmark for the evaluation of such methods. The benchmark dataset called W4NT3D is particularly oriented towards evaluation of near-duplicate text retrieval models on multilingual text corpora in the presence of typos, word manipulations, and sentence/paragraph-level modifications. \n\nThe work presents results from the experiments comparing the proposed method with several baselines including some hash-based methods and some deep-learning-based methods. Results are presented for W4NT3D dataset. The proposed method performs better than all the baselines overall but not by a huge margin (Multilingual E5-Base is very close) and for Chinese and Japanese languages, it performs slightly worse than the best baseline for these languages (Multilingual USE). \n\nThe experimental study also reports results on different types of augmentations on the same dataset. Interestingly, all methods seem to do well for paragraph and sentence level augmentations. For word-level augmentations, hashing based methods do worse than other methods and for character-level augmentations, the proposed method fares best.\n\nFurther experimental results are provided for two real-world near-duplicate datasets - NEWS-COPY Deduplication dataset and CORE Near-Duplicates dataset. The proposed method gives significantly better results on NEWS-COPY dataset compared to all the baselines. On CORE Near-Duplicates dataset, the method fares marginally better than the hashing-based baselines but no comparison is made with neural-network-based methods.\n\nThe work discusses a practical application of the proposed method - spam email clustering. Results show significant improvement in clustering spam emails compared to three baselines.", "questions": "1. In the deduplication experiments on Wiki-40B (English), was RETSim trained on this portion of Wiki-40B? If so then could that also be one of the reasons why RETSim is able to detect substantially more duplicates than the baselines?\n\n2. Why isn't Multilingual E5-Base used as a baseline in spam email clustering experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. A well-designed approach for near-duplicate detection that is light-weight, robust to typos and works on multilingual data.\n2. Encouraging improvements in near-duplicate detection over both hashing-based and neural-network-based methods though Multilingual E5-Base seems to be reasonably competitive.\n3. A new dataset is made available that can be used for near-duplicate detection studies.", "weaknesses": "1. RETSim is generally slower than MinHash and particularly slow on CPU (Table 6).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "553tzWQpA7", "reviewer": "Reviewer_pVcX", "summary": "The paper introduces RETSim (Resilient and Efficient Text Similarity) to identify near duplicate whole text or partial text efficiently, with applications in deduplication, clustering, etc. The proposed model is designed to be lightweight and learned with simple corruption-like augmentations (hence mimicking the typical dedup errors). The proposed method has been carefully tuned and engineered and demonstrated better performance than both non-learned approaches and learned methods.", "questions": "See the questions asked in the \"Weakness\" part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The problem of dedup is a fundamental one and has many traditional applications. In the era of LLMs, it gains one more important role as deduplicating the training dataset, which is one of the crucial factors to training a good LLM. \n\n* Although the proposed technical solution is simple, it is designed with care (to balance the sophistication and effectiveness) and hyper-parameters are carefully tuned. \n\n* The performance results are excellent over a comprehensive set of tasks and datasets.", "weaknesses": "* There is no analysis on why the almost straight-forward design produces better results (F1) than other learned algorithms. From the ablation study (e.g., Sec 6), it seems the benefit is more from tuning parameters?\n\n* Similarly, how important is the training corpus and the data augmentation methods? E.g., what if the data distribution, corruption pattern, etc. on mC4 is drastically different from those in the testing corpus? \n\n* MinHash + LSH achieves a strong performance on real dataset (Table 4). It seems there is no detail on the parameter of the LSH part, and therefore it is hard to know how much performance loss is due to the LSH part (e.g., if the number of \"OR\" part is small, it may hurt the performance). Also why +LSH? I can understand that MinHash + LSH is a practical solution for large corpus due to its good runtime efficiency, but run-time efficiency is not report for the proposed method. In fact, it is not clear whether and which index you use to find the near duplicate (chunks) with your method. Do you use a vector index and which one? \n\n* It will be better to perform a case study or detailed error analysis to understand the false positive and false negatives for the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "usGBUg0i2p", "reviewer": "Reviewer_rc5k", "summary": "This paper proposes RETSim, a lightweight text embedding model designed for near-duplicate text retrieval, clustering, and dataset deduplication. RETSim is a cascade system comprising three models: a character-level vectorizer, a small transformer model, and an embedding averaging module. There are two variants, RETSim (Near-Dup) and RETSim (Partial-Dup), tailored for full matching and partial matching, respectively. The authors evaluate RETSim across multiple datasets, including their self-proposed W4NT3D benchmark, the two news deduplication dataset, and a span email clustering dataset, demonstrating its superiority over neural and hashing baselines.", "questions": "- It seems that the dataset used in the experiment in Section 5.2 is a non-open dataset. The authors should provide more details on the dataset.\n- For a same amount of input, does RETSim (Near-Dup) produce more embeddings than RETSim (Partial-Dup)? If yes, why isn’t the speed separately measured in Table 6? (I think RETSim (Partial-Dup) is slower, since it needs to retrieve against a significantly larger set of text embeddings during de-duplication?)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed model is effective and efficient.\n- RETSim trained on mC4 can be applied to a variety of datasets.", "weaknesses": "- The experiment in Section 5.1 makes no sense. Apparently, according to previous experiments, RETSim should be able to find more duplicates. How using RETSim may impact downstream language model pre-training remains unclear.\n- Some baselines are missing in Table 4 and Table 7.\n- Certain aspects of the method are not clearly explained, such as the process for obtaining the (512, 24)-dimension tensor from the vectorizer (Section 3.1) and the Multi-Similarity Loss, which appears to be a less common concept that deserves more detailed explanation (Section 3.2).\n- The paper lacks a dedicated Conclusion section, with Section 7 not effectively serving as a substitute conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VVQY3zuxDX", "reviewer": "Reviewer_XEvv", "summary": "the authors build a more light-weighted embedding for document similarity detection task, and conducted extensive experiment to prove the performance of their design. On their purposed dataset, they prove that the new model/embedding has better overall performance in multilingual setting retrieval. Overall the paper is complete and easy to follow. However there is not much original contribution in terms of model design, and the improvement seems incremental and marginal. See comments below", "questions": "table 4 column 4 - should be recall non-duplicates?\npalm 2 gecko embedding performed much worse in the paper than it should be. any hypothesis or analysis on this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "extensive experimentation - from the purposed multilingual dataset W4NT3D to NEWS-COPY dataset, the experiment is thorough and convincing in demonstrating purposed effectiveness\nefficiency over other nn models - personally I feel the paper is most valuable in that the purposed model is much simpler than the rest deep models, while their embedding maintains a relatively same or even better performance, although it's on the purposed dataset. the saving from efficiency instead of performance is the essence", "weaknesses": "originality - the purposed model arch is more like a concatenation of existing methods/building blocks, and frankly speaking I can see very little original or significant contribution\nincremental contribution - the improvement is claimed on the purposed multilingual dataset, while it's not substantially surpassing the other deep model counterparties. for the real world duplication detection task, the traditional minhash based method is a better choice with decent performance IMHO, considering computation constraint", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)