Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work addresses the challenge of creating an unbiased evaluation dataset for text de-duplication. It presents one such dataset with 27,210 documents and 122,876 positive duplicate pairs. This dataset was created from news wire data by leveraging the timeliness of news events. It discusses the systematic approach the authors took for creating the dataset.  It uses the dataset for evaluating several de-duplication methods: locality sensitive hashing, N-gram overlap and neural network based and shows that the neural approaches do significantly better than hashing and N-gram methods.

Creating an unbiased dataset for evaluating the efficacy of de-duplication algorithms is a challenge for large document corpora as editorial judgement is not humanly possible for the entire corpus. The paper addresses this challenge by leveraging the time time-sensitivity of news. The basic premise is that in news wire corpora duplicates can occur only within a narrow data range and therefore comprehensive human annotation of duplicates is feasible. Based on this premise, front page article from 973 newspapers on four randomly chosen days in 1930, 1955, and 1974 were reviewed by human annotators to create clusters of duplicate news articles. This resulted in ~27,000 documents ~123,000 positive duplicate pairs. Data from the period 1920-1977 was used as training data.

While n-gram overlap methods seem to be adequate on paper for detecting duplicates, noise in documents generated by OCR of scanned originals can affect higher-order n-grams and reduce the effectiveness of n-gram overlap methods for duplication detection. The work argues that neural approaches are more robust to noise and hence can potentially do better than n-gram methods. It proposes to use two different neural approaches -  1) contrastively trained bi-encoder plus clustering and 2) reranking based on a transformer bi-encoder to measure the pairwise document similarity followed by a cross-encoder to compute cross-attention between the passage and query. A S-BERT MPNET contrastively trained on over a billion sentence pairs is used as the base language model.

Experimental study shows that neural methods do substantially better than both hashing and n-gram overlap methods. Further, neural methods can de-duplicate a 10M article corpus in a matter of hours using a single GPU card. Neural method detected substantially more duplicate pairs than n-gram overlap.

Error analysis of neural methods suggests that false positives are due to articles that have some common content but don't meet the editorial criterion for duplicates. False negatives are due to truncation of the beginning of articles during the digitization process. 

Error analysis of n-gram methods suggests that false positives are due to some overlap between non-duplicate articles and false negatives are due to noise added by OCR.



strength_and_weaknesses: Strengths:

1. A new dataset for evaluation of de-duplication methods. 
2. Systematic evaluation of neural methods, hashing and n-gram overlap for de-deuplication using the new dataset.
3. Experimental finding that neural methods can give performance gains over the widely used hashing and n-gram methods for de-duplication.

Weaknesses:
1. Experimental evaluation is limited to one dataset and from one domain (news).
2. No novelty in the methodological aspects of representation learning.
3. Impact of the performance gains in deduplication by neural methods on any downstream task is not provided. 
clarity,_quality,_novelty_and_reproducibility: The submission is written well and easy to understand. All key steps have been adequately explained with sufficient details. Reproduction of the experimental results should be easy. Dataset creation is based on an interesting premise but it holds only for news corpora. Creating similar datasets for texts from genre other news would not be easy with the proposed approach. 
summary_of_the_review: The work discusses the creation of a new dataset for evaluating de-duplication algorithms. The approach taken for dataset creation is interesting but will not scale to genre other than news. Performance gain due to neural methods is very encouraging but as only one dataset from a single domain was used in the evaluation, not much can be said about the utility of the proposed methods for de-duplication in the real world.

---

I have read the reviews by my fellow reviewers and the responses of the authors to the reviews. I thank the authors for their detailed response where they have attempted to address the questions and concerns expressed by me and other reviewers. 

Given the very limited innovation in the methodological aspects of the work, I suggest the authors to seriously consider evaluating the relative impact of the proposed neural deduplication method on a few  downstream tasks. This will make the empirical contribution of the work stronger and stand out.  Test set leakage between benchmark datasets and the RealNews corpus is certainly a good step forward in this direction.

Regarding applicability of the proposed method to other corpora and especially from a different domain, the study done on RealNews and Google's online patent database is very encouraging. However, patent database is a relatively small database. Further, false positive rate on RealNews is high (16%) which needs deeper investigation as it can impact the downstream tasks.

Overall I am cautiously positive about this submission and I feel it has the potential to be strengthened significantly and made more useful to the community than it is currently.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces NEWS-COPY, a manually-annotated dataset of 27k documents in the news domain, to study de-duplication. The authors show that two neural approaches (bi-encoder and re-ranking) significantly outperform hashing and n-gram overlap on this dataset, despite the latter being more widely used in the literature. In addition, they show that the bi-encoder approach scales well, de-duplicating a collection of 10 million documents in a few hours using a single GPU.
strength_and_weaknesses: STRENGTHS
- The new dataset will be a very useful resource for the community. De-duplication is an important step in the data preprocessing pipeline, and this is the first manually annotated dataset to study it that I am aware of.
- Prior work has predominantly used old-school hashing or n-gram based approaches for deduplication, partly because of efficiency constraints. It is great to see a paper studying more advanced approaches, while being mindful about scalability.

WEAKNESSES
- I think that the empirical part of the paper is not convincing. The authors are training neural models on the same distribution that they are testing on, and I am not surprised that this outperforms hashing or n-gram approaches, which do not involve any training. However, this is an artificial setup, as it is not possible to have a labeled training set for de-dupication in the general case. The important question to me is whether the neural approaches also perform better in the wild, or they are just overfitting to this dataset. I understand that this is not trivial to answer, but I believe that the authors could (and should) have done a lot more in this direction. For instance, they could have used the different approaches to deduplicate a crawling corpus, manually evaluate their precision (the percentage of duplicated documents that are truly duplicated), and consider that together with the percentage of predicted duplicate-rate.
- Related to the previous point, all the evaluation in the paper is intrinsic: the authors simply compare the model’s output with the reference annotation according to some automatic metric. If the proposed approach is so much better than hashing and n-gram overlap, it would be good to show its potential in a more practical setup: e.g., show that a language model trained on your deduplicated data is better, revisit data contamination using your deduplication system etc.
- The paper attempts to do some analysis on the behavior and failure cases of different approaches, which is great (e.g. “when neural false positives occur, it is typically in the context of articles that have some repeated content”). However, these seem to be some general observations from the authors without any clear methodology behind and, in addition, no examples are reported, making it difficult for the reader to draw their own conclusions.
clarity,_quality,_novelty_and_reproducibility: The paper is generally well written and I believe has enough substance. Regarding reproducibility, I did not find anything obvious that was missing, but it would be valuable if the authors released their code in addition to the dataset.
summary_of_the_review: This paper introduces a very valuable dataset to study de-duplication. This dataset is used to compare neural and hashing/n-gram based approaches to de-duplication, which is an interesting exercise, but more experiments are needed to support that neural approaches are superior in the general case as concluded in the paper. All things considered, I find this to be a borderline paper, perhaps leaning a bit more towards the negative side.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a new data set for deduplication historical news articles, and evaluates several baseline algorithms on it, finding that neural methods perform very well compared to n-gram-based baselines.
strength_and_weaknesses: Strengths

The paper is well-written and introduces a resource and a baseline evaluation that will be of interest to a subset of the community.  I felt the experiments were generally well-done, although I had some questions and suggestions as detailed below.

Weaknesses

This paper is really more about OCRed text than the title and abstract let on---the failure of the standard n-gram-based methods which work well on born-digital content fail here.  Clarifying this in the title and abstract might help ensure the paper reaches the right audience.

Similarly, I was not sure that methods for correcting or compensating for the OCRed text were adequately explored here.  The paper does not include concrete examples of what the noisy n-grams look like (this should be added to support the statistics about n-gram noise early in the paper), but it would seem plausible that simple normalization techniques such as removing whitespace and using character n-grams, or mapping each word to a fixed dictionary, might mitigate some of the noise.  Techniques like this are left unexplored in the paper.

A few additional questions.  How does the accuracy of the re-ranker change as the bi-encoder threshold for re-ranking varies?  Further, the statement about the complexity of the cross-encoder (that it would require 10^14 cross-encodings) seems way too pessimistic given that you could use some kind of temporal constraint to decide which comparisons are worth trying (as I understood it---you have accurate metadata like date and news service).  As a separate point, I was also unclear on whether some kind of temporal constraint (not allowing clustering across large time spans) would have helped in the experiments reported in Table 2.  My guess is not, given the types of errors you describe that that evaluation only considers four distinct days, but it might be helpful to discuss this.



clarity,_quality,_novelty_and_reproducibility: Generally the paper is clear (see a few textual notes below) and I feel like the dataset and experiments are novel within their niche, although its niche is fairly small (dedupliicating historical texts).  The paper does not propose novel methods.

Sec 3.2 “days of content 1930” -> “days of content in 1930”
 
The hand-removed articles that had incorrect bounding boxes—what fraction was this?
 
The sentence “duplicates are defined…” seems to be missing something about the articles being about the same story, not just from the same service.
 
“Congruence labeling was used”--I don’t know what this means, did you measure interannotator agreement?  I don't see this in the paper.

summary_of_the_review: This paper is fairly narrowly-focused and does not present novel methods, and the dataset while helpful is fairly small and limited in scope (and at least one critical detail for a dataset paper, annotator agreement, is not reported).  However, I feel like the results are valid (even if some of the n-gram methods are underexplored) and would be of interest to some portion of the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
strength_and_weaknesses: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
clarity,_quality,_novelty_and_reproducibility: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
summary_of_the_review: Hi, I'm currently on a medical leave and won't be able to perform ICRL review duties. sorry for the late notice.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a new data set for news de-duplication and compares several existing approaches, including traditional n-gram-based approaches and neural approaches. The dataset consists of 27K articles and 122K duplicate pairs. 

Strengths: 
+ In general, the paper is clearly written
+ The research topic is timely and important. The reviews appreciate the creation of the new dataset.
+ There are no major concerns about this paper after the rebuttal

Weaknesses: 
- Although the reviewers appreciate the contribution of creating a new dataset, the evaluation provided in the paper is only intrinsic rather than in a practical setup. 
- The scope of the paper is narrow. Although it is legitimate for a paper to focus on a specific research problem in a specific domain, it is unclear if the dataset can lead to a general understanding of the ML algorithms and is suitable to publish in a general ML conference like ICLR. 


justification_for_why_not_higher_score: The evaluation in the paper is only based on intrinsic evaluation 
justification_for_why_not_lower_score: There is no major concern after the rebuttal.
Decision: Accept: poster