Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a corpus of OCR documents derived from historical newspapers in the United States. The data itself is extracted from the Chronicling America portal and have been processed to make them suitable for the task. The tasks involve viewing a snippet of a given newspaper and variously determining the date of publication, the location (lat/lon), and identifying a missing word. The dataset is designed to serve as a resource for improving ML methods for analyzing historical texts.
strengths: The dataset is clearly described and the development of an accompanying platform makes it accessible to other interested researchers. This dataset would likely be useful for researchers interested in classifying unknown historical documents and working on methods to spatiotemporally locate documents based on textual and other visual information. The dataset may also be useful for training language models on historical corpora, although other such resources like Google Ngrams already exist.
weaknesses: 

There is little discussion of the motivation for these three tasks in particular. Under what circumstances might one need to identify the date or location of a particular newspaper article, given that such materials are generally accompanied by relevant metadata.

The authors only show results for baselines on one of the tasks. It would be useful to see the results for the other two tasks. I would also appreciate more discussion of the kinds of performance we might expect from such systems. In other words, can we actually predict the date or location of a newspaper article with any reasonable degree of accuracy? To what extent do such predictions depend upon the presence of features like dates, place names, or specific events?
correctness: The approach appears to be sound. The authors clearly describe the procedure for selecting documents and take necessary measures to prevent downstream problems (e.g. splitting test/train/dev by publication). It is difficult to evaluate the correctness of the baseline methods as they are only very briefly discussed and the results are not shown.
clarity: The paper is generally well-written. I would recommend making the front-end (literature review) more concise and spending some more time discussing the benchmarks and results.
relation to prior work: Yes
additional feedback: N/A
review_title: A dataset and infrastructure for identifying dates, locations, and missing words in historical newspapers

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper provides a dataset named ChallAm, which is collected from historical newspapers collected on the Chronicling America portal. The resulted dataset contains 1665093 OCR image - text pairs Correspondingly, three tasks are introduced, including news date and location detection and language modeling. The challenge is published on a publicly available platform with a leaderboard.
strengths: - This paper introduces a new large-scale dataset, which may be beneficial for humanities research.
- The large-scaled dataset may be beneficial for training language models.
weaknesses: - I don't really see the utility of the introduced three tasks. Why people should work on the dataset is unclear.
- There is a lack of ethical discussions for the dataset, while there's a high chance that social biases are contained. As a result, models trained on this data may also be discriminatory.
- The paper is poorly presented.

correctness: The dataset is constructed in a reasonable way.
clarity: The presentation of the paper is weak. The formats of the figures (e.g. Figure 1) and tables (e.g. Table 2) are weird. For wrap figures, you can use vspace{-xmm} to adjust the top and bottom spacing. Also, I don't understand why the Baseline results of RetroTemp and RetroGeo are not presented in the paper as there are still a lot of spaces left.
relation to prior work: The paper has clearly discussed its relation with previous works.
additional feedback: N/A
review_title: Doubtful Utility with Potential Ethical Problems

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this paper, the authors present a new OCR excerpts dataset for historical newspapers from Chronicling America. The dataset is made of 1.6M documents crawled and pre-processed from Chronicling America, and it includes three subtasks: RetroTemp (the task of predicting the publishing date of the article given the title, and excerpts), RetroGeo (the task of predicting latitude and longitude given title, excerpt, and publishing date), and RetroGap (the task of predicting a masked word -- both in the text and the image -- in the context article). The authors report some preliminary results on the RetroGap and several statistics on the collected dataset. 



strengths: - the dataset motivation (Line 39-85) and the comparison (Line 96-116) with existing work is really well done. These two paragraphs explain in depth the importance of this dataset. The pre-processing step in Section 6.2 is quite clear as well. 
- Moreover, since news article spans from 1798 to 1963, can be a very useful resource to train LM as well, and especially to compare the representation of text through time. 

weaknesses: - the paper structure is a bit verbose and misses some important aspect as Train/Val/Test samples etc. Although the paper is well-written, it is very hard to grasp the main idea and the dataset tasks until page 5 or 6, which makes the task hard to understand. I would suggest moving the related work to a specific section and focus first on the proposed dataset and task. Moreover, many important statistics are not mentioned in the paper, such exact number of samples for each task, and misses examples for each of the task. 
- missing baselines for the RetroGeo and RetroTemp in the paper. The results are reported in the leaderboard, but there I don't know if these number are from the authors or from some other participants. Without adding this baseline is hard to evaluate how hard these tasks are.  Moreover, it is not clear what are the performance of an E2E system where the input is the img directly.
correctness: Yes, the dataset seems contructed in a sound way. However the experimental details and design are very poor, missing many baselines of the proposed tasks. 
clarity: Overall, the paper is clearly written. However, the structure of the paper needs work especially, it does not get to the point until pages 5 o 6. 
relation to prior work: The author did an excellent job here. 
additional feedback: - Adding a table that compares existing dataset statistics would be very useful for the readers
review_title: Interesting OCR++ dataset 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a new machine learning (ML) task in Gonito platform where the goal is to analyze digitized newspapers. The data consists of Chronicling America news written between the years 1798 and 1963 in image format. Excerpts between the years 1833 and 1925 make about 97 % of all data.

Comparison with previous datasets is provided. Similar machine learning tasks are also briefly discussed which gives a proper related work section to the paper.

The proposed ML tasks are simple and not very innovative:
- RetroTemp, given a normalized newspaper title and a text excerpt, the task is to predict the publishing date of the newspaper edition.
- RetroGeo, given the normalized newspaper title, text excerpt, and publishing date, the task is to retrieve the place where the newspaper is published (latitude and longitude).
- RetroGap, given the normalized newspaper title, the text excerpt, and the publishing date. The middle word of an excerpt is removed in the input document (in both text and image), and the task is to predict the removed word.

Undoubtedly the dataset with historical news is relevant, however, the paper could better motivate the need for such tasks. This is even more evident for the RetroGap task which in my opinion has little applicability. 

Regarding data divisions for train/dev/test split, the work separates newspapers from datasets, so for instance, if one newspaper edition is assigned to the dev set, all editions of that newspaper are assigned to the dev set. This procedure increases the dataset quality and decreases the risk of training newspaper-specific overfitted models.

Finally, the paper provides baselines for the tasks where BiLSTM was trained and Transformer models were used.

This paper gives more importance to the tasks and less to the dataset. For instance, little attention is given to the available variables in the dataset, (e.g., it would be interesting to have more details on dataset descriptives). 

strengths: - Interesting and valuable historical news dataset.
- Sound and correct scientific procedure.
- Both code and data are available and there seem to be no ethical questions.

weaknesses: - Too much emphasis in the ML tasks and no clear motivation for those.
- Missing to show more information on the dataset.


correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: This paper gives more importance to the tasks and less to the dataset. For instance, little attention is given to the available variables in the dataset, (e.g., it would be interesting to have more details on dataset descriptives).
review_title: Less innovative machine learning challenges for digitized news

Meta-review: This paper introduces three tasks based on a time-stamped, location-stamped large-scale corpus (roughly 398K excerpts for training, 18K for dev/eval), based on OCRed newspapers from the Chronicling America portal. Along with it, it proposes three tasks -- predicting timestamp of text snippet (RetroTemp), predicting the location of the text snippet (RetroGeo), and predicting masked token from the text snippet (RetroGap).

All reviewers (and AC) point that the motivations behind such tasks are rather weak. Further, the paper does not provide detailed documentation of the datasets, and what kind of machine learning challenges are posed by each task.  For example, how good is the OCR performance? what is the bottleneck for current models? are the performances better on newer documents vs. older documents? Specifically, the last task (RetroGap) is very similar to the standard language modeling task, and the authors introduce a less used metric for the last task, LikelihoodHashed, instead of perplexity, without a convincing justification. 

Also, while acknowledging potential ethical issues behind the dataset (section 9), the authors do not investigate this quantitatively in the proposed dataset. Given it is a historical text, the dataset might be present inappropriate biases from the past. Thus, a deeper study and investigation of the dataset should have followed. For example, a simple text-matching filtering approach with a list of blocked words would have been useful.
Decision: Reject