Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes a large-scale dataset for video text spotting. The dataset covers a diverse range of categories and provides multilingual text annotation for both scene text and caption text. It also establishes evaluation protocols and conducts comprehensive experimental results on tasks including text detection, recognition, tracking and end-to-end spotting.
strengths: + The dataset is larger and more diverse than existing datasets in terms of number of frames and number of text instances.

+ The scenarios that the dataset contains are diverse and open-world, which poses more challenges to research on the community text spotting.

+ The dataset annotates both caption and scene text in videos. 

+ The authors have clearly stated the evaluation metrics and open-sourced the evaluation protocols.
weaknesses: - Since the dataset is collected from Kuaishou, are the videos easy to access and free of copyright issue?

- Ethical issues: From example images in Figure 2, human faces are observed in multiple frames. Is there any potential risk to use them? Preprocessing such as blurring human faces may be considered.



====after rebuttal====

- The copyright issue is resolved according to the authors' response.

- A stronger blurring is better for privacy preserving.
correctness: Mostly correct. The data collection seems reasonable and the evaluation metrics are clearly stated.

- I have a particular question in terms of the experiments on multilingual text recognition. For the setting of "CH/AN", do you train a single network and combine both Chinese characters and alphanumeric characters and only evaluate separately or do you train two networks separately? This may help answer whether the multilingual property or the blur quality contributes more to the low accuracy.


====after rebuttal====

This question has been answered. A ultimate goal would be to use a single model to recognize bilingual or even multilingual characters. Though this setting is much more challenging and the overall accuracy would be even lower, this may make the recognition pipeline much simpler. Otherwise you would need two separate networks and an additional module to decide which language network to switch to. The authors can consider discussing these in the final version.
clarity: Yes. The paper is well written.
relation to prior work: Yes. The paper clearly discusses the relation to prior works.
additional feedback: No.
review_title: review of MMVText

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper proposes a new dataset for video text spotting. The dataset contains 510 videos with 1000K frames with 30 categories. It separates texts in the videos into caption text and scene text. Experiments are conducted and analyzed on the benchmark, showing the performance of existing baselines for text detection, text recognition and video text tracking.
strengths: 1. The dataset seems to contains the largest number of frames when compared with the existing benchmarks.
2. Video text spotting is crucial for many applications in the real world.
3. Much labor effort is needed to collect the data annotation. 
weaknesses: The reviewer has a lot of concerns about the paper.
1. The dataset has a similar size of the existing benchmarks and does not show enough evidences that the new benchmark is different from the existing dataset or has unique important features that other existing datasets can not provide.
2. Although the dataset seems to have the largest number of frames on video text spotting, the dataset seems to have smaller video number than existing datasets like [26]. How to make sure that the video texts in MMVT is more diverse than existing datasets since a video may contain many similar and duplicated frames. How many unique text instances are provided in MMVT and other existing datasets? 
3. The reviewer suggests the authors use a table to summarize the statistics of the MMVT and existing datasets, which will make the dataset comparison more clear and easier to distinguish.
4. The dataset is divided into two categories and claim that categories can favor other tasks such as video understanding, video retrieval and video text translation. However,  no experiments in Section 4 support the claim.
5. Experiments do not show much new insights about existing methods or the uniqueness of the new benchmark compared with the existing benchmark;
6. Figure 3 (a)-(b) are too small to read on the paper.
7. The current version of the paper still contains many typos or grammar  mistakes, e.g. Line 292 and Line 307. A proofreading is needed.
correctness: Yes.
clarity: The presentation is clear.
relation to prior work: Please see the "weaknesses" section.
additional feedback: None.
review_title: A dataset for video text spotting

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces MMVTest, a new benchmark for text detection/recognition (also referred to as “spotting”) in video. Key characteristics include:
- Size (1 million frames), while the largest prior work has 300k frames; over 4 million text annotations (largest prior work has over 1 million).
- Open domain (30 different video categories, over 500 videos in total), while most prior work either specializes in a single domain or has a somewhat limited diversity of data [this is somewhat difficult to verify, more on that below]
- Multilingual, primarily Chinese and English texts; only one prior work is also multilingual, LSVTD [4]
- Separate annotations for scene text and caption text, prior works typically do one or another.
- The authors also seem to introduce a new evaluation scheme for End to End Text Spotting [more on that below]
- The authors evaluate several existing methods (mostly designed for *static* text detection/recognition) on their video benchmark.

= UPDATE after the author response / discussion period =

I have updated my rating to 6 (Marginally above acceptance threshold). My rationale is below.
Overall, I buy the authors' arguments about the unique features of the proposed dataset. The paper originally had many issues with presentation/clarity, which the authors tried to address during the discussion period.
I maintain my opinion that the authors should not claim that this new benchmark is "more challenging" than prior work as they do not show that empirically; nor do they explore its complementarity to prior datasets, as I have detailed in my review. Overall makes it challenging to draw conclusions about any comparison to prior work.
PS One of the reviewers raised a criticism about not showing the usefulness of the proposed dataset to some other downstream tasks.  While this would have been a great addition, I do believe this is out of scope for the given paper, thus I disagree with this criticism.
strengths: The proposed MMVTest dataset is potentially significant/impactful for the community working on detecting/recognizing text in images/video, as it offers a larger/more diverse benchmark. It appears comprehensively labeled: texts are annotated with bounding boxes, ID information, textual content and category label (caption or scene text).

The dataset may also be of interest to those working on other video-related tasks (video understanding, video description etc) and has several other application scenarios (e.g. automatic video text translation), which however may not be strictly specific to this dataset, but to this research direction in general.

The evaluation supports several tasks, including text detection/recognition in individual images as well as Video Text Tracking (temporal localization and re-ID) and End to End Text Spotting in Videos (focus on recognition)

The authors promise to share the data and code. The given URL (https://github.com/weijiawu/MMVText-Benchmark) currently still has several TBDs. The project is open source under BSD-3 license.
weaknesses: Based on the weaknesses below, I recommend that the authors revise the paper after addressing the comments below.

= No detailed dataset statistics
The authors emphasize the diversity of the proposed dataset, however they do not provide a breakdown of how many videos/frames there are for each of the 30 categories.
Also, it would be helpful to include the text statistics broken down by language (Chinese, English, etc) as well as label type (scene/caption). 
This is *a dataset paper*, so all such details should be included (at least in the supplemental).

=Contribution/Comparison to prior datasets
While the authors discuss prior "video w/ text" datasets, it remains unclear how the proposed dataset complements them or whether it is indeed more challenging (L186). Some things the authors could have included, are e.g. discussion of dataset difficulty (prior vs. theirs) as well as to show that a model trained on the proposed large/diverse dataset transfers to prior smaller datasets. That would help demonstrate the dataset usefulness and complementarity to prior works.
I also wonder, can one combine all the existing datasets together? Will that not lead to the largest/most diverse training data? 
Besides (Sec 3.1), the authors start their search on a video platform, with *billions* of video, to collect ~500 videos in total. I found this number a little underwhelming. While the #frames is x3-4 times larger than in prior wok, it is not like an order of magnitude or more increase in size. Also it is a little unclear what the main bottleneck is in scaling this further, is it the manual annotation process?

=Chosen methods
The authors mostly focus on methods designed for static images, not video (with one exception). One argument for that is that no video-based detection/recognition methods have publicly available code. While I can see that this is an issue, it still seems not adequate to just focus on image-based methods. Moreover, the author make comparisons to how these methods performed on image-based tasks, which does not provide much insight, i.e. it would be more interesting to know whether the proposed dataset is more or less difficult that other existing *video-based* datasets (as discussed above).

=Limited takeaways
Sec 4.2 is supposed to provide some insights into the challenges of the proposed dataset. However, the authors provide little insights, e.g. what makes some of the video types (e.g. Campus, Travel, Celebrity) more difficult than others? Is it possible that these are simply less frequent? (This is where the dataset stats would help!) Or are there any specific features that make them harder? Perhaps some have more caption text and others have more scene text? An additional breakdown may be interesting! Fig 3 (c) and Table 2 could have included a more detailed language breakdown, e.g. include English specifically, especially since the authors argue that Chinese is more difficult to recognize (L311-312). Overall, more insights could be included!

=Paper presentation/clarity
Paper has multiple issues with presentation and clarity, writing has many issues, more on that below.
correctness: Little information is given about the exact instructions given to the annotators and the annotation workflow sounds somewhat confusing (Sec 3.1). Some example annotations would be great to include, as well as the annotation interface. (Could be part of the supplemental.)

The annotators only label a subset of frames, while for the remaining frames the annotations are interpolated. We do not know how good the quality of the interpolation is. What if an important transition occurs in between (new text appears, than disappears), that would not be captured, right?

Regarding the new proposed metric (Sec 3.3), L223: it is unclear how the "time matching, space location matching and recognition result matching" are defined. The code (if provided) may help clarify that.

To the Checklist question "Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?" the answer was given Yes, but this is not accurate.
clarity: The authors do not offer a very good motivation for the need of a new benchmark. One may guess that dataset size is one issue, but it is not clear what the other issues are.  Starting with an abstract's first sentence (which is hard to understand), what are these "tedious scenarios" that the authors keep referring to? 

Are there established terms for how different tasks in this work are typically referred to? E.g. we see "text reading" (does it mean OCR?), "text spotting", etc, the task names should be standardized and the tasks should be clearly defined. Table 1 mentions 3 tasks "Detection, Tracking, and Spotting", later in text the authors mention 4 tasks (L192). Later the paper, there are inconsistent names used, e.g. "End to End Video Text Reading" vs. "End to End Text Spotting".

L14 "promote multiple cultures live and communication" : unclear

L30-36: The authors argue for challenges of text detection/recognition in video as opposed to images, are there any references to cite to support this discussion?

L43 according to this, YouTube Video Text (YVT) also has two types of text annotations, captions and scene test. However, in Table 1 it is shown as only "Scene" text.

L54 "This makes it difficult to evaluate the effectiveness of more advanced deep learning models." - why? have people tried? have they failed? do you have references?

L64-66 odd to refer tho the dataset as "multidimensional", not sure what the authors mean; also what are the "multi-stage tasks"?

L147 what is "video frame recovery"?

L154 what is "double detection correction"?

Table 1: What is the difference between "Incidental" and "Open" scenario? I think both are open, so perhaps include #categories?

L 195, 203 Is "End to end video text spotting task' an existing task or a new task? Initially it seems to be an existing task, hence the authors discuss that it is "supported" by other datasets, but then they introduce *a new metric* specific to the task, suggesting it is new?

L197 "In conclusion, the high efficiency of MMVText for evaluating advanced deep learning methods is very favorable for promoting various text reading applications in real life." : the authors can not make such a claim before they showed any such results.

L202: strange to say that the dataset "mainly includes two tasks" after claiming that it supports 3 (or 4) tasks? and in the end reporting results of 4 tasks?!

L223: it is unclear how the time matching, space location matching and recognition result matching are defined

L227 supplarmentary => supplementary

Fig 3 (b) is supposed to show two methods (PSENet and CRNN) but only one plot is present.

L313 "distribution of characters is relatively smaller than the static image counterparts." - unclear

Some points are being repeated multiple times in the paper (e.g. about the properties of the proposed dataset), making a lot of content seem redundant (e.g. Sec 3.2 "echoes" the previous sections).
relation to prior work: The authors mention a survey for video w/ text methods [41] L82, however they only discuss two methods, [36] and [4]. I assume there are more than two methods in the survey?

Some additional references may be the new Vision&Language tasks that include scene text:
[a] Singh, Amanpreet, et al. "Towards vqa models that can read." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
[b] Sidorov, Oleksii, et al. "Textcaps: a dataset for image captioning with reading comprehension." European Conference on Computer Vision. Springer, Cham, 2020.


additional feedback: Sec 4: I would not start experimental results with Sec 4.2 (Attribute Experiments Analysis) but rather with the core 4 tasks, and later discuss this.

I recommend to illustrate MT/ML in %, no absolute numbers.

Sec 3.5 seems out of place where it is. Could go to conclusion/discussion?

In the Checklist, to the question "If you are including theoretical results..." you should probably say N/A, as this is not a theoretical paper.
review_title: A new large (1 million frames) and diverse (30 categories) multilingual (mostly Chinese/English) benchmark for text detection/recognition in video.

Meta-review: The paper introduces a new large-scale dataset for text spotting called MMVText which consists of 510 videos where all text instances have been annotated with bounding boxes, track IDs, transcription, and classification labels (scene text / caption). In total the dataset contains more than 1M labeled images and 4.5M labeled text instances.

Initially reviewers were somewhat negative on the submission, and raised a number of issues about the dataset, experiments, and clarity of presentation. The authors provided comprehensive responses that addressed many of the issues initially raised by reviewers; in the end reviewers were split, with Reviewer ZKz4 advocating the most strongly for acceptance, Reviewer qYEt marginally in favor of acceptance, and Reviewer NPLq advocating rejection.

The dataset itself is impressive. The scale, variety, and multilingual nature of the dataset are a clear improvement over similar existing datasets for this task. This dataset is likely to be useful and impactful within the subfield of text spotting.

However, the AC agrees with some of the weaknesses raised by Reviewers qYEt and NPLq. The most salient issue is that the paper does not empirically demonstrate the utility of the proposed dataset compared to existing datasets. The largest prior dataset (RoadText-1K) has more videos (1000 vs 510) but fewer frames (300k vs 1M), and covers a different data distribution (monolingual driving videos vs multilingual open-domain videos). The most similar prior dataset (LSVTD) is multilingual and contains videos of different scenarios, but annotates only scene text and is much smaller (100 videos vs 510, 67k frames vs 1M). Given the qualitative differences between MMVText and prior datasets, the authors should have provided a more convincing empirical comparison to demonstrate the quantitative benefits to MMVText and disentangle the utility of the various qualitative differences. For example, how well do models trained on RoadText-1k, or LSVTD, or the union of the two, perform when evaluated on MMVText? If they perform worse than models trained on MMVText, is the performance gap due to dataset size, or shift in scene distribution, or other factors? Controlled experiments could help provide insight and better quantify the utility of this new dataset.

In addition, Reviewers ZKz4 and qYEt raise some ethical questions about the dataset that have not been fully addressed. In particular, the authors “have blurred identifiable information or offensive content”, but do not provide any details about how this was achieved. The provided demo video (​​https://www.youtube.com/watch?v=WrQER3-AHsA) contains many instances of unblurred faces, which does not match the author’s response to this question. In addition, the initial submission did not discuss any potential negative societal impacts of the work; the author response amended the paper to include a one-sentence discussion mentioning “potential negative societal impacts for personal privacy”. This should be elaborated, as the proposed discussion does not suggest the authors have given much thought to the question of potential societal impact of their work.

Due to these issues, the AC feels that the paper is not ready for publication in its current form. However we emphasize that the dataset itself is novel and interesting, and is the type of contribution we want to encourage in this track. Therefore we strongly encourage the authors to take all feedback from reviewers into consideration, revise the paper, and resubmit to the second round of the Datasets and Benchmarks Track in August.
Decision: Reject