Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes a new task of Video Text Visual Question Answering (ViteVQA), which extends the previous text-based visual question answering task into the video domain. The proposed task aims at answering questions by reasoning texts and visual information spatiotemporally in a given video.  The authors build the first Multi-category Multi-frame Multi-resolution Multi-modal benchmark for ViteVQA. They also develop the first baseline method, T4-ViteVQA, for this task. It  first extracts optical character recognition (OCR) tokens, question features, and video representations, and then fuses multimodal information and generates the final predictions.
strengths: 1. The video text visual question answering task has never been explored before. This paper is the first one to explore this new task, collect datasets, and design methods for it. It might shed light on future research in this direction.
2. The proposed ViteVQA task is challenging because it requires spatial reasoning, temporal reasoning, reasoning on the text and visual contexts at the same time. None of the previous approaches is able to handle this task well.
3. The authors collect a dataset for this task and conduct in-depth analysis on the dataset statistics.
weaknesses: 1. Although this dataset is for video text VQA, 87% of the questions can be answered with the information from a single frame (``easy'' questions), according to lines 179-180.
2. The authors designed a "regular QA task'' and a ``domain adaptation task'', but the proposed baseline method are only designed for regular QA task and it is not mentioned how to deal with the domain adaptation task.
3. The proposed approach framework for this task is a transformer-based model with common designs. The technical contribution of the proposed framework is not strong enough. Most of the designs and the transformers are from previous work. For example, $T_t$ and $T_r$ are implemented by TransVTSpotter [53] and ABINet [61], respectively. $T_{vl}$ is implemented by All-in-one [40]. The fusion transformer consists of the transformer layers [66] and the generation module borrows [28,44].

Post-rebuttal:

The authors addressed part of my concerns, but I still believe that the baseline should be stronger. Especially there should be a baseline for the domain adaptation task. Although there are no existing techniques to solve this problem, the authors should at least provide a simple baseline so that following works can compare with the benchmark.
correctness: The claims are correct.
clarity: The paper is well written.
relation to prior work: It is clearly discussed how this work differs from previous contributions.
additional feedback: N/A
review_title: A benchmark and baseline method for video text VQA.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: 1. The paper proposed one novel task, namely he video text visual question answering.
2. One dataset is created with annotated questions as well as answers.
3. One benchmark method is also proposed.

My main concerns is the create dataset, especially the collected questions as well as the annotated answers.
1. According to the statistic and analysis results in Page 5, the lengths of the majority questions and answers are under 14 and 10. And the average lengths of questions and answers are 6.75 and 1.94. It seems that the questions as demonstrated in Figure 2 is very simple with a very simple format, namely “waht is XXXX”, and the answers are also very simple ,such as “yes”, “no”, and so on. In such a situation, the vqa task becomes very simple, which only exploit the characteristics of the short questions and answers.
2. The expression of the questions and answers is not diversity, which make the studies conducted on the dataset may not be able to reveal the true characteristics of the dataset.

strengths: 1. The paper proposed one novel task, namely he video text visual question answering.
2. One dataset is created with annotated questions as well as answers.
3. One benchmark method is also proposed.
weaknesses: My main concerns is the create dataset, especially the collected questions as well as the annotated answers.
1. According to the statistic and analysis results in Page 5, the lengths of the majority questions and answers are under 14 and 10. And the average lengths of questions and answers are 6.75 and 1.94. It seems that the questions as demonstrated in Figure 2 is very simple with a very simple format, namely “waht is XXXX”, and the answers are also very simple ,such as “yes”, “no”, and so on. In such a situation, the vqa task becomes very simple, which only exploit the characteristics of the short questions and answers.
2. The expression of the questions and answers is not diversity, which make the studies conducted on the dataset may not be able to reveal the true characteristics of the dataset.
correctness: It seems to me that the dataset is not constructed in one perfect way ,which still presents some weakness. 
clarity: yes
relation to prior work: yes
additional feedback: no
review_title: The paper present one novel task, video text visual question answer, create one dataset, and propose one baseline models. The task is interesting and inspiring. The proposed model is also interesting. What I am concerning is the creating of the database, especially the question and answers, which are very simple. As such, the studies on the database may not be able to push the boundaries of the multimodal learning.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper propose a new multi-modal dataset, namely M4-ViteVQA, to challenge the research community with a difficult Text + Video question answering task. Different from previous video-QA tasks, the model is expected to understand "visual text" in the scene, rather than text from human dialogue / narration. Moreover, the authors claim that temporal reasoning (over video frames) is also required to solve the task. A baseline model is proposed to establish Transformer-based models' performance on this task. However, there is still a large gap to human performance.

---------------- after rebuttal -----------------

The authors have addressed my concerns in the rebuttal. I have increased my final rating to 7. The contribution of the dataset itself is non-trivial; and it outweighs the weaknesses in my opinion. 
strengths: S1. A novel task combining several dimensions. Video + language + visual-text task is novel. 

S2. Data collection is non-trivial. A sophisticated collection process is done and the community will always appreciate this kind of manual effort (in contrast to auto-generated dataset).

S3. Human performance is estimated, and a large gap is presented for the research community to close.
weaknesses: W1. Dataset is small, which results in several limitations.

Each category only contains 1k-4k questions in total (Table 1). After splitting into test set, each category would only contain few hundreds of examples. This limits the robustness of per-category accuracy evaluation due to too few samples. This might also results in the need of external feature extractor (Faster R-CNN, FastText, and PHOC), which leads to a much more complex design and more overhead.

W2. Writing sometimes over-hype the dataset

Videos are naturally multi-frame, although the final model only uses 3 frames and I'm not sure if it can be seen as video. Table 1 also counts the total frame to try make a large dataset impression, but not fully utilized. Multi-resolution seems to be trivial. Online videos comes in different resolution, and the model downscales to 224 x 224 uniformly (L277). Text-Video-QA is already multi-modal. Considering W1, the lack of per-category evaluation also makes the multi-category argument weak. In my opinion, the "M4" is an overclaim that should be fixed in paper revision.

W3. Baseline might be too weak. Experiment section needs to be improved.

Given the fact that the model only takes 224x224 resolution and 3 frames, it is not surprising that human can achieve much better accuracy given the full video. More experiments should probably be done in higher resolution and frame rate. If the task can simply be solved by this, it would probably not live long in the community. It is not clear from the writing whether the BERT and All-in-one transformer are pre-trained and finetuned. Also, it is not clear why these 2 are chosen. Perhaps authors can try different text models (GPT, T5) and video encoder (Swin transformer).

Overall, I still lean positive about this paper. However, authors should try to address my above concerns during rebuttal and perhaps give revision plan for camera ready.

---------------------------- after rebuttal -----------------------------

Thanks for addressing my above concerns. The experiments are now stronger.
correctness: I did not find any major correctness issue.
clarity: The writing is mostly clear. However, some claims are over-hyped. Please see weaknesses.
relation to prior work: Related works are properly discussed.
additional feedback: A1. L sometimes refers to sequence length (L_vl, L_v), and sometimes number of layers (L_f). K, N, M also denotes sequence lengths. Perhaps rewriting the notations can help readability.

A2. Why Bi-LSTM in OCR features? Wouldn't a transformer layer work better or be more consistent?

A3. How is human performance done? No detail is provided. Are human workers also limited to 224 x 224 resolution and 3 frame? Definition of "Upper bound" is also confusing (L286-288). How to use it to "evaluate performance of the OCR systme used"? (Sec 5.2)

A4. I would suggest some more lower bound accuracy estimations such as: all "yes", all "largest OCR box".

----------------------------- after rebuttal --------------------------

Thanks for the response. I hope my suggestions helped the paper.
review_title: Official Review by JRq3

Meta-review: metareview: The paper presents a new video text VQA dataset, along a new baseline. The reviewers’s key concerns are around the size and difficulty of the dataset; the authors successfully address those during the discussion and revision period. The dataset is likely to be of interest to the community and was non-trivial to construct. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept