Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces a new dataset of video quality metrics for many video compressions. The dataset consists of various videos encoded by a wide variety of standards. The authors also provide subjective scores and results for various evaluated metrics.
strengths: The paper is well written and provides clarity of thought. It provides new avenues by introducing data for other encoding standards as well. The subjective scores have been computed on the basis of a large pool of responses. The evaluation scheme used is thorough.
weaknesses: The results section could have been written in a more lucid and better way. The tables in the paper could have been explained a bit more. Information about the process of video collection is missing, i.e. queries used for video collection, genres of videos, etc. The authors have not briefly explained the application field and the larger impact of the proposed work. 
correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: ..
review_title: Video Quality Metrics Benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces a dataset and benchmark for video and image quality of video encoders. The dataset consists of compressed videos and subjective scores, collected via crowd-sourced comparison. The benchmark consists of 2045 videos and 10,800 viewers and the authors provide assessment results on the open parts of the dataset. 
strengths: 1. A new dataset and benchmark for video compression, produced by new compression standards.
2. Extensive answers to the dataset
weaknesses: 1. The scale is relatively small (as shown table 1)
2. It's unclear how the video is selected (based on what methodology)

correctness: The claims are correct and the dataset is constructed in a sound way (but lack details on how they are collected from vimeo)
clarity: The paper is well-written. 
relation to prior work: The related works are described in section 2 and the paper clearly discusses how it differs from previous work. 
additional feedback: Please refer to the section above.
review_title: A very useful dataset and benchmark for video and image quality. 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors construct a new benchmark of video quality metrics on a wide range of video compression methods. It considers more video encoding standards and collects subjective scores via crowd sourcing. 
strengths: The authors carefully select videos, distortions, types of new encoding standards, collect subjective scores, and evaluate a wide range of quality metrics.
weaknesses: The authors argue that we need new benchmarks because existing benchmarks are commonly investigated using older compression standards, such as AVC. It implies that the artifacts generated by newer compressors may be differently distributed from the older ones and such distribution shift will affect the performance of video quality metrics. However, this claim is not evidenced in the motivation part or echoed in the results section. The results in Table 3 and Table 4 show that exiting metrics (e.g., VMAF) perform good even with the newer Codec with above or around 0.9 SROC. This weakens the necessity of constructing a new benchmark with newer Codecs.
correctness: The dataset is constructed in a sound way. Since subjective scores are collected via a crowdsource platform, I am little concerned about whether the viewers' environments (e.g., devices, screen size, network quality, etc.) affect the scores. It would be better to add more details on how the viewers give their scores.
clarity: The paper is written well.
relation to prior work: the related works are well discussed. 
additional feedback: NA.
review_title: A valuable dataset and benchmark for video quality metrics

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This work is dedicated to video quality assessment via trainable algorithms, focusing on measuring video quality after compression. The existing benchmarks contain videos compressed using outdated standards, thus they might seem irrelevant for evaluating modern algorithms. This paper introduces a new video quality benchmark for measuring the quality of videos encoded via nine actual standards. The authors acquired videos encoded using different standards, and performed a user study on subjective video quality using crowd-sourcing platforms. The collected dataset was used to evaluate a large number of no-reference and full-reference metrics. According to the conducted empirical study, the best full-reference metric is VMAF, which is highly correlated with the subjective quality, while the no-reference MDTVSFA metric is almost on par with full-reference metrics, yet being impractical due to computational complexity. 
strengths: Good paper overall! 

1) The study is well-positioned: the drawbacks of the existing evaluation protocols are stated clearly, and are being addressed with the novel benchmark. 

2) The large amount of data was collected in this study. The data acquisition procedure is well-structured and expounded clearly. I personally appreciate the authors verified the crowd-sourced measurements via additional questions. Noisy and misleading samples seem to be quite common in the open-source datasets due to inaccurate data acquisition protocols, so I find the verification part extremely important. 

3) The experiments are comprehensive and persuading, since a lot of metrics were tested and many aspects of quality assessment are covered. The amount of the quantitative results presented in the forms of tables, histograms, and plots makes a good impression.

4) The authors provide a theoretical explanation of the empirical results. The evaluated metrics are discussed, their limitations and strengths are explicated; for some metrics, the most suitable usage scenarios are proposed (e.g., gaming/animation videos, or sports videos).
weaknesses: I am a bit confused with some aspects of the paper, and would be grateful for some additional comments from the authors.

1) In 3.1, it is stated that any metrics that have already shown low correlations in other studies are excluded from the evaluation. What are these metrics? Could you please provide references for the studies proving such metrics to be inadequate?

2) In 3.2, the authors claim the video quality for FullHD resolution is more difficult to distinguish visually during subjective tests. Is this conclusion based on a common sense or is it a result of an empirical study? It seems that the major advantage of the proposed benchmark is that it is modern and uses up-to-date codecs. However, it also seems that FullHD will remain a popular format within the next few years, so it is important to study them thoroughly.

3) It is unclear to me, do the open and the hidden parts of the dataset differ statistically? I would like to see data statistics calculated for the both parts separately. 

4) The content types are UGC, gaming/animation, sports, nature, shaking. Could the authors shed some light on these categories& Are them mutually exclusive, e.g., an UGC video cannot be classified as shaking, or cannot depict nature? How these categories are assigned?
correctness: The submission is a video dataset and a video quality assessment benchmark associated with the newly collected data. The metrics are common and widely-used. The data acquisition protocol relies on the statistical rules and includes a verification step. Overall, both the data acquisition and evaluation procedures seem to be correct and appropriate.
clarity: The paper is clear and easy to follow.
relation to prior work: The authors provide a quantitave summary of benchmarks on video quality measurement in Tab. 2 and explain their motivation. As stated, the most recent video encoders produce artifacts, which are not considered by the quality metrics, making them irrelevant. The proposed benchmark addresses this issue by including videos compressed via multiple modern video encoders.
additional feedback: See Weaknesses and Documentation.
review_title: A well-positioned and thorough study of video quality assessment

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: There are many video and photo quality metrics. However, most of them focus on resolution and quality subsets, temporal pooling, and computational-complexity-evaluation, but ignore video compression quality, which has become increasingly important in today's network era. As a result, the authors present the largest compressed video dataset, which include new coding standards and subjective ratings.
strengths: (1) First, because of the high cost and complexity of implementation, most previous video datasets with scoring is small-scale and performed at the laboratory scale. However, in this work, the authors not only use extremely large scale crowdsourcing to perform the video scoring annotation work, but also set up a reasonable mechanism to detect mislabeling. 
(2) Meanwhile, in the process of video data selection, the authors adopt the K-Means clustering method, thus dividing the videos into 87 classes based on spatio-temporal complexity, which greatly ensures the diversity of the dataset. 
weaknesses: It is a great pity that we do not have a full picture of the dataset due to the authors' choice of partial disclosure.
correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: No
review_title: review for “Video compression dataset and benchmark of learning-based video-quality metrics”

Meta-review: metareview: The submission receives 5 reviews form 5 reviewers and all are positive about the proposed dataset presented in the submission. AC reads all reviews and discussions and agree with the reviewers. AC recommends to accept the submission as a spotlight.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept