Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this paper, the authors propose MVP-N dataset for multi-view object classification to combine information from multiple views for multi-view object classification. Their contributions are proposing a real-world multi-view fine-grained dataset with HPIQ annotations, summarizing recent multi-view-based feature aggregation methods and a new metric.
strengths: 1.	4 feature aggregation and 12 soft label methods are benchmarked on MVP-N.
2.	According to Table2, the proposed dataset includes validation set and view-level annotation. 

weaknesses: 1.	I cant find the definition of MCDU in this paper.
2.	There is no new feature aggregation method proposed in this paper.
3.	The multi-view object classification framework is very straightforward and there is no detail about how to train the network.
4.	There is no new proposed framework in this paper to exploit the advantage of this dataset (e.g. view-level annotation).

correctness: Yes, the claims are correct, however, there is a missing metric MCDU.
clarity: Yes.
relation to prior work: Yes, they show the difference in Table2.
additional feedback: Already mentioned in weakness.
review_title: MVP-N: A Dataset and Benchmark for Real-World Multi-View Object Classification

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The work provides an interesting dataset for multi-view object classification. The dataset will benefit the related research. Besides the dataset, it also provides a benchmark for the multi-view object classification. The work also provides a new metric and evaluation protocol for the dataset. 
strengths: 1. The collected dataset is interesting and fair. Plus, a lot of details of the dataset are given.
2. A benchmark of the tasks is given on the collected dataset.
3. The paper is well written and well motivated.
4. The experiments are comprehensive with released code.
weaknesses: Some points are confusing and need to be clarified in the revision.
1. what is the relation between "16k real captured views, and 9k multi-view sets" in line 39？ Are they overlapped or are they independent of each other? It is better to clarify the relation between "16k" and "9k".

2. In line 46, what does "high inter-class view similarity causes uncertainty of class labels for uninformative views, which generates multi-view label noise" mean? Concretely, how the uncertainty exists in the uninformative views?

3. For the "bounding boxes" in line 131, since the dataset has 10 experts to annotate the bounding box, how does the final bounding box is decided among the annotated 10 bounding boxes for each object in a view.

4. What does "sufficiently informative, less informative, and uninformative" mean？ It is better to have definitions or even a figure to introduce the difference between these three concepts.
correctness: Yes, it is correct and the collection is in a sound way.

clarity: Yes, the paper is well written and easy to follow.
relation to prior work: Yes, the authors provided a comprehensive discussion with an impressive table to compared with the prior works.
additional feedback: Please refer to the weaknesses and ethics parts.
review_title: The work is meaningful and well-designed. Besides an interesting dataset, it also provides benchmarks on the multi-view object classification. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The major contribution of this paper is a dataset that poses a challenging multi-view object classification problem. This dataset includes HPIQ annotations that reflect human intuition on what regions of an object may be important for differentiation of alike objects. In addition to this dataset, the authors also provide a benchmark of multiple existing feature aggregation methods and discuss their viability in real-world deployment scenarios. The authors evaluate the benchmark using a number of metrics, including a novel metric that they propose.
strengths: This work introduces data to solve a difficult object classification problem. This dataset likely has a sufficient number of samples to inspire diverse learning techniques, while still providing substantial validation and testing data. While the types of objects included in this dataset may be somewhat limited in form and function, this work inspires the application of fine-grained classification methods to any sort of similar object classification. Object classification at this granularity can be especially important to robotic grasping research. 

The HPIQ annotations are an interesting concept that prompts further investigation. On top of the evaluation of informative/uninformative combinations, the HPIQ annotations inspire other research questions to investigate human thought processes to determine which areas of an object are most important for similar-object differentiation.

This work extensively evaluates other methods and datasets, including statistical analysis of performance with error margins. 

Discussion of hyperparameters, compute hardware, supplemental work, availability of code and data contribute to this work's reproducibility and usability in the future. In-depth discussion of data collection procedures and setup facilitate expansions and reproducibility.
weaknesses: A main weakness of this paper is the clarity and quality of some technical descriptions. See "clarity" section of the review.

I do not understand the decision to make the validation and test sets as large as they are with respect to the training set. It seems that such a small training set is not sufficient for robust learning. 

Conclusion, societal impacts, future work discussions are too brief.

Authors mention the noise in their dataset but do not elaborate on the benefits of having this noise.


correctness: I feel that this dataset is constructed soundly, though the data collection methods (only a few object perspectives) and decisions to crop images to a relatively small image tile may limit future use cases of the dataset. The use of experienced human annotators and manual evaluation of annotation quality lends to the credibility of this dataset. 
clarity: The data collection methodology sections are very detailed, though I feel that my understanding of some other parts of the paper is limited due to inconsistencies in grammar and verbage. Namely, the parts of the paper that I feel should be reworked are: related work (soft label), training/validation/test split details and rationalization, novel soft label evaluation protocol, justification of conclusions, discussion of real-world deployment applications.
relation to prior work: It seems that this work is differentiated from previous work in a number of ways, including the composition of the dataset with respect to inter- and intra-class similarity, and the HPIQ annotations 
additional feedback: I will consider a stronger accept if you can more accurately explain the train/validation/test split decision and alleviate some of the technical description issues. Also, I think it is worth discussing the aforementioned dataset/annotation noise more and proposing uses for this noise in the "broader impacts" section.

It seems that this dataset has potential to enable solving of tough similar-item classification problems for everyday objects and may spur research into human inuition for object differentiation! I think that this dataset is a valuable contribution to the community, and despite the problems that I mentioned, I marginally accept.
review_title: 265 Review

Meta-review: metareview: ***Quality:***  
* \+  The presented work has strong use-case for multi-view object classification with diverse  examples for 44 products which are  captured  in multiple views.
* \+  The proposed dataset is also benchmarked with 4 feature aggregation and  12 soft label methods on the dataset which promotes strong adoption.
* \-  The proposed dataset has relatively few categories compared to larger-scale datasets.

***Clarity:***  
* \+ The final draft of the paper has added revisions that have addressed the reviewers concerns about clarity
* \- The paper still has lot of abbreviations that need to be connected.
* \+ The paper's description of the dataset is very detailed and clear

***Originality:*** 
* \+ Proposed dataset has been described by  reviewers as comprehensive in its collection of views and partitions
*  \- There are similar datasets released by Amazon and Alibaba that contain multi-instance product images which are larger scale.

***Significance:*** 
* \+ The reviewers' consensus is that the HPIQ annotations are interesting addition to the dataset to identify differentiating areas for an object.
* \+ The work is released on github with code and benchmarks which will ensure its reproducibility and reach.
confidence: 5: The area chair is absolutely certain
Decision: Accept