Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a theory and an algorithm for certifying the robustness of a 2D image classifier against 3D camera motion disturbances. Building off of a series of studies on randomized smoothing [22, 25, 41, 43] that focus on perturbations in 2D image or 3D point cloud, the paper connects perturbation from 3D camera motion to 2D semantic perception through the projective relationship between colored 3D point cloud and its corresponding RGB image. The paper shows that by augmenting image classification training with the proposed random smoothing algorithm, the resulting classifier has provable robustness bound under camera motion perturbations. 
strengths_and_weaknesses: Strengths:
- Certifiable perception is of great importance to the general robotics community. The paper extends the random smoothing theory to perturbations camera motion that is potentially more practical in common robotics settings, e.g., object navigation, mobile manipulation, and on-board sensing for drones.
- The problem setting of certifiable robustness of 2D perception under 3D camera rotation & translation is new.
- The algorithm is practical and conceptually simple. 

Weaknesses:
- The method section is hard to follow. I think the main reason is the lack of intuitive explanation / illustration of basic concepts such as random smoothing and mapping noises from 3D motion to 2D image space. My suggestion is to provide an intuitive explanation of the basic components and intuition of the proposed algorithm, and the strategies that the section will take to prove the correctness of the algorithm. The current Figure 1 is somewhat confusing. I would suggest further polish Figure 1 and cross-reference it in the Method section to enhance the exposition. 
- No quantitative evaluation for real robot experiment. 
- Practically speaking, how would one generate the perturbed training set without having access to the data generation process? This seems to be a big assumption that should be addressed in the limitation section. 
- I would appreciate more discussion on how to extend the framework to tasks beyond classification, e.g., segmentation, object detection, keypoint detection, depth estimation, which I believe would be of great interest to the robotics / geometric perception community.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall I believe the technical contribution is strong. But I would strongly urge the authors to rewrite part of the paper to further enhance the exposition. For details, see my strengths & weaknesses comments. 
issues: See my comments on weaknesses.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper tackles the problem of ensuring robustness certification of a visual perception model, under camera perturbations. This is defines as a requirement that the model output shouldn't change, for camera perturbations, within a bounded region. The proposed solution is a simple smoothing technique. The paper provides for a theoretical result establishing robustness certification. The experimental results appear to be solid and validate the idea. The problem considered is very timely and pertinent, and is addressed with sufficient depth. 
strengths_and_weaknesses: The paper is well written. Analyzed to good depth. Good theoretical content and solid experiments to justify the claims.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is a solid contribution in the evolving field of certifiable perception. It makes good contribution. The paper analyzes the problem considered in good depth, provides solid theoretical results as well as experimental results, enough to justify the claim. This is a strong accept. 
issues: Minor comments:
1. $\mathcal{Z}_{\alpha}$ being the set of all camera motion as $\alpha$ also denoting an arbitrary element in the set, is confusing notation.
2. In definition 2, $\subset \mathbb{R}^6$  seems like a typo.
3. Certification definition in (4), there seems to be a mistake. If one has to find Z_radius as well (which is a subset of Z_alpha) then a trivial solution would suffice, wouldn't it?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses an important problem - robustness of perception models to input noise introduced by camera motion. The authors consider consider the problem of image classification on images generated from point clouds, and the camera motion noise is considered in the point cloud sensor. The bounded noise in the point cloud input can then be mapped to bounds on the projected image. Then training a smoothed classifier for the image enables provable robustness bounds on the input camera noise. 
strengths_and_weaknesses: Strengths
- Robustness of perception models to hardware noise is a very important problem to consider
- Approaching it from a certification perspective makes sense considering the importance of safety guarantees for robotics applications


Weaknesses
- The use of "camera" and "image" is very confusing  when the certification is applied to point cloud inputs (which are projected into an image, but the noise is still captured in the point clouds). 
- Lemma 1 is not used anywhere in the paper from what I can tell. Perhaps it can be moved to the supplementary materials
- More background can be provided for the smoothed classifiers. It's a key component of the pipeline and can benefit from more introduction. In particular, equation 8 uses a key result of smoothed classifiers to obtain certification guarantees without providing the background.
- Line 190-192 "The robustness can be evaluated as the classification accuracy under worst-case perturbations. We uniformly sample 5 perturbed camera motions within the radius and consider the model is not robust." This metric seems a bit meaningless to me, if we take any classifier and sample random minor perturbations, the output is unlikely to change. I believe a better empirical evaluation is to perform an adversarial attack against the model and that should demonstrate a significant gap between the smoothed model and the vanilla model.
- In my opinion, the work is incremental. I see it as an application of training smoothed classifiers for images and propagating the guarantees to the camera extrinsics. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper definitely studies an important problem in certifying the robustness of perception models due to sensor motion and noise in camera extrinsics. However I believe the manuscript and experimentation has a lot of room for improvement:
- More clear delivery, I found the problem formulation and terminology quite confusing and the diagrams were not the most informative
- More background can be provided on smoothed classifiers, since it's an integral part of the method
- The empirical robust accuracy is evaluated by sampling 5 random perturbations and does not seem like a meaningful metric

issues: See my comments in strengths and weaknesses
review_title: null

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper addresses the sensitivity of visual perception model to camera perturbations.
Overall, although the paper seems to provide meaningful results to computer vision community, it is not well suited for CoRL audience. It essentially hits all the below desk rejection criteria:

* No robotics: This is pure a computer vision problem, and the connection to robotics is through the argument that perception is one of the tools that are used in robots. However, there is no robotics experiment involved and I found the connection to robotics as a stretch.

* An algorithm that was only evaluated in simulation without credible evidence on the possibility of transfer to a real robot learning due to sim2real problems or data efficiency. The images from MetaRoom looks far from being close a realistic real-world robotics tasks.
strengths_and_weaknesses: Please refer to the summary.
robotics_focus: Irrelevant to robotics
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: Out of scope for CoRL.
issues: Out of scope for CoRL due to the lack of any robotics experiment or evidence/results on how this could improve a robot learning problem.
review_title: null

Meta-review: metareview: This paper addresses the problem of ensuring robustness certification of a visual perception model, under camera perturbations via a smoothing technique.

The major contribution is the theoretical analysis to establish robustness certification for visual perception under camera perturbations.

The authors addressed the reviewers' concerns in a thorough manner, including a real-world robotic experiment which demonstrates the value to the robotics learning community.


confidence: 2: The area chair is not sure
Decision: Accept (Poster)