Robustness Certification of Visual Perception Models via Camera Motion Smoothing
Authors: Hanjiang Hu
Zuxin Liu
Linyi Li
Jiacheng Zhu
Ding Zhao
Venue: CoRL 2022 Poster (2022)
Keywords: Certifiable  Robustness
Camera Motion Perturbation
Robotic Perception

Abstract: A vast literature shows that the learning-based visual perception model is sensitive to adversarial noises, but few works consider the robustness of robotic perception models under widely-existing camera motion perturbations. To this end, we study the robustness of the visual perception model under camera motion perturbations to investigate the influence of camera motion on robotic perception. Specifically, we propose a motion smoothing technique for arbitrary image classification models, whose robustness under camera motion perturbations could be certified. The proposed robustness certification framework based on camera motion smoothing provides effective and scalable robustness guarantees for visual perception modules so that they are applicable to wide robotic applications. As far as we are aware, this is the first work to provide robustness certification for the deep perception module against camera motions, which improves the trustworthiness of robotic perception. A realistic indoor robotic dataset with a dense point cloud map for the entire room, MetaRoom, is introduced for the challenging certifiable robust perception task. We conduct extensive experiments to validate the certification approach via motion smoothing against camera motion perturbations. Our framework guarantees the certified accuracy of 81.7% against camera translation perturbation along depth direction within -0.1m ~ 0.1m. We also validate the effectiveness of our method on the real-world robot by conducting hardware experiments on the robotic arm with an eye-in-hand camera. The code is available at https://github.com/HanjiangHu/camera-motion-smoothing.

Robustness Certification of Visual Perception Models via Camera Motion Smoothing
Hanjiang Hu' Zuxin Liu
Linyi Li
Jiacheng Zhu' Ding Zhao
CCannege Mellon University (hanjianghu, dingzhao demu.edd,
Unniversity of Illinois at Urbana-Champaign
(zuxinl,jzhan) jzhuu)aandrew........ Candrew. .ww.e.ccm...................... cmu, edu, linyi20illimois.cm edu
Abstract: A vast literature shows that the learning-basee based visual perception model is sensitive adversarial noises but few works consider the robustness of robotic perception models under widely-existing camera motion perturbations. To this end, we study the robustness of the visual perception model under camera motion perturbations to investigate the influence of camera motion on robotic perception. Specificallyy, we propose motion smoothing technique for arbitrary image clas- sification models, whose robustness under camera motion perturbations could be certified. The proposed robustness certification framework based on camera mo- tion smoothing provides effective and scalable robustness guarantees for visual perception modules so that they are applicable wide robotic applications. As far we are aware, this is the first work to provide the robustness certification for the deep perception module against camera motions, which improves the trust- worthiness of robotic perception. A realistic indoor robotic dataset with the dense point cloud map for the entire room, MetaRoom, is introduced for the challenging certifiable robust perception task. We conduct extensive experiments validate the certification approach via motion smoothing against camera motion perturba- tions. Our framework guarantees the certified accuracy of 81.7% against camera translation perturbation along depth direction within -0.1m ~ 0.1 We also vali- date the effectiveness of our method real-world robot by conducting hardware experiment on robotic arm with an eye-in-hand camera. The code is available on https:///gthube com/Hany inngHu/ camera-motion-seee oootthing..
Keywords: Certifiable Robustness, Camera Motion Perturbation, Perception
Introduction
Visual perception has achieved great success in recent years by leveraging the powerful represen- tation capability of neural networks and large-scale datasets [1. 2, 3]. Deep lcarning models have been dominating a wide range of computer vision tasks, such as image classification [4, 5], object detection [6. 8] and segmentation [9, 10, 11]. However, applying deep perception models real- world robotic applications is still challenging. Since visual perception is the core upstream module of an autonomous robot system, failure can cause the robot to sense the environments incorrectly, which may result in catastrophic consequences. Therefore, developing trustworthy perception system that can guarantee functionality in diverse real-world scenarios is necessary [12, 13, 14].
We study the robustness of a visual perception system against sensing noise due to camera motion perturbation that commonly exists in robotic applications [15, 16, 14], which is important while challenging for trustworthy robotic applications. The difficulty arises from two perspectives: inter- model vulnerability and external sensing uncertainty. On the one hand, rich literature suggests that deep visual models are vulnerable to adversarial perturbations that would be stealthy to hu- man eyes: small perturbations added to the input of neural networks can significantly corrupt the perception performance [17, 18, 19, 20]. On the other hand, several recent works indicate that the perception performance also sensitive to data acquisition, such as sensor placement and sensing perspective [21, 22, 23, 24). Both internal vulnerability and external sensing uncertainty make it challenging to guarantee the robustness visual system in real- real-world robotic applications.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Base classifier
Correct
motion perturbation
Base classifier
Wrong
Correct for any camera rotation within 7
Motion smoothed classifier
Test on base model with berign data
Test baoe model wlse pertuttskin
smoothing
poinl
Cameraa molion smoothhin
Figure I: Certification framework via camera motion smoothing
Prior works propose several techniques to improve the visual perception system robustness [25, 26, 27], though most of them are demonstrated to be effective empirically and no theoretical robustness guarantees are given. recent line of work aims provide provable robustness certification veri- fication such that the perception system is guaranteed to function properly under any bounded adver- sarial perturbations [28, 29, 30] or semantic image transformations and deformations [31, 32, 33], which improves the trustworthiness of the perception model. However, most of them are studied un- der static 2D image datasets without viewpoint changes, while the robotic visual perception systems process 2D images projected from the 3D physical world, which may not be robust under different perspectives of the moving camera and make it challenging certify the robustness for real-world robotic applications.
To tackle the challenge, we first study the robustness of the robotic perception model against the camera motion perturbation. Next, we propose the first framework with a certified robustness guar- antee for robotic perception models against arbitrary bounded camera motion perturbations based on a novel motion smoothing strategy as shown in Fig. 1. Extensive experiments have been conducted on realistic indoor robotic dataset MetaRoom with the dense point cloud map as an given oracle for image projection, which is collected from the Webots simulation environment [34] to show the effectiveness of the proposed robustness certification method against camera motion perturbations. To the best of our knowledge, this is the first work to study the certifiable robustness of image-base based robotic perception under camera ego-motion based on motion smoothing. The contributions are summarized as follows.
We demonstrate that the camera motion perturbation can significantly influence perception perfor- mance and introduce motion smoothing improve robustness over camera motion perturbation. We propose a smoothing algorithm for any black-box image classification model such that its robustness against camera motion perturbations be certified by our certification framework. We conduct extensive experiments on the realistic indoor robotic simulation to validate the ef- fectiveness of the motion smoothing certification, achieving over 80% certified accuracy against any perturbations within radii of 0 or 7 for camera translation or rotation along the depth axis. Further experiments on real-world robot arm validate the effectiveness of camera motion smoothing for perception models.
2 Related Work
Robust Robotic Perception. The robustness of robotic perception has been studied from dif- ferent viewpoints. rich literature in the robust machine learning community shows that deep learning-based robotic perception models are vulnerable and can be easily fooled by adversarial samples [17, 18, 19, 20]. Another perspective for robotic perception the external sensing uncer- tainty, which is caused by sensor placement [21, 23], camera distortions [24], geometric outliers [14], long-teri term robustness [35, 36], sim-to-real adaptation [37, 10, 38], etc. However, the robust- ness of deep perception models given the perturbed projected images from 3D physical world with a moving camera sensor relatively understudied.
Certifiable Robustness under Perturbations. In recent years, significant number of approaches has been proposed to provide robustness certification for deep neural networks [39, 40]. In con- trast to empirical robustness approaches [41, 25, 26, 27), i.e., which train robust models against

---

adversarial perturbations, the robustness certification approaches aim to guarantee the accuracy of the perception model long as the perturbation magnitude is bounded by some threshold. Such robustness certification approaches have been proposed against both fp-bounded pixel-wise pertur- bations [28, 29, 30, 42, 43, 44] and semantic transformation or deformations [45, 31, 32, 33, 46), providing either deterministic guarantees based on function relaxations [46, 47, 48, 32] or high- confident probabilistic guarantees based on random smoothing [49, 31, 33, 50, 45]. However, they consider either 2D or 3D transformations. To the best of our knowledge, no prior work studies certi- fiable robustness associated with the movement sensors and 3D-2D projected images, despite the in trustworthy robotic applications. Therefore, we aim to bridge the camera motion perturbation with deep learning robustness certification for perception systems.
fact that it is commonly
3 Methodology
In this section, we introduce the robustness certification framework against camera motion perturba- tions through the motion-smoothed perception model. We first define the image projection in terms of camera motion. Then, we clarify the certification goal and define the camera motion smoothed classifier using image projection. Finally, we present the robustness certification for each decom- posed translation and rotation translation.
3.1 Image Projection with Camera Motion
We first define the positive projection in Def. based on the camera imaging concept in geometric computer vision [51]. The follow-up Def. 2 defines the relative projective transformation Φ(x, a:) parameterized by relative camera motion a with respect the image z at motion origin.
Definition 1 (Position projective function). For any 3D point P = (X, (X,Y,Z) Z) E I C I under the camera coordinate frame with the camera intrinsic matrix K, based on the camera motion t) E Z C R6 with rotation matrix R exp(0*) SO(3) and translation vector t E R3 we define the position projective function p I x Z + R² and the depth function D : P x 2 + R
for point P as
1 (p(P,o), D(P,a) KR KR (P-I,, 100,0)==(0..1))
Definition 2 (Channel-wist projective transformation). Given the position projection function p P. PxZ Z + 2 and the depth function D P x Z - IR over dense 3D point cloud I,, define the 3D-2D
global channel-wise projective transformation from C-channel colored point cloud V = C CB3 to H x W image gird X C RRCHHW as O: V.V. - parameterized by camera motion a E Z using Floor function [-],
Lerr, ((VV,a)e, = Ve,Pa, where Pa
argmin (PE [o(Pa))]((raa)
D(P, (:)
(2)
Specifically, if OV,,), 0), we define the relative projective transformation ob X 2 - X' as ((2,0) = OV..a).
3.2 Certification Goal and Motion Smoothed Classifier
Certification Goal. We consider the classification task as the most fundamental perception task. For any deep learning-based classification model h. given the projected image at the origin of camera motion the motion space Z, the certification goal is to find a set within radius Zradius C such that, for any relative projective transformation Φi E Zradius, with high confidence we have
h(x)= h(0(z, a)), Va E Zradius-
(3)
Based on the relative projective transformation & over camera motion space 2 defined above, we define the camera motion smoothed classification model as follows.
Definition 3 (Camera motion e-smoothed classifier). Let Φ : X - X be relative projective transformation given the projected image x at the origin of camera motion in the motion space Z, and let s ~ Ps be a random camera motion taking values in Z. Let h: X + be base classifier h(x)) = argmax ye p(y 1)), the expectation of projected image predictions Φ(z, e) over camera motion distribution Pe is q(y x;.) EgNP.Ply ((z,))). We define the E-smoothed classifier X + yas as
((z;e) argmax q(v x; argmax Ee- ((x,))). yEy
(4)
yEy
3

---

3.3
Certifying Motion- parameterized Projection with Smoothed Classifier
In order to achieve the certification goal with smoothed classifier, prior works [31, 50, 52] show that smoothed model can be certified if the image transformation is resolvable. To this end, we first show that the relative projection is generally compatible with the global projection, which indicates that image projection can be regarded as resolvable transformation.
Lemma (Compatible Relative Projection with Global Projection). With global projective trans- formation O : V x 2 - X' from 3D point cloud and a relative projective transformation @ X x Z + X' given some original camera motions, for any on E 2 there exists an injective, continuously differentiable and non- non-vanishing Jacobian function Yoo. : Z ++ Z such that
(5) The high-level idea for the proof is that SO(3) has the rules of multiplication and inverse operation which can be transferred to the Ya, function (referred as for convenience) in (5), and min-pooling defined in the projective transformation in (2) does not break these rules as well. The full proof can be found in the supplementary materials. Specifically, if the camera motion is with translation and exed-axis rotation, we have the following certification theorem.
6(O 01), 02) OVV, Ya; (a2)), V E V. 02 E Z.
Theorem (Robustness certification under camera motion with fixed-axis rotation). Let a E 2 C RR be the parameters of projective transformation & with translation (1 (2) E IR3 and fixed- axis rotation (0n1, Onz, 0na)² E & Ei n = I, suppose the composed camera motion E E z (l EO), a) given some E Z and zero-mean Gaussian motion Eo with variance oz, y. 0²2 o for tz. ty. tz,6 9 respectively, let PA.PB E [0, 1] be bounds of the top-2 class probabilities for the motion smoothed model, i.e.,
satisfies δ(x,81)
q x xo) 2 PA PB 2 max q x x.,0)). VFVA
(6)
Then, it holds that g(ő(1, o)) = g(2;60) ifa= = (tz, lu. 1,, On1, 0nz 0na)² satisfies
0
2
by
2
2
< (0 (pa) + (pB))
(7)
The proof sketch is that based on Lemma 1, the relative projection is resovable and compatible with the global projection, and specifically, y function in (5) will be additive Ja, (a2) = 0j + 02 with fixed-axis rotation. The full proof can be found in the supplementary materials following previous certification work [28, 31J. We remark that for the general rotation without a fixed axis, although ει a + en does not hold, the function can also be found based on multiplication in SO(3) since Lemma 1 holds in general cases.
Since all the camera motions can be regarded the composition of each l-axis translation or rota- tion, we make the following corollary to show the certification for any l-axis translation or rotation. Corollary (Certification of 1-axis motion perturbation). For any 1-axis camera motion perturba- tion a with non-zero entry a; satisfying a; < Ti (T>)(pA)- ক (pb)) under motion smoothing EO ~ (O, of), it holds that g(ó(2, a)); = go); the motion smoothed classifier g.
Remark 1. Cor. I directly follows Thm. 1 by taking only one non-zero entry in a E 2 C IR& where each entry of in a (y: L2, On,; Ong, 0ng))² means Tj: translation along depth-orthogonal horizontal axis, Ty: translation along orthh-orthogoal vertical axis, Te: translation along depth axis, Rz: rotation around depth-orthor orthogonlt pitch axis, Ry: rotation around depth-orthogonal yaw axis and Rg: rotation around depth roll axis, as shown in Fig. 3.
Therefore, we conduct comprehensive experiments for each 1-axis camera motion based on the Cor. and Remark I for robustness certification.
Experiments
In this section, we aim address two questions: 1) How does the perturbation of camera motion influence the perception performance empirically? 2) How can we certify the accuracy using the motion-smoothed model under the camera perturbation within some radius? To answer these ques- tions, we first set up simulated indoor environment MetaRoom with dense point cloud maps and conduct extensive experiments based on it. Besides, we also conduct real-world experiments on a robotic arm with an eye-in-hand camera to investigate motion smoothing in robotic applications. Supplementary materials contain more experiment details.
4.1 Experimental Setup

---

Figure 2: MetaRoom images under 6-axis perturbations (first two rows) and real-world robot with an eyc-in- hand camera (last two rows). Left to right: global scenarios, perturbations over T2, Tz, Ty. Rg, Ra, Ry
MetaRoom Dataset. To implement the 3D-2D projec- tion from point clouds to images through the camera, we first introduce the MetaRoom dataset based on We- bots [34]. The MetaRoom dataset contains 20 commonly- seen indoor objects and for each object, we place it on small table located in the center of an empty room with the size of 3 m X 3 m. For each object, the collected data is associated with the point cloud of the entire room, all the camera poses, and camera intrinsic parameters. We
The illustration of coordinates in MetaRoom and real robot with camera mo- tion.
Figure 3:
first reconstruct the 0.0025 m-density point clouds of the room together with each object on the ta- ble through random snapshots from an RGBD camera with 500 camera poses for training and 120 camera poses for testing. The objects are 0.1 m higher than the origin of the global coordinate. For the training set, the camera is oriented toward the origin and uniformly randomly moves within semi-spherical area of 2.3 m ~ 2.9 m radius with the yaw range of 360", the pitch range of 0 90- and roll range of -60 ~ 60%. The poses in the test set are 2.8 m away from the original with 6 fixed yaw angles uniformly distributed in 0 ~ 360", the pitch angle of 10,, and the roll angle of 0 Note that the poses in the training set are at least 15 away from any yaw, pitch, roll angle of every pose in the test set to force the model to generalize rather than just memorizing the training data. The coordinate is illustrated in Fig. 3.
Model Training. We adopt two off-the-shelf representative architectures, ResNet-18 18 and ResNet- 50 [4), to train base classifiers on MetaRoom for 100 epochs. For baselines, we adopt motion-specific augmentation as defense to train the Motion Augmented models while Vanilla models are not trained with the augmentation since the augmented classifier could generalize well to the images with corresponding perturbation following [31, 50J. The motion augmentations are consistent with smoothing Gaussian distributions with a shown Tab. 2. Here motion augmented and vanilla are different strategies to train a Base classifier for Smoothed model via motion smoothing in Def. 3. Evaluation Metrics. The Benign Accuracy is calculated as the ratio of correctly classified pro- jected images without any perturbation, which is used to show the robustness/accuracy trade- off [28]. Based on the literature on spatial robustness [53, 54] that gradient- attack methods [17, 41] do not perform better than grid-search-based attacks for spatial adversarial samples due to the highly non-convex optimization landscape in semantie transformation space, we use grid search to evaluate the worst-case perturbations. We uniformly sample 5 and 100 perturbed camera motions within the radius and consider the model is not robust if any of them is wrongly classified, and report the ratio of robust ones over the whole test set as 5-perturbed Empirical Robust Accuracy and 100-perturbed Empirical Robust Accuracy, respectively. To provide a rigorous lower bound on the accuracy against any possible perturbations, we report Certified Accuracy [31, 50] to evaluate the certification results, which is the fraction of test images that are both correctly classified and sat- isfy the certification condition of (7) within motion perturbation radius through smoothing, meaning models will predict correctly with at least this certified accuracy for any camera perturbation within the given motion radius with high confidence. Following convention [28], use the confidence of 99% and 1000 smoothing camera motions under zero-mean Gaussian distribution for test images.
4.2 Empirical Robustness against Camera Motion with Smoothing
We answer the question about the influence of camera motion perturbation on the perception per- formance empirically. For models trained with motion augmented defense, we compare the base
5

---

Camera Motion Types
Motion Aug. ResNet18
Motion Aug. ResNet50
Tg, radius [-0. 1.,m, 0.1m)
Base/ / Smoothed
Base / Smoothed
Benign Accuracy
0.858 0.842
0.875 0.867
100-perturbed Emp. Robust Acc.
0.758 / 0.817
0.775 0.825
Tz, radius [-0.05m, 0.05m]
Base/ / Smoothed
Base/ / Smoothed
Benign Accuracy
0.925 0.900
0.883 /08667
100tperturbed Emp. Robust Acc.
0.785 / 0.867
0.633 0.800
Tg radius [-0.05m, 0.05m]
Base/ / Smoothed
Base / Smoothed
Benign Accuracy
0.925 0.892
0.944
100-perturbed Emp. Robust Acc.
0.808 / 0.842
0.842 / 0.908
Rg, radius [-7", 7*]
Base / Smoothed
Base / Smoothed
Benign Accuracy
0.933/ / 0.958
0.933 / 0.950
100-perturbed Emp. Robust Acc.
0.875 / 0.892
0.867 0.917
Ra, radius [-2.5", 2.5%]
Base/ / Smoothed
Base / Smoothed
Benign Accuracy
0.975 0,950
0.925 0.942
100 100-perturbed Emp. Robust Acc.
0.908 0.892
0.850 0.917
Ry radius [-2.5%, 2.5%
Base/ / Smoothed
Base / Smoothed
Benign Accuracy
0.917/0.923 0.925
0.975 / 00999
100-perturbed Emp. Robust Acc.
0.867 0.925
0.933 0.983
Table I: The comparison between base and motion smoothed models which are both trained with motion augmentations in terms of benign and 100-perturbed empirical robust accuracy for all camera motions. The higher one between each base and motion smoothed model bold.
and smoothed classifiers over benign and empirical robust accuracy in Tab. 1. It can be seen that under each motion perturbation, the empirical robust accuracy is lower than the benign accuracy for the base and smoothed models, showing that the perception models are not robust under motion perturbations even with the motion augmented defense when training. However, the gap between empirical robust accuracy and benign accuracy of motion smoothed models become less than those of base models due the effective motion smoothing that improves robustness. Specifically, the motion smoothed models perform better in terms of empirical robust accuracy than the base models for camera motion perturbation along all axis. Interestingly, under a similar magnitude of radius, rotational smoothing will mostly increase the benign accuracy while translational smoothing tends to decrease the benign accuracy due to the robustess/accuraacy trade-off [28]. The comparison and analysis between 5-perturbed and 100-perturbed empirical robust accuracy can be found in supple- mentary materials Section B.3 and Table 4.
4.3 Comparison of Certified Accuracy
Here we aim show how we certify the robust accuracy using the motion smoothing strategy against motion perturbation, so we compare the smoothed augmented models with the smoothed vanilla models in Tab. 2. For the motion perturbations on each axis, under the same smoothing strategy, most smoothed augmented models have better performance in benign, empirical robust and certified accuracy compared to smoothed vanilla models, showing that the motion augmentation improves the overall perception performance, empirical and certifiable robustness. Besides, the certified accuracy is only slightly lower than empirical robust accuracy under the same perturbation radius, which shows that our certification guarantees are effective and scalable against all camera motion perturbations for different baseline models.
4.4 Ablation Study
Certified accuracy under different radius. Fig. 4 shows the certified accuracy with respect to the motion perturbation radius on each axis. We find that as the certified radius increases, the certified accuracy decreases and comes decaying certified accuracy. For motion augmented models, the certified accuracies beyond the certified radii of Translation z and Rotation y still remain high until two times larger than a in motion smoothing, while Translation y and Rotation x have quickly decayed certified accuracies, showing that Translation 2 and Rotation y can be better certified over larger perturbation radii. Comparison of different model complexity. Smoothed vanilla ResNet- 18 performs better in cer- tified accuracy compared to ResNet-50 within smoothing radii a. With motion augmented defense, although most empirical robust accuracy of base ResNet-50 is worse than base ResNer- 18 in Tab. 1, ResNet-50 has better certified accuracy after motion smoothing under larger perturbation radii for most 1-axis camera motions in Fig. 4, which indicates that motion -aggente smoothed classifiers
some radius. Smoothed vanilla models have slowly

---

T3 within radius [-0.1m, 0.1m]
Benign Acc
S-pert Emp. Acc Certified Acc
Smoothed Vanilla ResNet-18
0.858
0.817
0.792
Smoothed Augmented ResNet-18
0.842
0.833
0.817
0.1m
Smoothed Vanilla ResNet-50
0.675
0.617
0.558
Smoothed Augmented ResNet-50
0.867
0.850
0.817
Ox
Tz within radius [-0.05m, 0.05m]
Benign Acc
5-pert Hmp. Ace
Certified Acc
Smoothed Vanilla ResNet-18
0.825
0.783
0,700
Smoothed Augmented ResNet-18
0.900
0.875
0.833
0.05m
Smoothed Vanilla ResNet-50
0.767
0.675
0.508
Smoothed Augmented ResNet-50
0.867
0.825
0.708
By
Ty within radius [-0.05m, 0.05m]
Benign Acc
5-pert Emp. Acc
Certified Acc
Smoothed Vanilla ResNet-18 18
0.850
0.825
0.767
Smoothed Augmented R8
0.892
0.875
0.817
0.05m
Smoothed Vanilla ResNet-50
0.792
0.767
0.683
Smoothed Augmented ResNet-50
0.942
0.925
0.892
R₂ within radius [-7?, 7?]
Benign Ace
5-pert Emp. Ace
Certified Acc
Smoothed Vanilla ResNet-
0.817
0.742
0.608
Smoothed Augmented ResNet-18
0.958
0.933
0.883
7
Smoothed Vanilla ResNet-50
0.758
0.717
0.633
Smoothed Augmented ResNet-50
0.950
0.917
0.883
Rg within radius [-2.5%, 2.5%]
Benign Acc
5-pert Emp. Acc
Certified Acc
Smoothed Vanilla ResNet- 18
0.842
0.800
0.633
Smoothed Augmented ResNet-18
0.950
0.942
0.867
2.5
Smoothed Vanilla ResNet-50
0.767
0.742
0..567
Smoothed Augmented ResNet-50
0.942
0.933
0.867
0g R, within radius [-2.5", 2.5*]
Benign Acc
5-pert Emp. Acc Certified Acc
Smoothed Vanilla ResNet-18
0.892
0.875
0.708
Smoothed Augmented ResNet-18
0.925
0.925
0.917
2.5
Smoothed Vanilla ResNet-50
0.808
0.783
0.717
Smoothed Augmented ResNet-50
0.992
0.992
0.967
Table 2: The comparison hetween smoothed vanilla and smoothed augmented models terms henign, 5-perturhed empirical robust (5-perb Emp.), and certified accuracy for with all camera motions. The higher one between vanilla and motion augmented models is in bold.
10 10-perturbed Empirical Robust Accuracy
Model
Benign Accuracy
T.
T,
T:
Rs
Ry,
R2
Base Smoothed 83.3%
76.3%
78. 1%
79.8%
77.1%
74.6%
82.5%
78.9%
81.6% %
82.5%
77.2%
75.4%
83.3%
Table 3: Quantitative results of real-world robotic perception model. The perturbation range of translations
is 11.25mm, 1. 1.25cm] and perturbation range of rotations (Rz, Ry. R2)) is 1-2.5 2.5%). The variance zero-mean Gaussian smoothing distribution is 0.625cm translations and 1.25 for rotations. The higher values between the base and smoothed ones are bold.
(Tz, Tw,T.) T.)
with larger complexity tend be more certifiably robust to camera motion perturbations, although they may suffer from lower empirical robust accuracy due to overfitting.
4.5 Real-world Experiments
Experiment setup. We conduct hardware robotic experiments using the Kinova-Gen3 Arm of 7- DoF with an eye-in-hand camera in the pick-place task environment. These 6 objects with different shapes and colors and the perception model deployed on the arm is based on the ResNet50 classifi- cation model. For the data collection, following the spherical coordinate in Figure 3, 2500 images along the random trajectories are captured training set and test set includes 114 test poses with 19 non-overlapped fixed poses for all 6 objects. Following the metrics in Section 4.1, for both the base model and smoothed model with variance 0.625cm for all translations and 1.25 for all rotations, we adopt 10 uniform samples over [-1. 1.25cmm 1. 1.25cm] and 2-5.0 2. as empirical robust accuracy. We report benign accuracy without any perturbation using the base model for comparison. More details can be found in supplementary materials Section B.4.
Results and analysis. From Table 3, it can seen that the 10-perturbed empirical robust accuracy is lower than the benign accuracy for the base model, which means the small perturbations along
7

---

Translation smoothing 1.1m
Translation Smoothing 0.05m
Transiation smoothing -0.05m
100
100
Contict MUDAATCY (Q)
Conther PoDuNNY (M)
Smacrid Val bacheci
300 sted var Rester 36
20
Smactied Aegnersddd Ruskanc4
Incoteed Augin rad kamen 50
Smocred wwweeeee Su
8.00
8oce
8.000
8030
Rodissm ml
Radie om
Rodid mn)
Rotation smoothing
Rotation smoothing 2.5
Rotation smoothing 2.5*
Cartifed
bud Midi boraet Augi Auginered sertd00
aracl bu praitis bedao
30leed supearae Reclancs
20.3
37.3
2.30
Rackes
Figure Certified accuracy with respect to radius for all ego-motions
Figure 5: Smoothing process to improve robustness against camera motion of Rs (top) and T. (bottom). The left four columns are randomized smoothing samples, and the right column is the classification result after smoothing.
each axis do influence the performance of the real- sal-world robotic perception. Besides, our proposed smoothed model improves the robust accuracy against all the perturbations, and results of T, and R: after smoothing are very close to benign accuracy. Since the perturbations are small, the perturbation influence is not that significant. But the smoothing variance is also tiny enough to be reasonable and practical in robotic applications, the improvement is observable and eligible to validate the effectiveness of the smoothing method. See Figure 5 for qualitative results.
Conclusion
In this work, we study the robust visual perception against camera motion perturbation the pro- jective transformation from 3D to 2D. We propose a robustness certification framework via a camera motion smoothing approach to provide robust guarantees for image classification models for real- world robotic applications. We collect a realistic indoor robotic dataset MetaRoom with the dense point cloud map for robustness certification against camera motion perturbation. We conduct exten- sive experiments to compare the empirical robust accuracy with the certified robust accuracy for the motion smoothed model within large radii of 6-axis camera motion perturbations, i.e., translations and rotations, to guarantee the lower bounds of accuracy for trustworthy robotic perception. We further conduct real-worke robot experiments show the effectiveness of the smoothing method.
8

---

Acknowledgments
We gratefully acknowledge support from the National Science Foundation under grant CAREER CNS-2047454. We also would like to thank Prof. Bo Li from UIUC for the thoughtful feedback and Shiqi Liu from CMU for helping conduct the real robot experiment.
References
[1] E. landola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint arXiv: 1602.07360, 2016.
[2] H. Hu, B. Yang. Z. Qiao, D. Zhao, and H. Wang. Seasondepth: Cross-season monocu- lar depth prediction dataset and benchmark under multiple environments. arXiv preprint arXiv:201 1.04408, 2020.
[3] Deng, W. Dong, R. Socher, Li. Li, and L. Fei-Fei. Imagenet: A large-scale hicrar- chical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icee, 2009.
[4] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[5] A. Dosovitskiy, L. Beyer, A. Kolesmikov, D. Weissenborn, X. Zhai, Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transform- ers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[6] Redmon, S. Divvala, Girshick, and A. Farhadi. You only look once: Unified, real- time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788, 2016.
[7] R. Xu, H. Xiang, X. Xia, X. Han, J. Li, and J. Ma. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehhcce communication. In 2022 International Conference Robotics and Automation (ICRA), pages 3589.2589. IEEE, 2022.
[8] R. Xu, H. Xiang, Z. Tu, X. Xia, M.-H. Yang, and J. Ma. V2x-vit: Vehicle-to-everything cooperative perception with vision transformer. arXiv preprint 2xXiv:2203.10633 2022.
[9] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[10] H. Hu, Z. Qiao, M. Cheng, Z. Liu, and H. Wang. Dasgil: Domain adaptation for semantic and geometric-aware image-based localization. IEEE Transactions on Image Processing, 30: 1342 1353, 2020.
[11] R. Xu, Z. Tu, H. Xiang, W. Shao, B. Zhou, and J. Ma. Cobevt: Cooperative bird's eye view semantic segmentation with sparse transformers. arXiv preprint arXiv:2207.02202 2022.
[12] K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song. Robust physical-world attacks on deep learning visual classification. Proceedings of the IEEE conference computer vision and paltern recognition, pages 1625-1634, 2018.
[13] H. Hu, B. Yang. Z. Qiao, D. Zhao, and H. Wang. Seasondepth: Cross-season monocu- lar depth prediction dataset and benchmark under multiple environments. arXiv preprint arXiv:201 1.04408 2021.
[14] H. Yang and L. Carlone. Certifiably optimal outlier-robust geometric perception: Semidefi- nite relaxations and scalable global optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[15] J. F. Ferreira and J. M. Dias. Probabilistic approaches to robotic perception. Springer, 2014.

---

[16] G. Liu, A. Amini, M. Takac, and N. Motee. Robustness analysis of classification using re- current neural networks with perturbed sequential input. arXiv preprint arXiv:2203. 2022.
[17] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint
2014.
[18] A. Kurakin, Goodfellow, and S. Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 99-112. Chapman and Hall/CRC, 2018.
[19] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39-57. IEFE, 2017.
[20] C. Xiao, J. Y. Zhu, W. He, M. Liu, and Song. Generating adversarial examples with adversarial networks. In 27th International Joint Conference on Artificial Intelligence, IJCAI 2018, pages 3905-3911 1. International Joint Conferences on Artificial Intelligence, 2018
[21] Z. Liu, M. Arief, and D. Zhao. Where should we place lidars on the autonomous vehicle?- an optimal design approach. In 2019 International Conference on Robotics and Automation (ICRA), pages 2793-2799. IEEE, 2019.
[22] Feng. J. Lee, M. Durner, and Triebel. Bridging the last mile in sim-to-real robot perception via bayesian active learning. arXiv preprint arXiv:2 /547, 2021.
[23] H. Hu, Z. Liu, S. Chitlangia, A. Agmihotri, and D. Zhao. Investigating the impact of multi- lidar placement on object detection for autonomous driving. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 2550-2559, 2022.
[24] Z. Tang, R. G. Von Gioi, P. Monasse, and J.-M. Morel. A precision analysis of camera distor- tion models. IEEE Trunsuctions on Image Processing, 26(6):2694-277 2704, 2017.
[25] Tramèr, D. Boneh, A. Kurakin, I. Goodfellow, N. Papernot, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. In 6th International Conference on Learning Rep- resentations, ICLR 2018-Conference Track Proceedings, 2018.
[26] X. Ma, Li, Y. Wang. M. Erfani, Wijewickrema, G. Schoeneheck, D. Song. M. E. Houle, and J. Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference on Learning Representations, 2018.
[27] Tramer, N. Carlini, W. Brendel, and A. Madry. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems, 33:1633-1645, 2020.
[28] J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smooth- ing. In International Conference on Machine Leurning, pages 1310-1320. PMLR, 2019.
[29] V. Tjeng, K. Y. Xiao, and Tedrake. Evaluating robustness of neural networks with mixed integer programming, In International Conference on Learning Representations, 2018.
[30] E. Wong and 7.. Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pages 5286-5295. PMLR, 2018.
[31] L. Li, M. Weber, X. Xu, L. Rimanic, B. Kailkhura, T. Xie, C. Zhang, and B. Li. Tss: caansformation-speciff smoothing for robustness certification. Proceedings of the 2021 ACM SIGSAC Conference Computer and Communications Security, pages 535-557, 2021.
[32] A. Ruoss, M. Baader, M. Balunović, and M. Vechev. Efficient certification of spatial robust- ness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2504-2513, 2021.
[33] M. Alfarra, A. Bibi, N. Khan, P. H. Torr, and B. Ghanem. Deformrs: Certifying input defor- mations with randomized smoothing. arXiv preprint arXiv:2107.00996 2021.
[34] Webots. http://www.cyherbotic.. com. URL httphtttp/ww...brrbbbbbeeeee Mobile Robot Simulation Software.
Open-source
10

---

[35] H. Hu, H. Wang, Z. Liu, and W. Chen. Domain- invariant similarity activation map contrastive learning for retrieval-baaee long-term visual localization. JEEE/CAA Journal of Automatica Sinica, 922):133-32, 2021.
[36] H. Hu, H. Wang, Z. Liu, C. Yang, W. Chen, and L. Xie. Retrieval- based localization based on invaaininvariian feature learning under changing environments. In 2019 JEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pages 3684-321 3684-3689. IEEE. 2019.
[37] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[38] Z. Qiao, H. Hu, W. Shi, S. Chen, 7. Liu, and H. Wang. A registration-aided domain adap- tation network for 3d point cloud based place recognition. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1317-1322. IEEE, 2021.
[39] L. Li, T. Xie, and B. Li. Sok: Certified robustness for deep neural networks. arXiv preprint arXiv:2009.041. 2020.
[40] C. Liu, T. Arnon, C. Lazarus, C. Barrett, and M. J. Kochenderfer. Algorithms for verifying deep neural networks. arviv preprint arXiv: WwXiv:/90300677 1903.0675,, 2019.
[41] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning mod- els resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[42] H. Zhang, T.-W. Weng, P.-Y. Chen, C.-J. Hsieh, and L. Daniel. Efficient neural network ro- bustness certification with general activation functions. In Advances in neural information processing systems, pages 4939-4948 2018.
[43] G. Singh, Gehr, M. Püschel, and M. Vechev. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages, 3(POPL):1, 2019.
[44] S. Dathathri, K. Dvijotham, A. Kurakin, A. Raghunathan, J. Uesato, R. R. Bunel, S. Shankar, Steinhardt, Goodfellow, S. Liang, and P. Kohli. Enabling certification of verification- agnostic networks via -effory-efficient semidefinite programming. In H. Larochelle, M. Ran- zato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5318-5331, 2020.
[45] H. Zhongkai, C. Ying, Y. Dong, H. Su, and J. Zhu. GSmooth: Certified robustness against semantic transformations via generalized randomized smoothing. In International Conference on Machine Learning. PMLR, 2022.
[46] M. Balunović, M. Baader, G. Singh, T. Gehr, and M. Vechev. Certifying geometric robustness of neural networks. Advances in Neural Information Processing Systems 32, 2019.
[47] Mohapatra, T.-W. Weng, P.-Y. Chen, S. Liu, and L. Daniel. Towards verifying robustness of neural networks against a family of semantic perturbations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 244 252-252, 2020.
[48] T. Lorenz, A. Ruoss, M. Balunovié, G. Singh, and M. Vechev. Robustness certification for point cloud models. arXiv preprint arXiv:2103.1663 2021.
[49] M. Fischer, M. Baader, and M. T. Vechev. Certified defense to image transformations via randomized smoothing. In NewrlPS, 2020.
[50] W. Chu, L. Li, and B. Li. Tpe: Transformation-specife smoothing for point cloud models. In International Conference on Machine Learning. PMLR, 2022.
algorithms and applications. Springer Science & Business [51] R. Szeliski. Computer vision: Media, 2010.
[52] Z. Hao, C. Ying, Y. Dong, H. Su, J. Song, and J. Zhu. Gsmooth: Certified robustness against semantic transformations via generalized randomized smoothing. In International Conference on Machine Learning, pages 8465-8483. PMLR, 2022.

---

[53] L. Engstrom, B. Tran, D. Tsipras, L. Schmidt, and A. Madry. Exploring the landscape of spatial robustness. In International conference on machine learning, pages 1802-1811. PMLR, 2019.
[54] C. Sitawarin, Z. J. Golan-Strieb, and D. Wagner. Demystifying the adversarial robustness of random transformation defenses. In International Conference on Machine Learning, pages 20232-20252 PMLR, 2022.
12


Reviews:
[{"review_id": "wQbWMFY7LAHU", "reviewer": "Reviewer_C4we", "summary": "This paper addresses an important problem - robustness of perception models to input noise introduced by camera motion. The authors consider consider the problem of image classification on images generated from point clouds, and the camera motion noise is considered in the point cloud sensor. The bounded noise in the point cloud input can then be mapped to bounds on the projected image. Then training a smoothed classifier for the image enables provable robustness bounds on the input camera noise. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Robustness of perception models to hardware noise is a very important problem to consider\n- Approaching it from a certification perspective makes sense considering the importance of safety guarantees for robotics applications\n\n\nWeaknesses\n- The use of \"camera\" and \"image\" is very confusing  when the certification is applied to point cloud inputs (which are projected into an image, but the noise is still captured in the point clouds). \n- Lemma 1 is not used anywhere in the paper from what I can tell. Perhaps it can be moved to the supplementary materials\n- More background can be provided for the smoothed classifiers. It's a key component of the pipeline and can benefit from more introduction. In particular, equation 8 uses a key result of smoothed classifiers to obtain certification guarantees without providing the background.\n- Line 190-192 \"The robustness can be evaluated as the classification accuracy under worst-case perturbations. We uniformly sample 5 perturbed camera motions within the radius and consider the model is not robust.\" This metric seems a bit meaningless to me, if we take any classifier and sample random minor perturbations, the output is unlikely to change. I believe a better empirical evaluation is to perform an adversarial attack against the model and that should demonstrate a significant gap between the smoothed model and the vanilla model.\n- In my opinion, the work is incremental. I see it as an application of training smoothed classifiers for images and propagating the guarantees to the camera extrinsics. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper definitely studies an important problem in certifying the robustness of perception models due to sensor motion and noise in camera extrinsics. However I believe the manuscript and experimentation has a lot of room for improvement:\n- More clear delivery, I found the problem formulation and terminology quite confusing and the diagrams were not the most informative\n- More background can be provided on smoothed classifiers, since it's an integral part of the method\n- The empirical robust accuracy is evaluated by sampling 5 random perturbations and does not seem like a meaningful metric\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See my comments in strengths and weaknesses", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "nQr95SI8tWr", "reviewer": "Reviewer_BZ5K", "summary": "This paper addresses the sensitivity of visual perception model to camera perturbations.\nOverall, although the paper seems to provide meaningful results to computer vision community, it is not well suited for CoRL audience. It essentially hits all the below desk rejection criteria:\n\n* No robotics: This is pure a computer vision problem, and the connection to robotics is through the argument that perception is one of the tools that are used in robots. However, there is no robotics experiment involved and I found the connection to robotics as a stretch.\n\n* An algorithm that was only evaluated in simulation without credible evidence on the possibility of transfer to a real robot learning due to sim2real problems or data efficiency. The images from MetaRoom looks far from being close a realistic real-world robotics tasks.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Please refer to the summary.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Out of scope for CoRL.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Irrelevant to robotics", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Out of scope for CoRL due to the lack of any robotics experiment or evidence/results on how this could improve a robot learning problem.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "aWsTKyxkfM", "reviewer": "Reviewer_vwtf", "summary": "The paper presents a theory and an algorithm for certifying the robustness of a 2D image classifier against 3D camera motion disturbances. Building off of a series of studies on randomized smoothing [22, 25, 41, 43] that focus on perturbations in 2D image or 3D point cloud, the paper connects perturbation from 3D camera motion to 2D semantic perception through the projective relationship between colored 3D point cloud and its corresponding RGB image. The paper shows that by augmenting image classification training with the proposed random smoothing algorithm, the resulting classifier has provable robustness bound under camera motion perturbations. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Certifiable perception is of great importance to the general robotics community. The paper extends the random smoothing theory to perturbations camera motion that is potentially more practical in common robotics settings, e.g., object navigation, mobile manipulation, and on-board sensing for drones.\n- The problem setting of certifiable robustness of 2D perception under 3D camera rotation & translation is new.\n- The algorithm is practical and conceptually simple. \n\nWeaknesses:\n- The method section is hard to follow. I think the main reason is the lack of intuitive explanation / illustration of basic concepts such as random smoothing and mapping noises from 3D motion to 2D image space. My suggestion is to provide an intuitive explanation of the basic components and intuition of the proposed algorithm, and the strategies that the section will take to prove the correctness of the algorithm. The current Figure 1 is somewhat confusing. I would suggest further polish Figure 1 and cross-reference it in the Method section to enhance the exposition. \n- No quantitative evaluation for real robot experiment. \n- Practically speaking, how would one generate the perturbed training set without having access to the data generation process? This seems to be a big assumption that should be addressed in the limitation section. \n- I would appreciate more discussion on how to extend the framework to tasks beyond classification, e.g., segmentation, object detection, keypoint detection, depth estimation, which I believe would be of great interest to the robotics / geometric perception community.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall I believe the technical contribution is strong. But I would strongly urge the authors to rewrite part of the paper to further enhance the exposition. For details, see my strengths & weaknesses comments. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See my comments on weaknesses.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "aRtNE1MdaR", "reviewer": "Reviewer_MJ8E", "summary": "The paper tackles the problem of ensuring robustness certification of a visual perception model, under camera perturbations. This is defines as a requirement that the model output shouldn't change, for camera perturbations, within a bounded region. The proposed solution is a simple smoothing technique. The paper provides for a theoretical result establishing robustness certification. The experimental results appear to be solid and validate the idea. The problem considered is very timely and pertinent, and is addressed with sufficient depth. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written. Analyzed to good depth. Good theoretical content and solid experiments to justify the claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is a solid contribution in the evolving field of certifiable perception. It makes good contribution. The paper analyzes the problem considered in good depth, provides solid theoretical results as well as experimental results, enough to justify the claim. This is a strong accept. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "Minor comments:\n1. $\\mathcal{Z}_{\\alpha}$ being the set of all camera motion as $\\alpha$ also denoting an arbitrary element in the set, is confusing notation.\n2. In definition 2, $\\subset \\mathbb{R}^6$  seems like a typo.\n3. Certification definition in (4), there seems to be a mistake. If one has to find Z_radius as well (which is a subset of Z_alpha) then a trivial solution would suffice, wouldn't it?  "}}]

Author rebuttal:

Decision: Accept (poster)