Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a framework for having multi-modal adversarial attacks in self-driving domain, and uses both pixel images and LiDAR attacks. The authors perform extensive experiments grounded in practical self-driving settings, and proposes consistent perturbations among the input modalities. 
strengths_and_weaknesses: Strengths:

The paper made a clear contribution to the study of multi-modal adversarial attacks in self-driving domain. The design of the experiments are technically sound, and the authors gave a clear and extensive literature review of the prior works and introduced the method in this paper pretty well. 

Weaknesses (or some thoughts upon the paper):

One thought I'm having is whether it's worth considering more subtle adversarial attacks on top of the adversarial mesh. For example, in image classification adversarial attacks work there has been minor changes on road signs, changing colors of certain parts of the images etc that are semantically meaningful to human's discretion. I wonder if the same thing could be done to the images and LiDAR inputs in the self-driving setting.

Also, is it possible to have some qualitative analysis of the result of detecting adversarial attacks? For example, it could show some success / failure examples of different methods and therefore shows how this method outperform the previous ones in a qualitative way.

One other minor point is potentially some implementation details could go into the appendix, and the main paper could be spent on more discussion / analysis. 
summary_of_recommendation: I recommend the paper for acceptance as multi-modal adversarial attacks is an interesting yer understudied problem in self-driving domain, and the authors has demonstrated success in attacks in multi-model inputs. 
issues: It would be great if the authors could respond to some of the concerns listed in the Strengths and Weaknesses section.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses the problem of assessing the robustness of
multi-sensor neural networks, as used in robotic applications. The
authors develop a physical-like attack (by simulating an object on the
vehicle in front) such that the attack affects both measurement
modalities (namely, images and lidar) in a consistent way. The authors
provide an evaluation over two datasets, which suggests that using
multi-sensor networks does improve robustness to individual-sensor
attacks but may be susceptible to multi-sensor attacks.
strengths_and_weaknesses: The paper is well written, and the approach is clear. I like the idea
of using sensor fusion to improve robustness to adversarial attacks.

My first comment has to do with the paper's contribution. The authors
claim that their main contribution is developing an attack that is
consistent across different sensor modalities. However, no
quantitative evaluation of the proposed attack generation is
provided. How can one tell (quantitatively) that the proposed attack
is physically consistent across sensors? A real-world example(s) would
also help demonstrate that this attack choice is valid.

My second comment is that I think the sensor fusion aspect is not
sufficiently explored. In particular, it seems that the utilized
sensor fusion method is putting too much weight on the image
processing part, which is why even image attacks on their own are
quite successful. Instead of retraining the detectors using
adversarial training, I think that coming up with a better sensor
fusion approach would bring more robustness in general. The reason for
this is that adversarial training is performed for the specific attack
implemented in this paper. On the other hand, a robust sensor fusion
approach would make sure that the physical attack needs to be very
precisely coordinated in order to modify the various sensor modalities
in an adversarial way.
summary_of_recommendation: Overall, I think that using sensor fusion to improve robustness is a promising research direction and should be explored further. Although I think the paper could provide better sensor fusion analysis, I think it is a good first step in this field.
issues: N/A
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an adversarial perturbation method to multi-modal inputs (e.g., LIDAR point cloud & image) showcase the susceptibilities of multi-sensor object detection systems on self-driving cars. Different from existing works that only investigate proof-of-concept adversarial methods on image data via perturbation with imperceptible noises (whose physical realizability is questionable),  the presented method focuses on physically realizable attacks via (adversarial) object insertion. In this approach, the adversary is represented via a parametrized texture mesh, which can be optimized and rendered into an object to be mounted on top of self-driving cars.

This is achieved by inserting the mesh into the LIDAR point cloud and project it onto the original 2D image, from which pixel depths can be estimated via an off-the-shelf depth estimation algorithm. These results in parameterized LIDAR & image modifications (their defining parameters are those of the texture mesh) which can be optimized to maximize the rate of false positive & false negative over the detection algorithm's bounding box proposal. The optimized texture mesh is then treated as an input-agnostic adversary that can be inserted in any (test) images & corresponding LIDAR point cloud. The effectiveness of the proposed method is also studied empirically in a transfer setting where it is computed on one dataset (e.g., KITTI) but applied & tested on another (e.g., XENITH). All reported results show promising & interesting results. 
strengths_and_weaknesses: STRENGTH:

The method is well-motivated from a very practical angle. Instead of simply demonstrating a proof-of-concept via crafting imperceptible noises to perturbs images adversarially (which might not be realizable physically), the proposed method studies a new method that aims to render a physical object with adversarial texture that can be naturally inserted into the scene to perturb both the LIDAR & image input to the object detection system on self-driving car.

The paper is very well-written & the experiments are also well-presented, which sufficiently demonstrate the effectiveness of the proposed method.

From the proposed attack, an adversarial training algorithm is also presented which helps decrease the success rate of the attack, thus making this a completed & motivating research.

WEAKNESSES:

I noticed that the trade-off coefficient of the Laplacian regulariser is rather small in the experiment, which apparently makes the size of the mesh relatively large (as shown in many visual excerpts). This unfortunately makes the adversary less realistic & realizable -- while the attack is effective, the perturbed images do not look natural at all.

Could the authors elaborate more on the effectiveness of the method with varying size of the texture mesh? In addition, I think it would also be important to compare with image-based attack methods (e.g., simply put a sticker on the host vehicle rather than mount a 3D object on top of it) to highlight the advantage of being able to create an attack on multiple sensor modalities.

Furthermore, the attack method has been evaluated against specific-defense methods but it remains unclear how effective it is against black-box defense methods? It would be great if the authors could comment on whether such experiment is possible?

Another (more minor) point: For thoroughness, please consider running the evaluation on multiple, independent data split to report the confidence of the success rate estimation. 
summary_of_recommendation: In general, I like the proposed idea, especially its practical angle on a physically realizable attack via natural object insertion. That being said, I feel that the evaluation here is still pretty much artificial at this point but I acknowledge that this is an interesting concept that might inspire more follow-up investigations. Given this, my rating for this work is weak accept. 
issues: Please refer to my specific comments & questions in the weakness section above.
review_title: null

Meta-review: This paper investigates the problem of adversarial robustness of multi-sensor perception systems, with a focus on the autonomous driving domain. While the reviewers are generally positive about this paper, they also raise a number of important concerns. The authors should carefully address the reviewers' comments in their rebuttal. 

UPDATE POST DISCUSSION PHASE: I would like to thank the authors for their comments and clarifications during the discussion phase. The reviewers agree that this paper provides a valuable contribution, and I concur with this assessment.
Decision: Accept (Poster)