Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - The paper proposes a framework for multiparty perception: on leverage multiparty computing techniques in a fully observable environments (with multiple cameras), such that the agent learns to take actions (e.g., driving direction) without observing the raw images from the multiple cameras.
- The core idea is that only encrypted ciphertexts extracted from the cameras are publicly exchanged between the parties, which are used to determine the actions.
- The approach is evaluated on an "obstacle world" dataset which will be released publicly. The goal of the agent is to reach a goal location while avoiding obstacles.
- Evaluation indicates that the performance of the proposed model reaches accuracies very close to the plaintext model (96.9% vs 97.1%).
strengths_and_weaknesses: ### Strengths
1. The problem tackled by the paper is well-motivated: while fully observable environments are highly useful for navigation, they incur a privacy cost. The approach suggests MPC can help achieving great performances with rigorous privacy guarantees.
2. The authors plan to publicly release the dataset which will be beneficial to the community.
3. The evaluation is rigorous and is accompanied with appropriate baselines and discussions.
4. The writing of the paper for the most part is clear and precise.

### Concerns

**1. Agent vs. "Party" vs. Camera views**
- The concern is to do with difference between agents and parties - are they the same? As in, is "two-party" case as shown in Figure 2 involves two agents? In which case, it appears that that first person view observed by each party would change making it unclear on how actions generalize between different agents with different goals.
- Alternatively, does one agent taking an action entail receiving encrypted ciphertexts from all other agents (each with their own first person view and computations)?

**2. (Nitpick) Some conclusions in Qualitative Analysis**
- I appreciate the authors presenting and discussing the success/failure examples of the approach.
- However, some conclusions drawn from the failure modes are not evident. For instance, take the conclusion L269 "the agent incorrectly reasons the goal location is behind the obstacle location" isn't entirely evident in Fig. 7 (middle row, left).
questions: 1. See concern 1: I would appreciate some clarification on agents vs. parties (since I think in MPC literature, generally agents = parties?)
limitations: Yes, the paper has a good discussion on limitations and societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method for robot navigation from multiple third-person views that additionally attempts to preserve the privacy of the information captured by the cameras. This is accomplished by training the networks used in the model to operate on encrypted feature vectors, thus eliminating the need to share the original data. This allows for navigation without any party having access to all the information. The method is tested on a Gym Mini World map. The results show that the method has comparable accuracy to a non-private baseline. 
strengths_and_weaknesses: Strengths
=========
- Preserving privacy in robotics and computer vision applications is an area of growing importance and thus methods that can help achieve this goal are much needed. This is partially because images usually capture so much additional information that is not particularly needed for a task, and thus sharing them with a third party can have major privacy concerns. This is common in robot vacuum cleaners, for example, where computation is usually limited to the edge in order to preserve privacy. The proposed method is a good solution to this if the processing has to be done remotely.

- The method seems quite novel. While preserving privacy in neural networks is an active area of research, as far as I’m aware there aren’t any learning-based privacy-preserving navigation approaches. 

- The method is simple but effective. The method works with standard embeddings computed by a neural network that are then encrypted using modular arithmetic. The network then learns to perform the action using this arithmetic decomposition of the representations. This could be easily applied to any existing navigation solutions that make use of feature representations.

- The results show that the encrypted navigation pipeline is only a fraction of a percent less accurate than the corresponding plaintext pipeline which is quite surprising given that the networks operate on encrypted feature vectors. 


Weaknesses
==========
- If the action prediction network can successfully predict actions using the encrypted features, it might be possible to train a similar network to reconstruct the either the multi-view images or the images from the agent. The paper states “Each $\left[E_{\theta}\left(c_{i}\right)\right]_{p}$ is constructed such that it is insufficient to reconstruct anything about the scene” but it’s not clear why this is the case.

- The evaluation is only performed on the Gym Mini World map which is not very realistic. How would the method work in a real-world environment where there is a lot more variation between scenes and the map and cameras exhibit noise, occlusion and artifacts. Would the performance between the plaintext and encrypted methods still be similar or would the proposed approach be affected more by this variation and adverse conditions? 

questions: - How does the method ensure that each encrypted feature vector can't be used to reconstruct anything about the scene?
- How is the conversion from (assumingly) continuous feature vectors to integers done?
- Does the system need to share keys between the parties?
- Why during training (Eq 6) is finite field arithmetic not used?
limitations: The paper addresses the societal impacts adequately. However, I would probably suggest including some information on the limitations i.e. perhaps some information about the images could be reconstructed by repeated querying the navigation network? Could the system be attacked by a third party?
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper presents CipherNav, a novel framework for private multiparty perception for navigation. CipherNav aims to predict the sequence of actions needed for an agent to reach a goal destination. The inputs to CipherNav are (a) camera views from P participants, (b) the agent's ego-centric camera view, and (c) a map of the environment containing public information about the layout of roads. In order to preserve privacy between the participants, inputs (a), (b), and (c) are encrypted via arithmetic decomposition, such that each party receives a partial share of the secrets. The predictions predictions from each participant are then arithmetically combined to produce the final action sequence. In order to validate CipherNav,  a novel dataset based on the open-source Gym MiniWorld environment is generated with 5x5 maps. Obstacles are randomly inserted into the maps to impede the agent from reaching the goal state. Experiments on the generated dataset show that the CipherNav achieves comparable performance to a model operates on plaintext images.
strengths_and_weaknesses: **Strengths**
* Novel setting of private mutiparty perception for navigation is introduced. Given the increasing importance of privacy-preserving ML, this is an important setting to consider. 
* Novel dataset is introduced. The proposed Obstacle World Dataset, while simple, can serve as a good baseline for the task of private multiparty perception for navigation. 
* Results on the novel Obstacle World Dataset show that the proposed approach achieves comparable performance to a model that operates on plaintext images and does so with a reasonable computational overhead.
* The proposed CipherNav network architecture is novel and seems to work well for the proposed task


**Weaknesses**
* Limited technical contributions.
questions: 1. What are the technical contributions beyond applying SMPC to a new problem?
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: *Summary*

-  The paper proposes a framework for multiparty perception: on leverage multiparty computing techniques in a fully observable environments (with multiple cameras), such that the agent learns to take actions (e.g., driving direction) without observing the raw images from the multiple cameras.
- The core idea is that only encrypted features extracted from the cameras are publicly exchanged between the parties, which are used to determine the actions.
- The approach is evaluated on an "obstacle world" dataset which will be released publicly. The goal of the agent is to reach a goal location while avoiding obstacles.
- Evaluation indicates that the performance of the proposed model reaches accuracy very close to the non-private model (96.9% vs 97.1%).

*Reviews*

- The paper received 3 reviews with final ratings: 7 (Accept), 6 (Weak Accept) and 4 (Borderline Reject).
- In general the reviewers found the paper to be well-motivated (mitigate privacy costs), well-written, with rigorous experiments and a dataset contribution.
- The reviewers raised some minor concerns regarding terminology and presentation, which have been resolved and are easily addressed in a camera-ready version.
- The more substantive concerns raised by reviewers were:
  - It might be possible to train a network to reconstruct camera images from the encrypted feature vectors. The authors' have convincingly responded to this concern by pointing to prior work and providing an additional experiment.
  - Would the method work in a real-world environment where there is a lot more variation between scenes and the map and cameras exhibit noise, occlusion and artifacts? The authors' argue that the paper is a first step towards this aim, which would constitute follow-up work.

*Decision*

- It's the view of the AC that the 'Borderline Reject' review (N4st) did not actually identify any substantive weaknesses in the paper that would warrant rejection. Reviewer N4st's main concern is 'limited technical contributions' which is convincingly rebutted by the authors. I'm swayed by the two positive reviews; the paper should be accepted.
confidence: Certain
Decision: Accept