Private Multiparty Perception for Navigation
Authors: Hui Lu
Mia Chiquier
Carl Vondrick
Venue: NeurIPS 2022 Accept (2022)
Keywords: Visual Navigation
Privacy
Multi-party Computation

Abstract: We introduce a framework for navigating through cluttered environments by connecting multiple cameras together while simultanously preserving privacy. Occlusions and obstacles in large environments are often challenging situations for navigation agents because the environment is not fully observable from a single camera view. Given multiple camera views of an environment, our approach learns to produce a multiview scene representation that can only be used for navigation, provably preventing one party from inferring anything beyond the output task. On a new navigation dataset that we will publicly release, experiments show that private multiparty representations allow navigation through complex scenes and around obstacles while jointly preserving privacy. Our approach scales to an arbitrary number of camera viewpoints. We believe developing visual representations that preserve privacy is increasingly important for many applications such as navigation. 

Private Multiparty Perception for Navigation
Hui Lu, Mia Chiquier, Carl Vondrick Department of Computer Science
Columbia University
New York, NY 10027 (h13231, mac2500, (v2428)0columbb edu
https:/ /pppss//visaampccccm ca columbia. edu
Abstract
We introduce framework for navigating through cluttered environments by con- necting multiple cameras together while simultancously preserving privacy. Oc- clusions and obstacles in large environments are often challenging situations for navigation agents because the environment is not fully observable from a single camera view. Given multiple camera views of an environment, our approach learns to produce multiview scene representation that can only be used for navigation, provably preventing one party from inferring anything beyond the output task. On a new navigation dataset that we will publicly release, experiments show that private multiparty representations allow navigation through complex scenes and around obstacles while jointly preserving privacy. Our approach scales to an arbitrary number of camera viewpoints. We believe developing visual representations that preserve privacy is increasingly important for many applications such as navigation.
Introduction
Navigation through cluttered environments is a fundamental challenge in computer vision and robotics, with many applications autonomous vehicles and assistive technology. In the typical settings, agents receive camera view of the surroundings and the task is to predict the sequence of actions needed reach goal destination (Figure([eeft), Today, many methods exist for solving this problem
when the environments are fully observerable or predictable []] 3l 37 20 both in simulation
22 and the physical world [5] [3] However, large environments often have many obstacles and
occlusions, which create irreducible uncertainties that cannot be resolved from just single input camera view.
Recently, the field has proposed different methods to combine multiple camera views together in order create complete representations of scenes 31 22
These methods use the geometry of cameras and visual correspondences between views to learn to reconstruct maps of environments, and they highly efficient, scaling to the size of full neighborhoods 31. Since multiview representations make the environment fully observerable, they allow for efficient navigation through complex scenes. However, while these representations have many exciting applications, the concern for invasion of individual privacy is significant. Without third parties that we can trust, it is increasingly important develop scene representations that safeguard individual identity and other sensitive information contained inside visual data.
In this paper, introduce framework that connects multiple camera views together without revealing private information about any of the camera views (Figure right). Our approach is based on the observation that any Boolean circuited parties such that each party can only learn the output of the circuit and nothing else (38 39 Our main result that neural network inference for navigation is compatible with secure multiparty computation and can be distributed between multiple camera views in this way. Our model creates
be cryptographically distributed between multiple
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Puble Information
pradiction
Privtto infermation
Goel
Figure I: Secure multiparty computation for perception and navigation through cluttered enviromments. (left) We show navigation task where the red needs to navigate to the goal position, but there are obstacles that are occluded. Calculating the most efficient path to the destination requires planning ahead avoid obstacles. (right) Our framework creates a representation from multiple camera views that preserves privacy by only allowing the representation to be used for path prediction, and nothing else (such as facial recognition).
representation of the scene that prevents any other party from accessing the contents of the scene, and guarantees that the representation can only be used for predicting the actions needed for the navigation task (and not face recognition, for example).
On a new navigation dataset that we will publicly release, experiments show that private multiparty representations allows navigation through complex scenes while preserving privacy. Compared to plaintext models, our encrypted model results in only a minimal drop in navigation success rate (a gap less than 1%). The approach scales to arbitrary number of camera views while preserving privacy, making it possible to scale the method to large spaces. When there are multiple valid paths from the source to the destination, the method frequently finds the most efficient path.
The main contribution this paper a framework for multiparty perception, which preserves privacy for navigation tasks. The remainder of this paper will describe the architecture and representation for performing this task. Section briefly reviews the related work in multiparty computation and robotic path planning. Section presents our approach securely distribute the computation among multiple parties, where each party is a camera. Section 4 analyzes the performance and efficiency of our approach new navigation dataset. Due to the ubiquity of cameras today, we believe developing visual representations that preserve privacy is increasingly important for many applications such as navigation, and we will publicly release all code, data, and models. We call our method CipherNav.
2 Related Work
We briefly review related work in multiparty computation, visual navigation, and robot path planning. This paper integrates secure multiparty computation into visual navigation.
Multi-party computation (MPC). Secure multi-party computation (MPC) KS is a major subfield
of cryptography that allows mutually distrusted private data owners (i.c. multiple parties) compute
a function over their private inputs with minimal knowledge transfer [27
g 39 The proposal
of MPC has led many interesting applications, such Yao's millionaire's 's problem where two
millionaires try to figure out who is richer without revealing their actual wealth 3X
MT. Other 17 60,, privacy preserving
18
applications include secure auctions [3), secure machine learning [16
genomics 35, [2], and more R common example of MPC protocol includes secret sharing where Knowledge of shares able to reconstruct the secret whereas knowledge of - 1 shares are not able to reconstruct the secrets The majority of the diseussion in this paper assumes /
private data owners split their secrets in into shares 27 g
Multiparty computation in machine learning. Privacy preserving machine learning has gathered much interest in recent years to integrate multi-party computation with machine learning [6]
30
25
17
33
There are protocols that assume two-party 25 21 three parties [7
30 33 four
and arbitrary parties [] computation. Most frameworks assume semi-hones security while others are secure against malicious parties with honest majority assumption Security against malicious parties comes with computation overhead [17]. However, any known protocols with semi-honest security can be converted to malicious security with an additional zero knowledge proof [19] Frameworks in this paper assumes semi-honest security level.
parties 6
16
30
25
21
34
Partial observability in robotic path planning. Partial observability and the lack of full knowledge of world state have been long- standing problem in robotic navigation 2
28 [5]
The existing

---

Party
Camera
views
[C]
Public
Camara
Canara
fealures
Gy
computioon
First person view
Observation
[0]+
features
Map
Map fealures
[M]
Gy
[A]
[M]
Figure 2: CipherNav action prediction network architecture (for clarity, only illustrate the two-party case). Public information is gray, agent's private information is blue and external cameras' private information red. Private operations are locally computed by each party and secret shared in an encrypted manner.
solution either assumes fully observable and predictable environments
attempt environment mapping to achieve better navigation [36 [2] Existing solutions include modelling navigation partially observable markove decision process (POMDP) [4 2
reconstructing the
scene using neural fields to reason about occluded regions through attention based networks 32
3 Multiparty Path Prediction
We present model to utilize additional private information to achieve better navigation path planning in privacy preserving way. Particularly, in self-driving navigation and robotic path planning contexts, secure multi-party computation and secret sharing protocols can be used to encrypt the contents from each camera such that the representation can only predict navigation steps. By definition, parties will able to infer additional information other than what is reasonably inferred from the final action prediction sequence output S
Il K8 39
3.1 Action Prediction for Visual Navigation
Our goal is predict the sequence of actions {oo....ae) needed reach a goal destination in an environment, where the discrete action space E E$ consists of the four directions along a compass (move north, south, east, west) addition to action that corresponds to remaining still. order to produce this action sequence, the model will be conditioned on the agent's 's egocentric view and map of the environment m, which contains public information about the layout of roads. The maps can also contain information that is private to themselves, such as their goal destination. However, maps will lack any information that is private to other agents, such as the locations of objects and obstacles. The obstacles could block a path, requiring the agent to take a detour.
In order produce paths that efficiently navigate around obstacles, including obstacles occluded to the agent, our model will additionally condition on set of multiview camera images {c; ) where each camera will produce an image. Multiview images make the environment fully observerable (albeit encrypted in our case). We write our model as F, which predicts the next action to perform given the observations:
Unext F((c)) m) S.t. Maext f- M(m,a) and Onext R(o, a))
where F' a neural network that we will describe next. After each step. the agent is able to receive the updated map and the updated egocentric view. We denote these updates from the world with Ω.
3.2 Encryption and Secret Sharing
Unlike conventional multiview representations, we need to ensure that no information about each party is shared to other parties, except for the information necessary for navigating to the goal destination. We will achieve this by creating representations for each observation that encrypted. Let Es(c) be the feature encoder for the camera and agent views, and let (m(m) be the feature

---

encoder for the map. Our model learns to predict the action with privacy guarantees through the arithmetic decomposition:
0, G, 10)p ( (M],) mod Q
(2)
p=0
which operates over a finite field Q. The summation is over all P parties the system. We use notation [C]p to denote encrypted representation of the scene, which is the secret share that party p receives for all camera views. It is computed through the concatenation:
- - 11.ooolooll,,......EE.o Loolew s.t. Eo(c)) - -IE mod Q
(3)
p=U
where the constraint on the right hand side follows the arithmetic secret sharing protocol. Each Ep(c)) can be broken into P secrets, denoted as [F)(c))p which are only sent to their respective party. Each single [c)]], is constructed such that it is insufficient to reconstruct anything about the scene other than what is deducible from the final outputs [27 g However, operations can still be performed on [c)() such that when all shares combined, the training task (and only the training task) be predicted. We similarly define encrypted representations for both the agent's view and the map:
10 [Eo(o)), s.t. Eσ(o)
Σ
[Eo[o)]p
mod Q
(4)
P [M, = [m)l s.t. Em(m) Σ
[m)) p=U
mod Q
(5)
Crucially, this establishes a private visual representation where each party receives just shares of the secret [C]p, [0)ps [M].. which corresponds to a random large integer that does not reveal any information about the original secret feature.
This formulation extends to an arbitrary number of parties. In our experiments, demonstrate results for both P = parties and P = parties. As long as one party is honest in this protocol, the full protocol remains secure M. Assuming one piece of private information per party, with an increased number of parties, more information is given to the action predictor network.
3.3 Learning and Optimization
We use the mean- mean-squared loss function to train the model F in ciphertext. For training, assume we have a collection of ground truth sequences for the optimal path between a source and destination. We optimize the following learning problem with stochastic gradient descent:
min ((c.o.m,e) [(a a)?]
where à = F( 0, m)
0.0.7
Since action sequences variable length (up to maximum), the model also predicts stop symbol represented by O. We leverage the stop token to mask the section of the sequence following the stop token to prevent gradient back propagation in steps after the stop token. We use one-hot vector to represent the action space a E IRs. where the arg max corresponds to the action to perform.
We originally experimented with a cross-entropy loss, but the optimization failed to converge. Computing cross entropy requires logarithmic and exponential functions that are hard to compute in muuli--paty computation settings. For example, logarithmics are evaluated through Houscholder iterations [T] and exponentials need to use limit approximation [6]. The approximations affect performance and lead to numerical overflow issues in practice.
We split our encryped training scheme into two stages. First, since the feature extraction does not require secret sharing, the view encoder Eg and the map encoder Eg are pretrained in plaintext. We compute the camera features, observation features and map features locally in order to save computation cost, as the multi-party computation is expensive. In the second stage, where secret sharing paramount, we replace the G action classification network with a new encrypted G that we train with multi-party computation from scratch, while freezing the pre-trace pre-trained encoders.

---

Free space
Slart location
Goel location
X
X
Hidden obstade
Correct path
Insorrect path
Figure 3: Top view visualization of random samples from the Obstacle World dataset. There are 12 random permutations of the map states. Each map has hidden obstacles that will impede the agent from reaching the goal. The agent may need to make detours, but least one viable path exists.
MAXA
Figure 4: Possible obstacles visualization by multi-view cameras. Obstacles occur in random locations, colors, and shapes.
3.4 Implementation Details
The network encoders include the view encoder Eg and map encoder Egg. The view encoder Eg consists of two layers of convolutional layers followed by two linear layers, with ReLU activations in between. Eg takes in a 45 60 dimensional input view image and transform into 32-dimensional vector. The map encoder Eg consists of 3 linear layers with ReLU activations. The map encoder takes discrete square map represented by 5x5 matrix, and outputs map features as a 128ddimensional passed into action classification network G,
vector. The view image features and map features which is another multi-layer perceptron with four linear layers and ReLU activations. Only ReLU chosen the activation function due to the increased complexity involved approximate other functions ciphertext.
We implement CipherNav in PyTorch Z and use the Crypten 16 framework for privacy srivacy-preserving neural network operations. Each model trained with 600 epochs and 0.01 learning rate. The batch size equals to 500 for ciphertext models and 100 for plaintext models. The plaintext models are trained on single GPU for 1.5 days. The ciphertext 2-party computation models are trained on single GPU for days.
4 Experiments
4.1 Obstacle World Dataset
To analyze our framework, we construct a new dataset, The Obstacle World, which we generated based the open-source Gym MiniWorld environment (7). Each instance of the environment is randomly generated out of 12 possible permutations of the map as shown in Figure 3 The agent starts at the top left corner of the map, and chooses randomized goal location. However, obstacles are scattered throughout the world that may prevent the agent from reaching the goal in shortest path. There I obstacles generated with random locations, random shape, and random colors $ The cones are rare wild card obstacles with lower chance of appearance compared to other obstacles. The dataset assumes that there is at least one possible obstacle- free path from the agents' starting location to its' goal location, though multiple paths often exist. The training and validation datasets are also engineered to be balanced. make a detour to reach the goal. In the other 50% cases, no obstacles lie on the shortest path and detour reach to goal will make the navigation planning less efficient. We use 15, 000 environments for the training set, and 2, 250 environments for the testing set, which is disjoint.
50%
cases, obstacles lie on the shortest path the agent has to
5

---

Agent
Camera
Camera
Free space
Map
A
Wall
Goel locatio
Gamara
Camera 4
X
Hidden obstade
X
6
Camera
Incorrect path
Figure 5: Without multiple views, the agent fails to avoid obstacles en route to the goal.
Figure 6: With multi-view observations from many cameras, the agent reaches the goal while avoiding obstacles. The red cube obstacle camera 4 corresponds the red cross in the map.
The dataset is challenging to solve when the agent only has partial observability and imperfect knowledge the world state. Each agent is given a map, which includes public information such as roads and walls well private information such current start location, end goal location, and first-person view. The agent tries to reach the goal with the shortest path. However, as seen in Figure failure cases often occur when agents fail to take into consideration obstacles beyond their first person view observations.
The presence of multi-view observations (which our method encrypts) offers additional information that help agents reason about hidden obstacles in Figure 0 There are four fixed location cameras in Obstacle World to provide multi-view observations. We set camera at each of the four corners, such that each camera able to look over an entire lane. Additionally, there exists fifth camera, which is the agent's first person point of view. At cach time step, view, which changes as the agent takes steps forward. The four cameras offer full observability to the world state, allowing our network to estimate about whether detour is necessary.
image is generated from the agent's 's
4.2 Baselines and Methods
We compare against multiple methods that predict action sequences. These methods have varying privacy guarantees and input information.
Random walk. In the random walk method, the agent has access any additional information in the environment besides the starting location, and has to make random guesses to possibly reach towards the goal state. The agent is able to freely pick action till maximum number of steps without any knowledge on the possible path reaching goal state. To avoid an almost zero success probability, the random walk baseline designed to be clash-free, meaning that the agents will only choose available steps that do not result in a clash to the wall or to the obstacles.
Map only. We train network predict action sequences based exclusively the starting location, the goal location, as well as the input map. This means the agent able to access public information such roads and walls, similar to how are able to access fixed information from google maps. This approach has a strong privacy guarantee because all operations are performed locally by the agent. A multi-layer perceptron is used as map encoder, and the output features are fed into another multi-layer perceptron to classify the action.
First person view. This is strongest achievable baseline without any privacy compromise, where the agent has access to the starting location, the goal location, the map, and its' own first person point of view. No multi-party computation is involved as all the private data belongs to the agent and local computation is possible. There are two difficulty levels for the agent. The deterministic start baseline assumes that the agent is always entering from left to right and has full view of the top lane. Hence, the agent is able to effectively learn that obstacle view means do not enter the top lane. The random start baseline assumes that agent can enter from left to right or top to bottom with 50% probability each. Hence, in the random start baseline, the agent view does not help in taking the first step.
Encrypted camera views (our method). We propose to train the network with the same inputs as plaintext with camera views above, i.e. starting location, the goal location, the agent's own first-person view, as well as the multi-view observations, all via a neural network with multi-party computation. The network architecture seen in Figure 2 Fach piece private information broken down into 2 or 5 shares depending the number of parties involved. One of the parties is

---

Table 1: Plaintext and ciphertext test accuracy. The last experiment plaintext with camera views allows full information access while breaching privacy. All other approaches provide strict privacy guarantees. All start views are random unless otherwise specified as deterministic start.
Experiment
Detour Required
No Detour
Overall
Random
6.2%
53.1 %
29.9%
Map only
64.3%
77.5%
74.0%
First Person
64.5%
79.7%
72.0%
First Person (deterministic starting view)
86.0%
91.6%
88.8%
2-party MPC w/ Camera Views
93.6%
98.8 %
96.6%
5 5-party MPC w/ Camera Views
94.3%
99.2%
96.9%
Plaintext w/ Camera Views
95.2%
99.5% %
97.1 %
always the agent taking an action. In the 2-party setting, all the security cameras are owned by one party, such as the government. This assumes the security cameras trust each other. In the n-party setting, each security camera is owned by different party, so there are many parties. This assumes the security cameras do not trust each other. The CipherNav models are both trained and evaluated in privacy-preserving ways through secret sharing protocols. The experiments assume public network parameters and private input data. Multi-party computation assumes that if at least one party is honest, then the protocol preserves privacy. Hence, any agent who wants to keep its information private will have strong incentive to remain honest, rendering the entire privacy guarantee secure.
Plaintext with camera views. To understand the performance in the idealized case, we train a network similar our method, but without any privacy guarantees. This neural network uses architectures similar to Figure 2 except that everything is trained in plaintext without multi-party computation. As such, either the agent has full knowledge of the security camera images, or the third-party has full access to the agent's 's first person camera view sequence, goal location, and current location. Plaintext training assumes the existence trusted third party with full knowledge of all the private information.
4.3 Quantitative analysis
Table shows that our proposed MPC models achieve significant improvement over other privacy- preserving baselines. Furthermore, our crivacy-presevving model's results are comparable to the accuracy of the non-privacy preserving models (less than 1% difference in performance).
To provide quantitative analysis on the effectiveness of CipherNav, seven experiments are conducted with five plaintext baselines and ciphertext multi-party computation models. With maps indicating the roads and walls, the agents are only able to get 64.3% accuracy in detour required cases and 77.5% accuracy in no detour cases. This is significantly lower than the plaintext results with multi-view camera supervision, where the accuracy is 95.2% in detour required cases and 99.5 99.5% in no detour cases. However, the plaintext with camera views experiment assumes full knowledge of camera images and provides no privacy protection to agents.
Other privacy- serracc-reeeeving baselines also calculated in cases where the agents' first person view is provided. The "first person" model barely improves in accuracy as compared to the map only model, assuming the default random start. In other words, due to the number of occluded obstacles in dataset, a first person view not helpful when a map and agent location is provided.
We additionally compare the "first person" model with a random starting view direction to another model trained with a deterministic starting view direction. We notice a significant improvement in accuracy, and attribute this to the fact that deterministic starting view would allow the model to orient the agent with respect to the map at the very beginning. In other words, the agents always have full information 86% for detour required case and 91.6% for no detour case. However, the overall accuracy still 8.3% below the theoretical best result assuming plaintext multi-view, camera observation.
the top lane to better choose the first step, and hence the accuracy increases to
In our proposed models, 2-party MPC with camera views and 5-party MPC with camera views both achieve signficant improvement with 96.6% and 96.9% overall accuracies respectively. In the 2-party MPC experiment, we find that all three experiments of "Detour Required," "No Detour' and "Overall" experience less than 2% drop in accuracy as compared to the upper-bound plaintext accuracy,
7

---

Poz
Wall
Statloccaton
Coal boatton
Hidden ocsaadi
Correct
recmee puth
Comoro
Currere
Camara
Comera
Curreru
Camera
Camera
A
Carrers
A
X
Cerrene
Currero44
Carreno 3
Cerrenu
3
Cemera
Camera
A
Camea
A
G
Camao
V
Camera
Canea
Camero
Camera
Canera
Camrea 2
Camera
A
A
4
X
Camrra
Camera4
Camera 3
Gamera
Camwra
Camera 3
Figure 7: We visualize several examples of agent navigating The Obstacle World while simultaneously preserving privacy. first row shows success cases, and bottom rows show instructive failures.
while providing full privacy guarantees. Similarily, in the 5-party MPC experiment, find all three experiments have a less than 1% drop in accuracy as compare to the upper-bound plaintext accuracy, while providing full privacy guarantees. This highlights the applicability of multi-party computation in private perception navigation tasks.
4.4 Qualitative Analysis
Qualitative analysis results are shown in Figure to illustrate success and failure cases provided by model outputs. The top row illustrate the two success cases. With multi-view supervision, the agent is able to perceive hidden obstacles and navigate complex detours to reach goal locations, even if the goal location is deep in a dead end. The second row illustrates the cases where the agent fails due to overly long and complicated scenarios. Four or five turns are required to maneuver the obstacles, which proven to be a challenge for both plaintext and ciphertext models. Solving the case requires more complex network architecture.
The examples in the last highlights the possible drawbacks of the ciphertext model after taking in additional information. In the first case, the agent could easily reach the goal by following the shortest path, but instead chooses make unnecessary detour. In the last example, the agent is stuck in an indecisive state, where the observations from camera views to avoid obstacles come into conflict with the understandings from maps to reach the goal. Upon reaching the bottom left corner, the observations from the multi-view cameras indicate the presence of an obstacle, prompting the agent to move one step back towards the right. However, since decisions are made agnostic about previous states, the agent repeats the same mistake and step forward in attempt to reach the goal. While additional private information hugely enhances the potential navigation outcomes, these examples illustrate the needs to account for edge cases and potential information conflict after including additional private information in downstream tasks, even if privacy is no longer a concern in the proposed models.
4.5 Inference Runtime and Result Distribution Analysis
To quantify the inference runtime multi-party computation, Table reboods records the forward pass time on both GPU and CPU on input data with batch size 100 on the action prediction network only. Features pre-computed for both plaintext and ciphertext networks. Multi-party computation is

---

Table 3: Failure counts different models of 4500 trials
Table 2: We show inference time comparison between ciphertext and plaintext models.
Crash Obstacle
Crash
No
Experiment
Wall
Crash
Runtime (scconds) GPU
Map only
1309
0
Experiment
CPU
First Person
1252
0
0
First Person (det. start view)
491
13
0
plaintext
0.00028
0.00032
2- pprtyy MPC w/ Camera Views
144
10
2- 2-party
0.15
0.23
5 5-party MPC w/ Camera Views
115
21
11
5-party 0.42
1.3
Plaintext w/ Camera Views
105
11
more expensive where forward pass in two-party computation has 500x overhead GPU and forward pass in five-party computation has a 1500x overhead on GPU. The large computational overhead highlights the need to perform computation in stages and off-loading computations to non-MPC stages.
We experimented with the success and failure counts for different models over different path lengths. While all four models achieve similar success count in shorter path length < 4, the difference in success count becomes obvious after path length > 5. On path length > 9 where detours are more necessarily to reach the goal, we found plaintext camera views and multi-party computation models perform signficantly better with lower failure rate and higher success rate.
Table 3 breaks down reasons failures into three categories: crash obstacle, crash wall, or no crash. No crash failure case usually refers to the scenario where the agent is stuck in an indecisive state due to conflicting information from external camera views and internal map and first person views. simple map baseline model can effectively learn not to crash into the walls, but has significantly higher rate of crashing into hidden obstacles. The introduction of additional information brings additional challenges but significantly reduce the chance of overall hazards. However, the majority of the limitations remain in plaintext, and the introduction of encryption and mulli-party computation does not seem to result in major decrease in performance.
Conclusion
Navigation is a fundamental challenge in computer vision and robotics, and the task is difficult for partially observerable environments that have many occlusions and visual obstructions. We have demonstrated a framework that integrates secure multiparty computation with multiview vision in order to create private representation for navigation. Due the number of applications for navigation and the importance privacy, we believe integrating the two areas is a promising direction to deliver robust yet secure navigation policies for robotics.
Limitations and Future Work. This paper only the first step, and there remains many important steps in order to realize the tangible applications of this method. Our method is designed for both fixed cameras positions and static environments, and future work will need to expand the architecture in order to relax this assumption. Additionally, an important limitation towards real-world deployments the wide area network latency and bandwidth, which must sufficiently fast to transfer shares between parties. Overcoming this challenge will require interdisciplinary advances at the intersection of computer networking, cryptography, and machine lcarning. Finally, multi-party computation introduces additional overhead to inference, motivating the need for new generation of neural network architectures that are designed to efficient under privacy guarantees.
Societal Impact. Our research is founded on ethical considerations, and we are excited for the potential for navigation push the frontiers in robotics and assistive technology, such as navigation for people with disabilities. Due to the ubiquity of cameras today, we believe visual representations that tightly integrate with cryptography and
techniques will be critical for delivering the future applications of computer vision, and we hope this paper spurs additional work along this direction. correctly implemented, multiparty computation provides rigorous guarantees that party cannot learn anything more from the computation than the task itself.

---

References
[1] Amos Beimel and Benny Chor. Interaction in key distribution schemes (extended abstract), In Advances in Cryptology CRYPTO '93, 13th Annual International Cryptology Conference, Santa Barbara, California. USA, August 22-26, 1993, Proceedings. volume 773 of Lecture Notes in Computer Science, pages 444 455. Springer, 1993.
[2] Amos Beimel and Benny Chor. Universally ideal secret-sharing schemes. IEEE Transactions Informa- tion Theury, 4063)7786-794 1994.
[3] Peter Bogetoft, Ivan Damgård, Thomas Jakobsen, Kurt Nielsen, Jakob Pagter, and Tomas Toft. A practical implementation of secure auctions based on multiparty integer computation. In International Conference on Financial Cryptography and Data Security, pages 142-147. Springer, 2006.
[4] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. learrnng to explore using active neural slam. arXiv preprint 05X5v:2004.05155, 2020.
[5] Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33:4247-42588, 2020.
[6] Harsh Chaudhari, Rahul Rachuri, and Ajith Suresh. Trident: Efficient 4pc framework for privacy preserving machine learning. arXiv preprint arXiv:1912.026:: 2019.
[7] Maxime Chevalier Boisvert. Miniworld: Minimalistic 3d environment for rl and robotics research.
https: https://github.. wttp://wwwww..////oteeoo gywh/yiin-mmmmiidddd 2018.
[8] David Evans, Vladimir Kolesnikov, Mike Rosulek, et al. A pragmatic introduction to secure multi-pane computation. Foundations and Trends@ in Privacy and Security, 2(2-3):70-244 2018.
19] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2616-2625, 2017.
[10] Alston Scott Householder. The numerical treatment of single nonlinear equation. McGraw Hill, 1970.
[11] Toannis Ioannidis and Ananth Grama. An efficient protocol for yao's millionaires' problem. In 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the, pages 6-pp. IEEE, 2003.
[12] Karthik A Jagadeesh, David Wu, Johannes A Birgmeier, Dan Boneh, and Gill Bejerano. Deriving genomic diagnoses without revealing patient genomes. Science, (57(6352)6692 695, 2017.
[13] Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexander Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Sim2real predictivity: Does evaluation in simulation predict real-world performance? IEEE Robotics and Automation Letters, 5(4):6670- 6677, 2020.
[14] Leslie Pack Kaelbling, Michael I. I Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134, 1998.
[15] Peter Karkus, David Hsu, and Wee Sun Lee. Qmdp-net: Deep learning for planning under partial observability, Advances in neural information processing systems, 30, 2017.
[16] Brian Knott, Shobha Venkataraman, Awni Hamnun, Shubho Sengupta, Mark Ibrahim, and Laurens van der Maaten. Crypten: Secure multi-party computation meets machine learning, Advances in Neural Information Processing Systems. 34:4961 19711 2021.
[17] Nishant Kumar, Mayank Rathee, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma. Cryptflow: Secure tensorflow inference. In 2020 IEEE Sympesium Security and Privacy (SP), pages 336-353. IEEE, 2020.
[18] Hsiao-Ying Lin and Wen-Guey Tzeng. An efficient solution to the millionaires' problem based on homomorphic eneryption. In International Conference Applied Cryptography and Network Security, pages 456-466. Springer, 2005.
[19] Silvio Micali, Oded Goldreich, and Avi Wigderson. How to play any mental game. In Proceedings of the Nineteenth ACM Symp. on Theory of Computing, STOC, pages 218 229. ACM, 1987.
[20] Piotr Mirowski, Matt Grimes, Mateusz. Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Audrew Zisserman, Raia Hadsell, et al. Learning to navigate in cities without map. Advances in Neural Information Processing Systems, 31, 2018
[21] Payman Mohassel and Yupeng Zhang. Secureml: A system for scalable privacy- preserving machine learning. In 2017 IEEE symposium on security and privacy (SP), pages 19-38. IEÉE, 2017.
[22] Petter Nilsson, Sofie Haesaert, Rohan Thakker, Kyohei Otsu, Cristian- Vasile, Ali-Akbar Agha- Mohammadi, Richard M Murray, and Aaron D Ames. Toward specification -uided active mars exploration for cooperative robot teams. 2018
10

---

[23] Sylvie CW Ong, Shao Wei Png. David Hsu, and Wee Sun Lee. Planning under uncertainty for robotic tasks with mixed observability. The International Journal of Robotics Research, 29(8):1053-1068 2010.
[24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chaman, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-peeformance deep learning library. Advances in neural information processing systems, 32, 2019.
[25] Theo Ryffel, Andrew Trask, Morten Dahl, Bobby Wagner. Jason Mancuso. Daniel Rueckert, and Jonathan
A generic framework for privacy preserving deep learning. arXiv preprint
arXiv: 1. 7,0017, 2018.
[26] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339-9347, 2019.
[27] Adi Shamir. How to share a secret. Communications of the ACM, 22 (:611)6611111, 1979.
[28] Reid Simmons and Sven Koenig. Probabilistic robot navigation in partially observable environments. 1995.
[29] Ke Sun, Brent Schlotfeldt, George J Pappas, and Vijay Kumar. Stochastic motion planning under partial observability for mobile robots with continuous range measurements. IEEE Transuctions Robotics, 37(3):979 995, 2020.
[30] Sijun Tan, Brian Knott, Yuan Tian, and David J Wu. Cryptgpu: Fast privacy- priveri-n machine learning on the gpu. In 2021 IEEE Symposium on Security and Privacy (SP), pages 1021-1038. IEEE, 2021.
[31] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large seene neural view synthesis. arXiv 2022.
preprint Xiv: X202:220..0222
[32] Basile Van Hoorick. Purva Tendulkar, Didac Suris, Dennis Park, Simon Stent, and Carl Vondrick. Revealing occlusions with 4d neural fields. arXiv preprint arXiv:2204. 109/6, 2022.
[33] Sameer Wagh, Divya Gupta, and Nishanth Chandran. Securenn: 3-party secure computation for neural network training. Proc. Priv. Enhancing Technol., 2019(3):26-49 2019.
[34] Sameer Wagh, Shruti Tople, Fabrice Benhamouda, Eyal Kushilevitz, Prateek Mittal, and Tal Rabin. Falcon: Honest-majority maliciously secure framework for private deep learning, arXiv preprint xrXiv:2004.022 2020.
[35] Xiao Shaun Wang, Yan Huang, Yongan Zhao, Haixu Tang, XiaoFeng Wang, and Diyue Bu Efficient genome-wide, privacy-preserving similar patient query based private edit distance. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pages 492-503, 2015.
[36] David Satkins-Valls, Jingxi Xu, Nicholas Waytowich, and Peter Allen. Learning your way without map or compass: Panoramic target driven visual navigation. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5816 5823. IEEE, 2020.
[37] Erik Wijmans, Abhishek Kadian, Ari Morcos. Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv: 1e1e111110000 1.0057,, 2019.
[38] Andrew C Yao. Protocols for secure computations. In 23rd annual symposium on foundations of computer science (sfes 1982), pages 160 164. IEEE, 1982.
[39] Andrew Chi-Chih Yao. How generate and exchange secrets. In 27th Annual Symposium on Foundations of Computer Science (sfes 1986), pages 162-167. IEFE, 1986.


Reviews:
[{"review_id": "mdM_fKYThih", "reviewer": "Reviewer_BTzU", "summary": "This paper proposes a method for robot navigation from multiple third-person views that additionally attempts to preserve the privacy of the information captured by the cameras. This is accomplished by training the networks used in the model to operate on encrypted feature vectors, thus eliminating the need to share the original data. This allows for navigation without any party having access to all the information. The method is tested on a Gym Mini World map. The results show that the method has comparable accuracy to a non-private baseline. ", "questions": "- How does the method ensure that each encrypted feature vector can't be used to reconstruct anything about the scene?\n- How is the conversion from (assumingly) continuous feature vectors to integers done?\n- Does the system need to share keys between the parties?\n- Why during training (Eq 6) is finite field arithmetic not used?", "limitations": "The paper addresses the societal impacts adequately. However, I would probably suggest including some information on the limitations i.e. perhaps some information about the images could be reconstructed by repeated querying the navigation network? Could the system be attacked by a third party?", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n=========\n- Preserving privacy in robotics and computer vision applications is an area of growing importance and thus methods that can help achieve this goal are much needed. This is partially because images usually capture so much additional information that is not particularly needed for a task, and thus sharing them with a third party can have major privacy concerns. This is common in robot vacuum cleaners, for example, where computation is usually limited to the edge in order to preserve privacy. The proposed method is a good solution to this if the processing has to be done remotely.\n\n- The method seems quite novel. While preserving privacy in neural networks is an active area of research, as far as I’m aware there aren’t any learning-based privacy-preserving navigation approaches. \n\n- The method is simple but effective. The method works with standard embeddings computed by a neural network that are then encrypted using modular arithmetic. The network then learns to perform the action using this arithmetic decomposition of the representations. This could be easily applied to any existing navigation solutions that make use of feature representations.\n\n- The results show that the encrypted navigation pipeline is only a fraction of a percent less accurate than the corresponding plaintext pipeline which is quite surprising given that the networks operate on encrypted feature vectors. \n\n\nWeaknesses\n==========\n- If the action prediction network can successfully predict actions using the encrypted features, it might be possible to train a similar network to reconstruct the either the multi-view images or the images from the agent. The paper states “Each $\\left[E_{\\theta}\\left(c_{i}\\right)\\right]_{p}$ is constructed such that it is insufficient to reconstruct anything about the scene” but it’s not clear why this is the case.\n\n- The evaluation is only performed on the Gym Mini World map which is not very realistic. How would the method work in a real-world environment where there is a lot more variation between scenes and the map and cameras exhibit noise, occlusion and artifacts. Would the performance between the plaintext and encrypted methods still be similar or would the proposed approach be affected more by this variation and adverse conditions? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hirEJVD5QnB", "reviewer": "Reviewer_N4st", "summary": "The paper presents CipherNav, a novel framework for private multiparty perception for navigation. CipherNav aims to predict the sequence of actions needed for an agent to reach a goal destination. The inputs to CipherNav are (a) camera views from P participants, (b) the agent's ego-centric camera view, and (c) a map of the environment containing public information about the layout of roads. In order to preserve privacy between the participants, inputs (a), (b), and (c) are encrypted via arithmetic decomposition, such that each party receives a partial share of the secrets. The predictions predictions from each participant are then arithmetically combined to produce the final action sequence. In order to validate CipherNav,  a novel dataset based on the open-source Gym MiniWorld environment is generated with 5x5 maps. Obstacles are randomly inserted into the maps to impede the agent from reaching the goal state. Experiments on the generated dataset show that the CipherNav achieves comparable performance to a model operates on plaintext images.", "questions": "1. What are the technical contributions beyond applying SMPC to a new problem?", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* Novel setting of private mutiparty perception for navigation is introduced. Given the increasing importance of privacy-preserving ML, this is an important setting to consider. \n* Novel dataset is introduced. The proposed Obstacle World Dataset, while simple, can serve as a good baseline for the task of private multiparty perception for navigation. \n* Results on the novel Obstacle World Dataset show that the proposed approach achieves comparable performance to a model that operates on plaintext images and does so with a reasonable computational overhead.\n* The proposed CipherNav network architecture is novel and seems to work well for the proposed task\n\n\n**Weaknesses**\n* Limited technical contributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PpbexpJ3QIm", "reviewer": "Reviewer_t8m7", "summary": "- The paper proposes a framework for multiparty perception: on leverage multiparty computing techniques in a fully observable environments (with multiple cameras), such that the agent learns to take actions (e.g., driving direction) without observing the raw images from the multiple cameras.\n- The core idea is that only encrypted ciphertexts extracted from the cameras are publicly exchanged between the parties, which are used to determine the actions.\n- The approach is evaluated on an \"obstacle world\" dataset which will be released publicly. The goal of the agent is to reach a goal location while avoiding obstacles.\n- Evaluation indicates that the performance of the proposed model reaches accuracies very close to the plaintext model (96.9% vs 97.1%).", "questions": "1. See concern 1: I would appreciate some clarification on agents vs. parties (since I think in MPC literature, generally agents = parties?)", "limitations": "Yes, the paper has a good discussion on limitations and societal impact.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The problem tackled by the paper is well-motivated: while fully observable environments are highly useful for navigation, they incur a privacy cost. The approach suggests MPC can help achieving great performances with rigorous privacy guarantees.\n2. The authors plan to publicly release the dataset which will be beneficial to the community.\n3. The evaluation is rigorous and is accompanied with appropriate baselines and discussions.\n4. The writing of the paper for the most part is clear and precise.\n\n### Concerns\n\n**1. Agent vs. \"Party\" vs. Camera views**\n- The concern is to do with difference between agents and parties - are they the same? As in, is \"two-party\" case as shown in Figure 2 involves two agents? In which case, it appears that that first person view observed by each party would change making it unclear on how actions generalize between different agents with different goals.\n- Alternatively, does one agent taking an action entail receiving encrypted ciphertexts from all other agents (each with their own first person view and computations)?\n\n**2. (Nitpick) Some conclusions in Qualitative Analysis**\n- I appreciate the authors presenting and discussing the success/failure examples of the approach.\n- However, some conclusions drawn from the failure modes are not evident. For instance, take the conclusion L269 \"the agent incorrectly reasons the goal location is behind the obstacle location\" isn't entirely evident in Fig. 7 (middle row, left).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept