Seeing through Uncertainty: Robust Task-Oriented Optimization in Visual Navigation
Authors: Yiyuan Pan
Yunzhe XU
Zhe Liu
Hesheng Wang
Venue: NeurIPS 2025 poster ()
Keywords: Visual Navigation
Deep Reinforcement Learning
Task-based Model
Uncertainty Calibration
Primary area: reinforcement_learning

Abstract: Visual navigation is a fundamental problem in embodied AI, yet practical deployments demand long-horizon planning capabilities to address multi-objective tasks. A major bottleneck is data scarcity: policies learned from limited data often overfit and fail to generalize OOD. Existing neural network-based agents typically increase architectural complexity that paradoxically become counterproductive in the small-sample regime. This paper introduce NeuRO, a integrated learning-to-optimize framework that tightly couples perception networks with downstream task-level robust optimization. Specifically, NeuRO addresses core difficulties in this integration: (i) it transforms noisy visual predictions under data scarcity into convex uncertainty sets using Partially Input Convex Neural Networks (PICNNs) with conformal calibration, which directly parameterize the optimization constraints; and (ii) it reformulates planning under partial observability as a robust optimization problem, enabling uncertainty-aware policies that transfer across environments. Extensive experiments on both unordered and sequential multi-object navigation tasks demonstrate that NeuRO establishes SoTA performance, particularly in generalization to unseen environments. Our work thus presents a significant advancement for developing robust, generalizable autonomous agents.

Seeing through Uncertainty: Robust Task-Oriented Optimization in Visual Navigation
Yiyuan Pan Yunzhe Xu Zhe Liu" Hesheng Wang"
Shanghai Jiao Tong University
(pyy030406, xy29911, liuzhesjtu, vanghesheng) edu.
Abstract
Visual navigation is fundamental problem in embodied AI, yet practical deploy- ments demand long-horizon planning capabilities to address multi-objective tasks. A major bottleneck is data scarcity: policies learned from limited data often overfit and fail to generalize OOD, Existing neural network-based agents typically increase architectural complexity that paradoxically become counterproductive in the small- sample regime. This paper introduce NEURO, integrated Carning-to-optiming framework that tightly couples perception networks with downstream task-level robust optimization. Specifically, NEURO addresses core difficulties in this inte- gration: (i) it transforms noisy visual predictions under data scarcity into convex uncertainty sets using Partially Input Convex Neural Networks (PICNNs) with conformal calibration, which directly parameterize the optimization constraints; and (ii) it reformulates planning under partial observability as robust optimization problem, enabling uncertainty-awant policies that transfer across environments. Ex- tensive experiments on both unordered and sequential multi-object navigation tasks demonstrate that NEURO establishes SoTA performance, particularly general- ization to unseen environments. Our work thus presents significant advancement for developing robust, generalizable autonomous agents. Code: https: //gg/tttt.u//t//////// com/ http//yyWllll/een
Introduction
Visual navigation has emerged as a comeertoon problem in robotics, where agents must reason complex 3D environments and pursue possible multiple goals under uncertainty. Among benchmark tasks, Multi-Object Navigation (MultiON) task [23] evaluates an agent's 's ability to locate multiple objects within 3D environments, requiring sophisticated scheduling strategies across multiple goals. This task presents significant challenges due to its multi-goal nature and severe data scarcity, which often causes agents overfit to training environments and generalize poorly unseen scenarios. While existing neural network (NN)-based approaches attempt to enhance performance by stacking complex network modules, these methods tend exacerbate overfitting rather than improve generalization in low-data regimes such as search-rescue missions [19, 24].
A potential solution is instead to couple learning with explicit optimization models to form task-aware training frameworks [9]: First, optimization models can explicitly articulate hard task constraints (c.g., collision avoidance in navigation) without additional parameters to capture shared dynamics across related tasks, thus enhancing generalization; Secondly, their inherent structure facilitates the simultaneous consideration and scheduling of multiple objectives. Therefore, it's natural idea
"Corresponding author. The authors are with the School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, and the Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240. Zhe Liu is also with the National Key Laboratory of Human-Maachine Hybrid Augmented Intelligence, Institute Artificial Intelligence and Robotics, Xi'an Jiaotong University.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Task-based Framework
Visual Navigation
Learning
Accurate
Scancity
Model
oo
Value Estination
Ensccuratc
deal
Brodknn
Rabust Sampling
Predicted
Perematirs
Optimization Model
Explicit Task Dynomics
Partially Csservab
Policy
Maddingg
Ooal [anfound]]
Robust
Optimization
Figure 1: Overview of the work. (Left) The foundational concept of task-based optimization framework and its inherent challenges when directly applied to visual navigation. (Right) NEURO framework addresses this mismatch through conformal visual-processing method and robust optimization formulation for decision-making.
to wed networks with optimization in visual navigation. However, two non trivial challenges are raised (see Fig. 1): First, the reliability of optimization outputs depends critically the accuracy of parameters predicted by the network and the data scarcity can lead to prediction errors, destabilizing the entire pipeline; Secondly, formulating which is unattainable in partially observable environments typical of embodied navigation. In essence, naïve integration might instead degrade system stability and overall performance.
optimization problem often requires global information,
To this end, we propose NEURO, a robust framework for training visual navigation agents end-to-end with downstream optimization tasks. Specifically, NEURO incorporates two key technical innova- tions: (i) For the unreliable network predictions, we employ Partially Input Convex Neural Networks (PICNNs) to distill complex, non-convex visual information into high-ddmeensioal convex embed- dings. These embeddings are then transformed by a proposed calibration method into a tractable, convex uncertainty set, which is subsequently passed to the downstream optimization problem; (ii) For the issue of partial observability, we formulate the visual navigation (formally modeled as a POMDP) pursuit-evasion game, casting it as robust optimization (RO) problem. Such formula- tion inherently manages parametric uncertainty arising from partial observations and demonstrates generality across diverse navigation tasks. We evaluate NEURO our proposed unordered MultiON and traditional sequential MultiON tasks, demonstrating superior performance and generalization over state-of-the-art (SoTA) network-based approaches. In summary, our contributions are threefold:
We propose NEURO, novel hybrid framework that synergistically integrates deep neural networks with downstream optimization tasks for end-to-end training, significantly improved generalization in data-scarce regimes.
We introduce a methodology to bridge unreliable navigation prediction and safe trajectory optimization. By leveraging PICNN-based conformal calibration and casting POMDP planning as robust optimization, NEURO effectively handles partial observability and parametric uncertainty.
Extensive empirical validation demonstrating that NEURO establishes superior performance on challenging MultiON benchmarks, significantly outperforming existing methods, particu- larly in generalization unseen environments.
2 Related Works
Visual Navigation. Visual navigation [6, 13, 20] are cornerstone research areas in embodied AI. A particularly challenging sub-domain is Multi-Targe Object Navigation (MultiON) [23], which demands sophisticated, long-horizon socision-makiinn for locating multiple objects. Prevailing ap- proaches in MultiON and broader visual navigation often augment agents via enhanced memory [15, 10] or predictive world models [21, 17]. While aiming for comprehensive environmental un- derstanding [10, 17, 21, 15], this pursuit frequently results in data-hungry models and excessive training overhead, without directly optimizing the crucial decision-maker making process itself [5, 8). Such box" decision mechanisms, often reliant on complex neural networks navigating the

---

Observation Speca
Componont
Network Colibration
CNN
Linerr
Trajectory
FEOEN
dRU
Prad G
Groand-Tre
Gosl Object
G-11
Actin Eneader
Doggt
Book
Optimizahoon Medel
Eatimated Uncertuinty
Vep
Scliion
Pat X ((tt),
Callbrated Uncertainty
s... (2)(())
(o) Boseline
(b) NeuRO
Figure 2: Architecture of NEURO. (Lefft) The decision-making process of a purely network-based agent at each navigation step. (Right) Our introduced optimization module, which redirects the agent's is training back toward the task itself.
apploratio-cxxploitation trade-off [22. 16], can suffer from information loss and high training com- plexity, especially under data scarcity, hindering robust and interpretable actions. Recognizing these limitations, particularly for demanding tasks like MultiON, our work departs from purely representa- tional learning and advances task-based training paradigm [1]. We advocate for integrating explicit optimization models within the decision pipeline, thereby refocusing learning towards task-specific optimal behaviors and improving decision efficiency.
Task-Based Optimization. The concept "task-based" end-to-end model learning was introduced by [9]. which proposed training machine learning models in way that captures downstream stochastic optimization task. Later, the upplicability of this framework was extended to various types of convex optimization problems via the implicit function theory [1, 2]. Subsequently, more sophisticated learning models were investigated in the context of the task-based model to accommodate more complex task needs [12, 26, 14, 25]. Nevertheless, prior studies have primarily focused short- horizon tasks such as portfolio management or inventory control, typically assuming access to complete and well-structured datasets. In contrast, embodied robotic tasks often operate in data- scarce settings and require long-horizon planning under uncertainty. Building on this gap, we propose NEURO, the first task-based learning framework explicitly designed for embodied agents' tasks.
3 Method
Problem Setup. We formalize MultiON for both unordered (U-MON) and sequential (S-MON) object discovery. In each episode, agent navigates 3D environment to locate a sequence of m target objects selected from a candidate set G of size k. At each time step I, the agent re- ceives an egocentric observation comprising RGB-D images o and a k-dimensional binary vector G. For U-MON, G, indicates the (n m.) remaining targets with entries set to for S-MON, G. identifies the single current target (i.e., n 1). The agent selects action a, from the set (FORWARD, TURN-LAFT, TURN-RIGHT, FOUND) based on ot, G., and its internal state. Upon exe- cuting a FOUND action, if a valid target is within predefined proximity, G, is updated. An episode succeeds if all m targets are found; it fails if the maximum step limit is reached or an incorrect FOUND action performed.
Method Overview. Fig. 2 illustrates the operational pipeline of NEURO. At its core, this work addresses the fundamental challenge of bridging neural networks with optimization models. Building upon the foundational network components (Section 3.1) and the optimization formulation (Sec- tion 3.2), then elucidates how NEURO tackles two pivotal questions: (i) How does NEURO transform network predictions into reliable inputs for optimization? (Section 3.3) (ii) How does NEURO exploit optimization feedback to improve network training? (Section 3.4)
3.1 Neural Perception Module
At each step I, the agent processes its egocentric RGB image et, depth image de from observation ot (Ct, de), and maintains egocentric map view my which is partially observed, agent-centric
3

---

perspective (rotated and cropped) of an inaccessible underlying global map M. A CNN block extracts visual features from or. which are then linearly embedded into a vector v;- Similarly, the egocentric map mh is embedded into t'm These two embeddings are concatenated with a goal object embedding Ng (from G₂) and an action embedding Pa- The resulting tensor 04- = concat( Cm; Vgg va) is fed into a GRU to compute the hidden state and a state feature ft = {ff 1= corresponding to the n pertinent objects.
Next, from the processed features f.. a policy module parameterized by derives a preliminary action policy π ((af"),ffa (fa)) and approximate value function (s+)). During training. the agents will receive an environmental reward signal r according to its chosen action at. which encourages goal-reaching efficiency while penalizing delays. For traditional purely network-based agents (our baseline), the final actions a are equivalent to these network-generanl the efficacy of such an approach is critically contingent upon the fidelity of ft, which often necessitates a comprehensive world understanding. Thus, such agents can impose a significant training burden and increase susceptibility overfitting under data-scarce conditions.
actions, Or = at However,
3.2 Robust Optimization Planner
We model the MultiON
task as a robust pursuit-evasion problem to handle the inherent partial observability and uncertainty in target locations. In this formulation, the agent and n remaining target objects are situated on discrete grid H with E edges and V E x E cells, which serves as an abstraction to simulate the agent's limited local field of view. Due the partial observability, the agent doesn't 't know the exact objects positions but instead maintains a belief about their potential movement, represented by a set of transition matrices ( M= E Ω (we omit episode's timestep I). M denotes the agent's 's estimated probability that object transitions from cell u to cell in one time step. As navigation unfolds, the agent belief evolves by refining likely objects locations. These estimated matrices are the source of uncertainty that our robust optimization will address.
We first define the agent positions Pt as a continuous variable for gradient-based optimization, while the objects' locations are mapped to these discrete cells. Then, the legality of the agent's 's path is ensured with the following constraints, where d the maximum travel distance.
||pe+1 pell2 S d, po = (E/2 E/2/ (p)).. (p))).())) S E, E {0}
AA E (O............... ( 1}
(1a)
(1b)
(1c)
Next, we introduce two key variables to track the evolving understanding of target locations: (i) Prior Belief State 88: A (V + 1)-dimensional vector representing the agent's confidence regarding object i's presence in each of the V grid cells and its capture status. Specifically, (3)),, for e E (I,..., V} quantifies the belief that object is in cell v, and value of implies the agent believes the cell is empty of object i; the 0-th entry, (3) Jo, represents a global capture belief, where a value approaching 1 signifies high confidence in successful capture, see Eq. (2c). By definition, the total belief sums to one, as successful capture implies the climination of uncertainty across all grid cells for that object. The initial Prior Belief State, p8. typically reflects no prior information, assigning a uniform belief of all cells except the agent's starting cell, see Eq. (2d). (ii) Posterior Belief State a²: A V d-dimensional vector serving as an intermediate representation. It derived by applying the agent's objects transition matrix M to the non-captured components of the Prior Belief State 3 as shown in Eq. (2b). at represents the belief distribution over grid cells after accounting for potential object "movement" but before incorporating new observations from time t.
af 10, 1], Vt T
(2a)
(at). Σ WEV (ir ),, VI E T\ VV
(2b)
(8))- = E
EVgy (8) WE (0), v V
(2c)
(o9), 0, -
if oor= otherwise
| +1
(2d)
The capture determination employs same-cell binary detection model: if the agent occupies cell v at time f, it can ascertain object i's presence in cell and eliminate local uncertainty. Consequently,

---

the updated Prior Belief State (g)) ) for v E V can be expressed as the element product of the Posterior Belief State of and an Detection Uncertainty Factor d(pt).. v. as shown in Eq. (3a). The detection uncertainty ). see Eq. (3c), is function of the agent's s current position and the cell index v. Intuitively, for cells distant from the agent (d(pt)r: - 1), the posterior belief (af), equals to the prior belief (.)).. u. as no new reliable information is obtained: for cells at or very near the agent's
position (d(pt). ) + 0), uncertainty is significantly reduced as ( approaches 0.
(g (d)) AA TETT {0), VEV
(3a)
d( d(pt). |0- (pt)al + I1/y (pt)yl E
(3b)
V-1
Ur (u mod E) 0.5, By = |b/ + 0.5
(3c)
Finally, we present the objective function of the optimization problem. For the U-MONtask, the objective to find agent path 10...--Pp that maximizes the accumulated discounted capture belief, robust to the uncertainty in M The formulation for S-MONIS provided in the Appendix A.
[U-MON] min max
Σn EET
(8)) EE
Pt E1
s.t.
2
(4)
3.3 Learning-Planning Interface: Prediction Calibration
This section details how NEURO transforms raw network outputs x/ concat(fi: sx) for each object into a well-defined objects transition matrix Mc under uncertainty. We omit the object index and time index / or conciseness. We first give universal formulation of uncertainty mapping with a general nonconformity score function y(r, M):
2(x) = (M E ERY g(r, M) <
(S)
The selection of the nonconformity score function g(z, M) is critical. To ensure that the uncertainty set (2x)) is convex in M-a property pivotal for tractable downstream optimization we employ PICNNs [3] instantiate g(r,, M) (details in Appendix B), yielding g(a,, M) convex in M for any fixed x. Furthermore, PICNNs can approximate complex, non-convex visual landscapes by learning collection of conditional convex functions, offering structured and expressive uncertainty representation from rich visual inputs.
With g(x, M) established through PICNN, defining the final uncertainty set Q(x) then hinges the principled selection of the threshold q, as balances the risk an overly optimistic set against an uninformatively large one. !(z) provides statistically sound basis for robust decision decision-making making, desire to reliably captures the underlying true object transition matrix with a prespecified coverage level a, formalized by the Definition below.
Definition (Marginal Coverage) An uncertainty set generator 52(-) provides marginal coverage at the (1 -ee-eeel for unknown data distribution P if (f.( E {(x))
Given the definition, the relationship between the threshold q and the desired coverage level a can be established by the following proposition (proof in Appendix C). Accordingly, we compute by sorting the scores {g(x), .M)) (4)))>=-1 obtained from an i.i.d. calibration set sampled at time t for object i, in ascending order and selecting the score at the [NNatti + 1)(1 a)] position.
Proposition 1 (Coverage with Quantile) Let the dataset D = {(I,, M,) to be sampled i.i.d from the implicit distribution P gained during training phase. And q is set be the (1 au--uinnlle for the set (g(zn, My then (2(x) gains the following guarantee.
- P..M.D E 2(()) < a ++1
Although the uncertainty set l(x)) is now well-defined, the original RO problem in Eq. (4) remains intractable directly due its inherent min-max structure. To overcome the issue, we reformulate it as solvable, linear counterparts by taking the dual of the inner maximization problem and invoking
5

---

strong duality (see detailed derivation and S-MON''s formulation in Appendix D):
[U-MON"]
min P.R.
Σ
-c - X =b
VEN.EET,VEV
8.1.
(2) (4)
D.V, 20, ViEn
D E.e
T. $ so,iin tET 10)
(6)
E -
WENNEV
(T+)T (vf) X G. 20,
I - Evev +.. (k;) =
G+ = 0,
if otherwise
-nEv T (k:)
CEFE where C,S,T I are constant value matrices and vectors. Th RE V/YEYY EYE 12Ld+1 k RV+Ld are decision variables. D,E gVXv is the diagonal matrix of vector d(p:). be R2Ld+1, Ae ((2Ld-1)x(V+Ld) are constructed from the PICNN's inherent parameters.
3.4 Planning-Learin Interface: Solution Feedback
To enable end-to-end training of our hybrid NEURO agent, the output of the downstream opti- mization problem is integrated back into the neural network module in two primary ways. First, refined action signal is derived. The optimal trajectory p* {pi T from the optimization solution provides a goal-directed action offset, denoted a task This offset is then combined with the network's s original action proposal of Secondly, the optimal objective value of the opti- mization problem serves as task-specific reward signal, r r!ask, which used to augment the reward rtask with respect
agent's intrinsic reward rt The gradient of this optimization-derivee to the network parameters can be computed via the chain rule, leveraging implicit differenti- ation through the
Kuhn-Tucker (KKT) conditions. These integrations are formalized as:
Ur A A a.eet + (1 (1-A)
A) ataak (7a)
Algorithm NeuRO Training Algorithm
rre a GVM(rē"M ,,task)
1: initialize episode.
2: for in mar max_steeeee steppsppp do
task 100 = task 10p" -0p" Dpp"/0n
100
(7b)
3: receive o, and G, from env (st, ft)
where is a blending factor for actions and GVM denotes Goal Vector Method, parameter-
4: output: 92(-) // unccrtaaitty calibration
5: solve [U-MON]" or [S-MON]*.
6: apply offset a and task
free approach that combines multiple reward sig-
nals to guide the policy towards Pareto-optimal solutions, mitigating issues of conflicting gradi-
7: receive S and reward r
8: update 0 with Vor:
directions from naïvely summed rewards.
9:
if G. = then
break
Furthermore, we provide an additional optimization-bassee theoretical guarantee regard- ing optimality and convergence in Appendix E to ensure the stability of this learning process.
11: end if
end for
Experiments
Tasks and Evaluation Metrics. We test on unseen settings with object counts m 2, across standard train, validation and test splits from U-MON and S-MON tasks. We adopt four commonly used evaluation metries from visual navigation studies. 1) Success: A binary indicator of episode success; 2) Progress: The fraction of object goals successfully FOUnD; 3) SPL (success weighted by path length): Total traveled distance weighted by the sum of the geodesic shortest path from the agent's starting point to all the goal positions; 4) PPL: A variant of SPL that weights based on progress.

---

Table I: Comparison with existing methods U-M0N and S S-MON tasks.
S-MON Test
S- S-MON Test
Methods
Success Progresst
SPLÍ
PPLÍ Success" Progresst SPLÍ PPL
SMT[II]
28
44
26
36
9
22
7
18
FRMQN[18]
29
42
24
33
13
29
11
24
OracleMap (Oce)
34
47
25
35
16
36
12
27
ProjNeuralMMap(33
45
57
30
39
27
46
18
31
ObjRecogMMp((33 [23]
51
62
38
45
22
40
17
30
OracleEgoMMp
64
71
49
54
40
54
25
36
OracleMap
74
79
59
63
48
62
38
49
Lyon[ 15]
76
84
62
70
57
70
36
45
HTP-GCN(17]
76
84
60
67
57
70
27
33
NEURO (Ours)
80
86
66
72
62
72
40
47
U-MONm=2 Test
U-MON Test
Methods
Successt Progresst SPL1 PPL1 Success" Progressf SPL PPL
OracleFgoMMp
54
62
38
44
36
42
33
38
NEURO (Ours)
62
52
56
45
53
45
50
4.1 Comparison with SoTA
Table I compares NEURO's performance against other agents on the S-MONtask (m 2). using Or- acleEgoMap (visualized in Fig. 2) our network-based baseline. Our NEURO framework empowers the agent to achieve superior performance on the S-MONbenchmark. Notably, NEURO demonstrates substantial 4% improvement SPL (without data augmentation), indicating enhanced navigation efficiency. This efficiency gain is attributed to the agent's 's ability to leverage the downstream opti- mization problem for more effective global path planning. Meanwhile, the performance gain is more pronounced at m-3 compared to m=2, which we attribute to the optimization model's 's ability to coordinate multiple targets. Furthermore,
NEURO architecture exhibits remarkable adaptability stemming from the customizable nature of its optimization component. By solely modifying the downstream optimization model-withoue requiring retraining networks agent can efficiently adapt distinct task formulations (i.e., S-MON and U-MON).
4.2 Quantitative and Qualitative Study
Quantitative Study. As illustrated in Fig. 3, the NEURO training framework facilitates sig- nificantly accelerated convergence with respect to the Success rate on specific tasks, ultimately surpassing the performance benchmarks set by purely network-based methodologies. This rapid learning curve strongly suggests an enhanced capability of NEURO agents to achieve superior task outcomes, particularly under conditions of data scarcity. Consequently, the NEURO architecture demonstrates heightened sample efficiency and a diminished requirement for extensive training datasets, underscoring its adaptability and practical utility in
settings.
Qualitative Study. Our NEURO agent demonstrates the capability to generate straightfor- ward yet effective internal representations that aid its decision-making process in visual naviga- tion, as illustrated
Fig. Specifically, we examine the object transition matrix M for an object i, which is derived from the solution of the downstream optimization model. The vi- sualization reveals distinct behaviors of Mi: for objects currently outside the agent's field of view, Mt tends reflect diffuse less certain belief state over possible locations. Con- versely, when an object clearly observed, the corresponding M, sharply localizes, accu- rately reflecting the agent's high confidence in the object's position on the internal grid H.
7

---

SMN MONOO (E:m))
SMON (m=1)
SMON (m=2)
Succes
34
53
Training Updates (le4)
NeuRO
NeuRO (tosk-unspecific)
Lyon
OracleEgoMapp
Figure 3: Learning curves for NEURO and baseline during training for different tasks.
Ground-Tree Trush
RGB
Deptr
Slond Map
Object
Object Ae
RSR
Deptn
Globol Map
Object Object se
Step
Drop
Expested
Ar
(b:1) ebyezhs
(b.1) abjects
Figure 4: Visualization of the learned object transition matrix M (Left) Mapping objects optimization grid and their belief representation in matrix M.; darkened columns in M denote high-confidence presence in corresponding cells. (Right) navigation scenarios, with the red arrows point to the cell v where the agent predicts the target object is most likely located.
is crucial note that this discriminative belief representation is learned without direct super- vised labels for M Its emergence is attributed to the implicit feedback loop established by the optimization component, guiding the network to produce features conducive to effective planning. This underscores the efficacy of leveraging an integrated optimization model to distill action- able insights from uncertainty and enhance the learning outcomes of navigation agents.
Table Impact of various coverage levels
Task Performance
Prediction Error
#
ax
Success
SPL
Mean
Variance
1
0.01
80
66
0.90
0.015
2
0.05
0.84
0.023
3 0.10
72
56
0.80
0.047
0.20
52
0.76
0.062
4.3 Ablation Studies
Conformal Coverage Level (1 a1) Table 2 presents an ablation study on the desired marginal coverage level (1 - a) for the conformal prediction sets, examining its impact both task perfor- mance and the agent's predictive accuracy S-MON(m = 2). As observed, decreasing the target coverage results in less conservative uncertainty sets 2(x). which consequently leads to poorer performance as agents fail to make robust estimations and decisions.
To further assess the agent's predictive accuracy (termed Prediction Error' in Table3), we evaluated the precision the learned M.. For each object i, we derived a belief distribution over the V grid cells from M. Non-Maximum Suppression (NMS) was then applied to identify the cell with the highest belief. This predicted cell index was compared against the ground-trunt object cell (discretized based its relative angle to the agent), and the prediction error was calculated. The mean and

---

Table 3: Impact of task weight À (Left) and optimization scale (E, (Right): S- S-MON
Task Weight
Optimization Scale (E, 7)
#
À
Successt
Progress
SPLt
PPL
E
Success
Progress
SPL
Inference Time (s)
1
0.05
67
74
50
53
4
75
79
64
0.008
0.1
77
81
59
64
76
80
66
0.040
3
0.2
80
86
66
72
5
80
86
69
0.547
0.4
52
57
35
40
80
85
72
1.084
standard deviation of this prediction error are reported. The results show that as increases, the mean prediction error also tends to increase, suggesting object location estimates derived from them become less reliable. This corroborates the observed decline in navigation performance, less accurate understanding of target locations naturally hampers efficient navigation.
Action Blending Factor A. Table 3 (Left) displays the impact of varying the action blending factor on the agent's final task performance S-MON(m 2). We observed that initially, increasing the weight assigned to atack correlates with improved task performance. However, beyond a cer- tain threshold of 0.4, an excessive reliance on atask leads to a notable degradation in navigation performance (approx. 20% drop in Success). This decline is likely because diminished role for the network's ' exploratory actions and insufficient exposure to environmental reward signals r hinder the agent's ability learn an accurate internal world model. Consequently, the learned object transition matrix M, may fail to capture meaningful environmental dynamics, appearing less informative for effective long-term planning.
Spatial and Temporal Tuning, We also investigate the influence of the optimization problem's scale -pale-specffiaally. the maximum number of optimization steps and the size of the optimization grid H--n the agent's navigation performance, with results presented in Table 3 (Right). These parameters collectively define the agent's spatiotemporal planning horizon. Our findings indicate that an increase in the scale of the optimization problem generally correlates with an improvement in navigation Success. Concurrently, while the inference time for solving the optimization problem does increase with scale, it remains consistently low overall. This highlights a key advantage of our parameter-free optimization component, which avoids the computational overhead typically associated with training additional complex predictive modules.
To better approximate real-world complexities and demonstrate the scalability, further explored larger grid sizes and developed an acceleration technique based on basis function expansion and sparse kernel approximation maintain computational tractability; however, to maintain focus NEURO framework in the main paper, these extensions are deferred to Appendix F.
Task-Specific Optimization. Finally, we in- vestigated the benefits of task-specific learning on S-MONm=2 task, with results in Table 4. To this end, we compared the agent's 's performance on the S-MON task under two optimization for- mulations: (i) The full S-MON" model, which incorporates constraints specifically designed for sequential target acquisition. (ii) A simplified formulation, denoted S-MON#, which is structurally equivalent to the U-MON' model (with n target). Notably, there 6% improvement in SPL, indicating that the explicit inclusion of task-specific objectives effectively guides the agent towards more efficient, shorter trajectories. This finding validates our approach of adapting the optimization component to better align with the nuances of different visual navigation tasks.
Table 4: Impact of task-tailored optimization
Formulation Success Progress
SPL
[S-MON]*
80
86
66
[S-MON]*
82
Generalization Across Task Variations. To assess the generalization capabilities of our agents, we conducted experiments where models trained on a specific instance the S-MONtask, denoted S S-MON,, (where m represents the number of navigation goals, and is the specific number of sub-goals used for training), were evaluated on different instances SMON The performance, typically measured by success rate, is reported in Table 5.
The results Table indicate a common trend: agents trained on tasks with fewer navigation goals (c.g., 1) tend to struggle when generalizing tasks requiring larger number of goals (c.g..

---

Table 5: Success scores on S-MONtasks, Agents are trained on task S-MON (rows) and evaluated on task S-MONm (columns). The table below indicates the relative score drop (%) compared to in-task learning performances (diagonal).
OracleEgoMap
Lyon (SoTA)
NEURO
m: train leval
2
3
2
3
1
3
83
47
28
86
61
50
90
68
57
79
64
32
82
76
54
88
80
60
77
63
37
81
75
57
88
79
62
-17
-9
-15
-7
0
-12
-5
-4
0
-5
-4
0
-3
-2
-2
0
-2
m 33). However, the NEURO framework appears to alleviate this degradation. The average drop in performance when transferring agents trained on a specific m to other m values comparatively lower for NEURO. We attribute this improved generalization the embedded optimization model, which likely captures underlying task structures and common rules that are transferable across variations in the number of goals. This allows NEURO to adapt more robustly to related but unseen task configurations.
Conclusion
We introduced NEURO, pioneering framework enabling, for the first time, end-to-end training of neural networks with downstream robust optimization for visual navigation. NEURO tackles the critical challenge of agent generalization in data-scarce, partially observable environments by synergistically integrating deep learning's s perceptual strengths with robust optimization' principled, uncertainty-awer aware decision- maiisin. Extensive experiments on U-MON and S-MON benchmarks demon- strate NEURO's superior performance and adaptability. This work not only establishes promising new paradigm for developing effective and robust embodied AI agents for navigation but also high- lights that NEURO's core fenet-uuiing learned predictions with formal optinization-extenndd well beyond this domain (e.g., power dispatch, see Appendix G), moving an important step towards more robust and capable AI systems. Such advancements have broad societal implications, and we acknowledge the need for consideration of ethical implications, such as ensuring safe deployment and mitigating potential biases in learned behaviors.
Limitations and Future Work. A key limitation is NEURO's reliance on convex approximations for uncertainty. While PICNNs effectively generate tractable convex uncertainty sets significant step-the cannot fully capture inherently non-convex uncertainties common in complex visual scenarios. Addressing general non non-convex robust optimization remains formidable open challenge. Our use of PICNNs thus a pragmatic, effective, yet approximate strategy. Future work will explore advanced techniques to model and incorporate non-convex uncertainty, aiming to further enhance agent robustness in complex real-world settings.
10

---

Acknowledgment
This paper was supported by the Natural Science Foundation of China under Grant 62303307, 62225309, U24A20278, 62361 166632, U21 and Sponsored by the Oceanic Interdisciplinary Program of Shanghai Jiao Tong University, and in part by National Key Laboratory of Human Machine Hybrid Augmented Intelligence, Xi' an Jiaotong University (No. HMHA11200208).
References
[1] Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. Differentiable convex optimization layers. Advances in neural information processing systems, 32, 2019.
[2] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as layer in neural networks. In International conference muchine learning, pages 136-145. PMLR, 2017. 3
[3] Brandon Amos, Lei Xu, and Zico Kolter. Input convex neural networks. In International conference on machine learning, pages 146-155. PMLR, 2017. 5, 20
[4] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and lang-augee navigation: Interpreting gisuully-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674-368 2018. 2
[5] Tina Balke and Nigel Gilbert. How do agents make decisions? survey. Journal of Artificial Societies and Social Simulation, 17(4):13, 2014.7
[6] Francisco Bonin-Font, Alberto Ortiz, and Gabriel Oliver. Visual navigation for mobile robots:
survey, Journal of intelligent and robotic systems, 53:263-296, 2008. 2
[7] Thomas S Collett and Matthew Collett. Memory use insect visual navigation. Nature Reviews Neuroscience, 3(7))542-552, 2002. 2
[8] Donald I. DeAngelis and Stephanie G Diaz. Decision-making agent-based modeling: current review and future prospectus. Frontiers in Ecology and Evolution, 6:237, 2019.
[9] Priya Donti, Brandon Amos, and Zico Kolter. Task-based end-to-end model learning in stochastic optimization. Advances in neural information processing systems, 30, 2017. 1,
[10] Ahmad Elawady, Gunjan Chhablani, Ram Ramrakhya, Karmesh Yadav, Dhruv Batra, Zsolt Kira, and Andrew Szot. Relic: A recipe for 64k steps of in-context reinforcement learning for embodied ai. arXiv preprint 10000224,, 2024. 2
[11] Kuan Fang, Alexander Toshev, Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 538-547, 2019. 7
[12] Mingjie Gao, Jeffrey A Fessler, and Heang-Ping Chan. Model-based deep enn regullrrlr reconstruction for digital breast tomosynthesis with a task-based enn image assessment approach. Physics in Medicine & Biology, 68(24)244502, 2023.
[13] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cogni- tive mapping and planning for visual navigation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2616-2625. 2017.
[14] Qigang Liu, Deming Wang. Yuhang Jia, Suyuan Luo, and Chongren Wang. multi-task based deep learning approach for intrusion detection. Knowledge-Batse Based Systems, 238: 138:107852, 2022.
3
[15] Pierre Marza, Laetitia Matignon, Olivier Simonin, and Christian Wolf. Teaching agents how to map: Spatial reasoning for multi-object navigation. In 2022 JEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pages 1725-1732. IEEE, 2022. 2,

---

[16] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 518(7540):529-583 2015.
[17] Sriram Narayanan, Dinesh Jayaraman, and Manmohan Chandraker. Long-hot: A modular hierarchical approach for long-horizon object transport. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14867-14874, IEEE, 2024. 2,7
[18] Junhyuk Oh, Valliappa Chockalingam, Honglak Lee, al. Control of memory, active perception, and action in minecraft. In International conference on machine learning, pages 2790-2799. PMLR, 2016. 7
[19] Yiyuan Pan, Yunzhe Xu, Zhe Liu, and Hesheng Wang. Planning from imagination: Episodic simulation and episodic memory for vision-are aaooo--nd-laguuaee navigation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 6345 6353, 2025.
[20] Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 9982-9991, 2020. 2
[21] Assem Sadek, Guillaume Bono, Boris Chidlovskii, Atilla Baskurt, and Christian Wolf. Multi- object navigation in real environments using hybrid policies. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 4085-4097 IEEE, 2023. 2
[22] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
[23] Saim Wani, Shivansh Patel, Unnat Jain, Angel Chang, and Manolis Savva. Multion: Bench- marking semantic map memory using multi-object navigation. Advances Neural Information Processing Systems, 33:9700-9712, 2020. 2,
[24] Yunzhe Xu, Yiyuan Pan, Zhe Liu, and Hesheng Wang. Flame: Learning to navigate with multimodal Ilm in urban environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 9005-9013. 2025.
[25] Christopher Yeh, Nicolas Christianson, Alan Wu, Adam Wierman, and Yisong Yue. End-to-end conformal calibration for optimization under uncertainty. arXiv preprint arXiv:2409.20534 2024.
[26] Zitong Yu, Md Ashequr Rahman, Richard Laforest, Thomas Schindler, Robert Gropler, Richard L Wahl, Barry A Siegel, and Abhinav K Jha. Need for objective task-based evaluation of deep learning-basse based denoising methods: study in the context of myocardial perfusion spect. 3
Medicul physics,
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Abstract and Introduction Section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: see Conclusion Section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
see Method Section and Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Method Section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: see then link in the Abstraction Section.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececccccccccct Sttp://///.iieeeeeeeuuubiiiiiiii ssisnnoillll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: see Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: see Experiments Section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eerrpp.ee/ee//ee cc
Answer: [Yes]
Justification: This work conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Conclusion Section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
16

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
17

---

Answer: [Yes]
Justification:
we include our code/model in the Supplementary Material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
18

---

Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
: Please refer to our LLM policy (https://neuri) htw.arrpps.cccooofeeerreeeee/e/e for what should or should not be described.
19

---

A Formulation of S-MON
We now present the optimization formulation of S-MON. Unlike U-MON, S-MONdoes not require balancing between multiple objectives. Therefore, in this setting, we additionally aim to reach the goal via the fastest possible path, rather than just prioritizing information (belief) gain.
[S-MON] min max M
ΣE
7
Σ
(3)) + Apr]
TET
En
s.t.
|pt+1 Pellz d,
Vt {0, (0,...,, 1}
Po = 1B/2 1B12
E/22T
5 (pi)2. (pi)y E, Vt T {0}
of,3% E [0, VEET
(a)); = E WEV (5 Ju. vt E T [0),
(f)) E
(89 V T) [0), V VEV
(8%). f0,
if 0or _T. +1
V-TT otherwise
(0t) (a)) v: Vt T (0}, eEV
d(pt) 10x (pt)2l + 10y (pe)ul Vt
V
0g (v mod F.) + 0.5, fy = |t/F] 0.5
Ap: ||pe+1 -pel12, V1 (O.....7-11 1}
B Architecture of PICNNS
To model the uncertainty set in flexible and general manner, adopt the PICNN as the score function g. Compared to traditional box or ellipsoidal uncertainty sets, which impose shape priors and thus arbitrarily restrict the form of the uncertainty, the PICNN can approximate the true uncer- tainty landscape- often non-convex- by composing multiple convex components, providing a more expressive and general representation. Using PICNNs allows us represent complex uncertainty sets while preserving convexity with respect to a subset of the inputs. For simplicity of notation, we omit the index i. Specifically, since the PICNN architecture ensures convexity only with respect its second input, we define ((x) as the sub-level set of g M) with æ fixed and M varying, i.c.,
((x)) == {M g(x, M) S p}-
for some threshold p. This formulation enables us to capture input-dependent uncertainty sets in principled and learnable way.
For PICNN [3], the score function g is defined as
g(z, M) WLOL V.M + bb,
where the internal layers are computed recursively follows:
ut+1 ReI.U( + rr), up = x,
0++ ReLU(Wia + ViM + bb), on 0,
Wi Wi diag((W/u + wi]+),
Vi = Vi diag( Vius + vi),
6 Bruq + b,
for layers / = 0,.., L 1. Therefore, the full set of parameters for the PICNN model is given by:
OPICNN
{ RRi Wi, Wr,wr Vr, Vi, v. B,, by O, O..... 1)
20

---

Note that, in certain cases, the inner maximization problem of the original problems with PICNN- parametrized uncertainty sets may be infeasible or unbounded ones. This problem stems from too small a chosen y (making Q((z)) empty) or Q((r)) is not compact. To address this concern, we modify PICNN architecture to ensure its sublevel sets are compact and introduce slack variable to prevent it from being empty. Such modifications won't alter the general form of our problem.
C Proof of Proposition: Coverage with Quantile
We present a standard proof for the proposition here:
Proposition 1 (coverage with quantile) Let the dataset 'D {WEn Ma) to be sampled iidd from the implicit distribution P gained during training phase. And q is set be the (1 -u)-nnntille for the set (g(Zn, Mn) n=1 then (1xx)) gains the following guarantee.
- a S P.M.D(( E [((x)) S a N+1
proof. Let (- g=g M.) for = = N, and 9 =
To avoid handling ties, assume the It distinct with probability We begin by proving the lower bound of the inequality.
Without loss of generality, sort the calibration scores such that 9i have that q = ([(m-1((1-o)) when a 2 n+1 and g = oo otherwise, where [-] denotes the ceiling
gN. In this case,
operation. Assume 2 nFI- Observe the equivalence of the following two events:
(MEN(() = {g < 4}
Using the definition of G. we have:
{M ((X)) = (9 & ()=+1)(1-)))
Then, due to the exchangeability of variables in (1:,, M,) for i = I,... N, the probability of G falling below a specific gl is:
k P(G S 9b) = N+1
In other words, equally likely to fall in anywhere between the calibration points 91.-.. 9N. Note that above, the randomness is over all variables 91 ....... 9N. From here, we next conclude:
[[n + 1)(1 a)]
P(g I ((+1)()))))
+11 21-a
Thus, the lower bound is proven.
For the upper bound, we assume the conformal score distribution is continuous to avoid ties. The proof follows similar process the one of the lower bound.
D Derivation of the Dual Problems
We begin by defining the following notations: Oy denotes a zero vector length V. Ivxv and Ovxv represents the V V identity matrix and all-zero matrix, respectively.
The decision variable the inner maximization problem M,, while pt is treated as a constant. We first rewrite the PICNN calibration constraint here for each and v.
gı(x', M') < q. Vi E
(8)
This can be equivalently reformulated as follows. Similarly, the indexes and are omitted for simplicity.
σ: 2 Os, = L,..,,L 0++1 Win ViM + b, VI O0......
Wror + VLB + by S
21

---

To see why this holds, observe Eq. (9) is relaxed form of Eq. (8), derived by replacing the equality constraints 11+1 ReLo + ViM bi) the definition of the PICNN with two separate inequalities 0141 2 O₂ and 01+1 2 Wio V,M + b,, for each / = 0, .... L 1. Conse- quently, the optimal value of the relaxed problem cannot be less than that of the one of the original problem. The relaxed constraint Eq. (9) can be expressed in the following matrix form, where A E (22A b E 2
AIM 01
01. b
-Id
0,8
La
0g8
(10)
A =
VO
-a
b
bo
W1
Ia WL 9 br.
b
VE
We first use the [U-MON] problem an example and re-index M with i, v ensure the completeness of the optimization problem. Additionally, we decompose the matrix M; into vectors Mÿ for = L,... V. We denote the vector [M, (01), (aL))l" E V- as k."
To this end, the uncertainty sets of the two-stage robust optimization problem transformed into linear constraints. We reformulate the optimization problem as follows:
[U-MON] min max Ph
Σ LET
Σ En
(B))
K.
s.t.
(1)
af, E 10.1),
T,a = (HK)) (B E (0),i
(11)
E3 = 1,
8. = c, ViEn
S3 = D'a,,WiEnttT
AV VVVVE
where
S [0v Iyvv] E EVV(V+L)
[1
= ІIVV OvxLal E (V((++L))
[d(pe))
D'
ERVXV
d(pe)v
[1, i
(T.):
0. otherwise T. E RIxV
are constant matrices for the inner maximization problem.
Assuming that the problem has optimal solutions and satisfies the Slater condition, strong duality. Let TT... Ms, X₁. vf, and EL represent the dual variables. The inner maximization prob- lem can then be reformulated as equivalent minimization problem using the KKT conditions and Lagrangian duality. Combining this reformulated minimization problem with the outer minimization. we derive the formulation the main paper. where (ER(V+LaX(V+)) and I'g RIx(++1) represent HT S and [ 0 0] 0] S, respectively.
invoke
22

---

Similarly, the dual form [S-MON"] of the [S-MON] problem can be derived as follows.
[S-MON*]
min Σ
Σ-c
X
E + Ap.
p.N.p.A......k
s.t.
(1) (3)
Ap: - |P - Palla, VET 100
V. SO, ViEn
DJ Eney पे - S0,i n, t LET \(0) (0)
E ,3, (A 00 20, Vi
(T')T + (vt)
LET
H - Ev T (k))
=
Gi.t 0,
if
Evev T,,
otherwise
E
Convergence and Optimality of NEURO
Convergence and Gradient Consistency
Network components, parameterized by are trained using composite reward signal derived from both optimization-spcciff rewards (r/aak) and non-o non-optimization rewards (r...). Convergence to desirable @ requires consistent gradients from these disparate sources.
We establish gradient consistency as follows. The gradient Vor!aek(0) (0), originating from the convex Robust Optimization (RO) problem, inherently directs parameter updates towards configurations of 9 that enhance the RO problem's objective value, thus steering towards an optimal parameter set 0r. Similarly, the gradient Verg (0) from non-optimization objectives (e.g., generation quality) is engineered to guide 0 towards the same 0*, The well-behaved nature of Vert"y (0) is maintained through techniques including gradient clipping and normalization, ensuring its boundedness and local consistency.
Since both raask and rnnn designed to drive @ towards common (near) globally optimal 8", their respective gradients, Vor(ask(0) and Vortask(()) (0), are expected to exhibit consistent alignment. This directional alignment implies that updates suggested by each gradient component are collaborative rather than contradictory, which is critical for stable and effective learning using gradient-based meth- ods. For applications involving non-convex or non-linear problem structures, NEURO's ' framework can be extended by incorporating PICNNs to maintain output convexity, by integrating advanced gradient estimators and convex relaxations.
Beyond theoretical analysis, we empirically address gradient consistency in this multi-objective reinforcement learning setting by employing the Goal Vector Method (GVM) to combine r rlask and reay_ GVM formulates each reward component a dimension in goal vector, and adaptively projects the multi-dimensional gradient onto a unified descent direction. This projection balances task and environment objectives while ensuring that parameter updates lie within a consensus direction that respects both reward signals. By avoiding naive scalarization, GVM mitigates gradient interference between competing objectives and facilitates more stable convergence during training.
Optimality Guarantees
NEURO's 's convergence to the optimal solution of the final, learned RO problem is guaranteed under two readily satisfied assumptions: (i) The RO problem, formulated with the uncertainty set (0) generated by the converged network parameters 6*, adheres to the structure of Disciplined Convex Program (DCP). (ii) The objective and constraint functions within this RO problem are differentiable with respect to the decision variables and any parameters influenced by 0.
Under these assumptions, and bolstered by the gradient consistency established above (which ensures the convergence of 0 to stable 0*), the RO problem defined by U((*) is convex. Consequently,
23

---

standard convex optimization solvers can efficiently identify its global optimum, x (8"). NEURO is thus guaranteed to converge the optimal solution for this specific, data-driven RO formulation.
While the learning process for (which defines the uncertainty set (+))) navigates a potentially non-convex landseape-meantnn ((())) itself may not be the globally "true" uncertainty set in an absolute cense-our framework guarantees optimality for the RO problem constructed with U ((0). Crucially, due to our calibration guarantees, the learned uncertainty set ((())) effectively encapsulates a (1 a)) confidence region for the underlying uncertain parameters. Therefore, the solution x (0*) derived from NEURO represents the optimal worst-case performance over this empirically validated, high-confidence uncertainty region.
F Scalability of NEURO
In this section, discuss the scalability of the NEURO framework, focusing its runtime and task performance under larger grid configurations. We first examine the performance of the stan- BardNeuROOfrmulltti and then introduce a method enhance its computational efficiency for larger-scale problems.
Baseline Scalability
As shown in Table 3, while the absolute solving time for the optimization model remains manageable for moderately sized problems, it exhibits noticeable growth with E and (this scalability trend and absolute solving time can be easily verified by constructing optimization problems of similar scale and solving them with Python library CVXPY). For typical indoor robot navigation, E 20 might suffice. However, for applications demanding larger fields of view (e.g., autonomous vehicles), the optimization grid at this scale may not adequately represent real-world environments with fidelity. The computational complexity of the navigation task's 's optimization component is O(E²)) indicating a quadratic dependence on the grid dimension E. Given that E often has a more significant impact on navigation performance than т, in practice, one might fix to smaller constant, resulting in a complexity of O(E²). To effectively scale to larger environments, it is crucial to mitigate this quadratic growth associated with iterating over E² grid cells.
Improving Scalability via Basis Function Expansion
To address the scalability challenge, we explore the use basis function expansion and sparse kernel approximation. Specifically, we represent the optimization variables at and / (representing aspects of the utility or value functions over the grid) via predefined or network-learne basis function decomposition (x).))))
Σ (z,
B = (z,((,
(12)
This representation subsequently modifies the original problem constraints. For instance, constraint involving these variables, such as constraint (2b) from the main paper, can reformulated based on the basis functions follows:
K
[J
a = Σ 2-11
Mu, v) v)ox(u, () e)dudu b5-1.
(13)
The integral term in Eq. (13), Jf (", v)or (u, v)ou(u, v)dude, be computed before solving with given M' and dR. At this point, the optimization grid size is determined by the dimensionality of basis function vector (af. (() ppeddiciing high-ddmensiioal vectors computationally inexpensive for neural networks. This adjustment reduces the time complexity of the underlying optimization module (referred to as U-MON in internal development) from O (EE')) to (KT), where K < E² is the number of basis functions. Empirically, this approach achieved 23.6x speedup (e.g., 0.023s inference time at r = 8, = 15) specific configurations, enabling consideration of significantly larger effective E.
Table presents the performance ofNeunOween employing this basis function expansion. The models were trained for 250k updates, consistent with the original experiments.
24

---

Table 6: NeuRO performance with basis function expansion. (S- (S-MONm=2)
E
Success($)
Progress (%)
SPL(%)
Inference Time (s)
20
82
85
76
0.251
50
85
0.670
100
85
87
80
1.310
The results in Table 6 demonstrate that the optimization problem's 's solving time now scales almost linearly with E (effectively, with K which might grow slowly with E, or E itself if K is chosen proportional to E rather than E²). However, we observe no substantial improvement in navigation performance for these larger E values. We attribute this two primary factors: (i) for typical indoor navigation tasks, the limited field of view and task characteristics may mean that smaller optimization grids (E 20) are already sufficient to capture the necessary environmental information; (ii) the newly introduced module for predicting basis function coefficients (o) 6() may require dedicated training strategies, more extensive data, or further architectural refinement fully realize its potential.
In summary, this exploration offers a promising direction for enhancing the scalability of theNeuROOfamework Nevertheless, consider the primary contribution of this work to be the introduction and validation of the therefore presented as supplementary material, highlighting potential avenues for future research and application to more demanding, large-scale scenarios.
This discussion on scalability is
G Broader Application: Power Grid Scheduling as Case Study
We recommend adopting the NEURO framework in the following scenarios. First, data-scarce environments where task performance is critical, as the optimization model can effectively capture general task rules that are intuitive humans. Second, networked systems characterized by multi- stage, multi-objective, multi-agent, and multi-constraint structures, where optimization-based models offer natural advantage globally coordinating multiple entities. In this section, we study representative application in the power market: the capacity expansion problem.
G.1 Problem Setup.
The problem address is classic one in the context of power markets: capacity generation. The background of the problem is as follows: To enhance energy utilization efficiency, the power utility has deployed additional battery storage devices at each node in the grid. Our task, the power utility, to determine the optimal grid scheduling strategy to minimize overall electricity costs. A key challenge in this process is the uncertainty of electricity purchase prices, which introduces variability into the decision-making process.
However, we propose the following modifications to the original NEURO framework, see Fig. 5: First, since long-horizon planning is not required, we utilize a simpler deep learning structure instead of deep reinforcement learning. Second, as scheduling decisions are only meaningful under accurate power prices, we incorporate the loss between the network's ' predicted values and the ground-truth values into the overall loss function.
Uncertainty Estimation
Input w
LSTM
Network Calibration
LSTM(W)
Capacity Exponsion
((w.
Optimal Solution
Epeed
X ('(w)
Solution Calibration
Lieck
Figure 5: The workflow of solving capacity expansion problem using the NEURO framework.
25

---

Network Component
We generated simple dataset D {Ww, Mn. Un)]az which contains weather data wn and the corresponding power prices Yn. over M time points. The weather data w,, = (h Sn Px)) include humidity An sunlight intensity Sn. and precipitation intensity Pn- The network module consists of an LSTM network capable of handling non-uniform step-size inputs, followed by a PICNN. To simplify the problem, the predicted uncertainty of power prices is represented as upper and lower bounds (i.e.. box uncertainty). During the i-th training step, the network takes historical electricity prices and corresponding weather data over the past K time steps as input, 2; {(wz, B5)1 K+1 and predicts electricity prices S = {(r } for the next T time steps. 8f and 0 represent the upper and lower bound, respectively. The prediction loss is defined as the mean squared error (MSE) loss between the uncertainty mean and the ground truth.
MSE 8J +
Ppeed =
(14)
2
Optimization Model
Our power grid model adopts the 3-phase lindist flow model. In the lin-dist flow power grid model, the grid structure is assumed be tree. We consider N nodes, where each node i and its downstream nodes form a subtree T, Let P. represent the set of all paths from node i to any downstream node. As the power utility, the decision variables include the complex power s and voltage matrix es at each node, with the voltage represented by voltage matrix. The objective of the optimization is to minimize the total electricity procurement cost. Given the tree structure, this is equivalent to minimizing the product of the actual power supplied by the slack bus (node 0) and the electricity price. The optimization model can be formulated as follows.
min max 8,0 DEE
(cRe(sh)
(15a)
s.t.
sf 00
JEN 84
(15b)
N1 Even, sh S/y = diag(X1))
(15c)
vf v/ -EGN)ER, ( + -1x=11
(15d)
synin < 5 $ sqaa
(15e)
uyein < diag(w)) S 1
(15f)
Re(s), b1 + Re(sg = Re(s))
(15g)
SOC j SOC, + +Re(s))
(15h)
SOC = O, sOc < SOC < SOS man
(15i)
In the aforementioned model, Re(-) denotes the real part operator, (.)d represents the Hermitian matrix operator, and diag(-) refers the diagonalization operator. Eq. (15b) to Eq. (15d) define the power flow constraints in the lindist flow model, where y is a constant phase matrix, 2j4 represents the impedance matrix between nodes and k; and S/. and X5 are intermediate variables. c15e and Eq. (I5f) specify the operational constraints for each node. Eq. (15g) describes the composition of the power injection s; at each node, where 8; is the dispatch power value of the battery, and sa represents the fixed constant power consumption at the node. Finally, Eq. (15h) and Eq. (15i) impose the state-of-charge (SoC) constraints for the battery devices.
Assuming the problem has an optimal solution = = (s" v²), the task loss Ltask is defined as the optimal value of the objective function correspond- ing to this solution.
Table 7: Comparison on the capacity expansion task.
# Learning Framework Prediction MSEJ Task Loss.]
1
ETO
275.44
810.52
2
NEURO
287.32
757.81
26

---

Experiments
We generated a set of synthetic weather data and corresponding electricity prices for training the network. In the downstream optimization model, the power grid structure is assumed to be binary tree with seven nodes. The weights of Loss Lpeed and Loss Lrosk in the NEURO framework are set to 0.8 and 0.2, respectively. As baseline, we adopted the traditional eestimate-hhen-opimii (ETO) task-based framework, where the network and the optimization model are decoupled. In the ETO method, the network is first trained to predict electricity prices, and during the testing phase, the network- preworr-predicted prices are fed into the optimization model to compute the scheduling strategy.
(o) ETO
(b)
Power Injection for Bus Nods
Model Prediction
Model Prediction
-21000
01000
Openstion Time
Openation
Figure 6: Visualization of prediction error on the capacity expansion problem.
The task loss and prediction loss for NEURO and ETO were recorded and are shown in Table 7, and the visualization of prediction loss can be found in Fig. 6 (the grey region is the visualization of the box uncertainty from the estimation). Table indicates that NEURO's 's prediction accuracy is slightly lower than the traditional approach. but its task performance improves by 7%. Therefore, we believe NEURO's 's learning ability inherently involves trade-off. However, in real-world applications, only accurate price predictions hold practical significance, making NEURO more suitable for tasks where precise recognition is not required. For example, in navigation, we care less about whether the robot forms human-like cognition and more about how it interacts with the environment to complete tasks.
H Impact of Target Object Geometry
We further investigate how the geometry of goal objects affects agent navigation performance in Table 8. We observe that agents find it easier to identify targets located near camera height (e.g.. Cylinder) compared taller objects that span the agent's full visual field. This suggests that object shape influences navigation success. Notably, NEURO significantly narrows this performance gap, likely due to its ability encode general task-level information while being less sensitive to low-level visual cues such as material or shape. As a result, agents trained under the NEURO framework demonstrate improved generalization and adaptability across different task configurations.
Table 8: Navigation performance under different object geometries.
Method
Cylinder
Cube
Sphere
Success
SPL
Success
SPL
Success
SPL
OraMleEgoMap
64
49
62
47
57
43
Lyon
76
62
74
57
73
57
NEURO
80
66
78
65
78
63
I Experimental Details
This appendix provides specific implementation details and parameter configurations for the key modules within the NEURO framework, complementing the descriptions in the main paper. The network components within the NEURO framework consist primarily of Neural Perception Module and PICNNs.
27

---

For the Neural Perception Module. the architecture and parameters are adopted from the OracleE- goMap (Occ+Obj) configuration described the original MultiON work; for the PICNNs, apply the following parameters:
Table 9: Hyperparameters for the Partially
Neural Network (PICNN).
Parameter
Value
Input dimension
32
Convex input dimension
768
Hidden layer dimension
256
Number of layers
3
Output dimension
V
Include y in output layer
True
Feasibility parameter
0.0
The 'output dim' of dimpputtdimmii along the last dimension, to produce the final output probabilities relevant to the navigation task.
subsequently reshaped into a E' x E grid, followed by a softmax operation
All models were trained for 250,000 updates. Training was conducted on single NVIDIA Quadro RTX 8000 GPU.
28


Reviews:
[{"review_id": "J18b9fPEDV", "reviewer": "Reviewer_M9u1", "summary": "The authors present a hybrid framework (NeuRO) that combines conventional neural network perception and policy modules with a task-based robust optimization pipeline to address data scarce scenarios which will often lead to poor generalization performance (out-of-distribution failure) for typical learned neural network approaches when faced with unseen scenarios during testing.\n\nThey apply their approach to the visual navigation task of sequential or unordered Multi-Target Object Navigation (MultiON) where the agent must navigate between a series of goals (represented by the $\\mathbf{G_t}$ vector). The agent operates on RGB-D image inputs, maintains an egocentric map and operates under grid-based actions (described in lines 91-100). \n\nTheir approach is composed of a number of modules and an overview of the proposed NeuRO method is illustrated in Figure 2. Namely a base neural network perception module, a Partially Input Convex Neural Network (PICNN) to convert the prior potentially unreliable neural network outputs to a set of calibrated convex uncertainty sets, and a robust optimization pipeline.\n\nThe base neural network perception module is described in section 3.1 (page 4) and forms the baseline method without the proposed robustness modifications. This perception network ultimately learns latent state $s_t$ and features $f_t$ from which an action policy network $\\pi(a_t^{\\mathrm{net}}|f_t)$ is formed.\n\nIn section 3.2 (page 4), their formulism for a robust optimization planner is presented. They describe a notion of uncertainty through a set of object transition matrices $M^i$ (representing the i-th goal object's probability to move between cells each each $M^i_{uv}\\in M^i$) on a discretized grid along with additional terms for prior $(\\beta_i^t)_v$ and posterior $(\\alpha_i^t)_v$ beliefs representing the agent's confidence that a target object occupies the cell. Using these terms, they seek to optimize the objective function of finding an agent path that maximizes the accumulated discounted capture beliefs (equation 4).\n\nSection 3.3 (page 5), describes the usage of PICNNs to output a calibrated convex set of the prior discussed transition matrices and finally section 3.4 (page 6) describes how the resulting action $a_t^{\\mathrm{task}}$ and reward $r_t^{\\mathrm{task}}$ from the robust optimization procedure are used modulate the policy network and agent reward signal.\n\n__Novelty and Contributions:__\n\nThe main novelty of the approach is to provide feedback from a robust optimization pipeline to a conventional neural network action policy. They do this by first using a Partially Input Convex Neural Network (PICNN) to determine a calibrated convex uncertainty set of object-tracking transition matrices from the base neural network's state vector. A robust optimization problem is solved using this set, and the resulting $a_t^\\mathrm{task}$ and $r_t^\\mathrm{task}$ are used to modulate the original action policies perceived reward and executed action (this overall process is described in section 3.4). They assert that this uncertainty-aware approach for tracking target objects during visual navigation is well suited for generalizing the the test environment under data scarce scenarios.\n\n__Experiments:__\n\nDuring experiments, the authors compare against and outperform a number of state of the art baselines (Table 1) in sequential and unordered multi-target object navigation (section 4.1). They hypothesize that the gain in formation is due to 1) the agent leveraging the additional reward and action signal from the downstream robust optimization solution and 2) the method’s ability to coordinate between multiple targets.\n\nIn section 4.2, the authors compare the NeuRO method vs the augmented neural network policy where their method obtains a significantly higher success rate versus training data (Figure 3).\n\nFinally, experimental section 4.3 evaluates various ablation studies.", "questions": "- The detection factor ($d(p_t)_v$ in equation 3b) seems to be a relatively simple distance scale. I would of expected this value to be based on line of sight / obstacle occlusion as well. Can this be clarified? Is line of sight information incorporated in some other fashion when updating the target object belief state?\n\n- In Table 3, could the authors please clarify what precisely \"Inference time\" refers to? On line 282 it states \"inference time for solving the optimization problem...\" Is this inference time only relevant during the training process to provide a reward feedback for training the agent's policy (as in Algorithm 1 on page 6) or is it also required during the testing phase? Currently this amount seems quite high for real-world usage if it is required at runtime.\n\n- Given the method appears to be currently tied to a discrete grid for the transition matrices for representing object locations, do the authors have any thoughts on making use of this method in a continuous space and what the implications of that modification might be?", "limitations": "Yes. Limitations are listed by the authors on lines 329-335.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "__Strengths:__\n\n- __Novelty and performance (originality and significance):__ Novel approach which makes use of reward and action feedback from a downstream robust optimization process to modulate the agent's perceived rewards and action policy. This is an especially useful method for the case of data scarcity (such as for robotics). They note improved performance versus a wide array of existing state of the art baselines (Table 1). This process is further described above in the summary section.\n- __Breadth of experiments (quality):__ This is done well for the most part (however, see weaknesses section below for additional comments). Comparison against range a range baselines is done (Table 1). Intuition regarding the improved results is provided (the agent leveraging the additional reward and action signal from the downstream robust optimization solution and 2) the method’s ability to coordinate between multiple targets). Ablation studies and hyper-parameter sweeps are also presented to provide justification regarding the selected method configuration. The authors further provide intuition as to the effects of these hyper parameters.\n- __Manuscript clarity:__ The manuscript is generally well written and generally easy to understand.\n\n__Weaknesses:__\n\n- __Experimental rigour (quality):__ Results in experiments lack any standard deviation / confidence interval metrics for repeated trials (for example with different network random seeds). This would be especially important for some metrics which are quite close to baseline performance in Table 1 (PPL for S-mon (m=2) and Progress for S-Mon (m=3)).\n- __Potential limited use (significance):__ The method appears to be limited to a discrete grid for object locations. In addition, the detection factor ($d(p_t)_v$ in equation 3b) seems to be a relatively simple distance scale. I would of expected this value to account for more information in the scene such as line of sight and obstacle occlusion as well. Likewise, the inference times in table 3 seem relatively large and require further clarification if this method can be used in real-time. See questions to below to authors for clarification.\n- __Experimental description (clarity):__ There is a very limited amount of context when beginning to describe the experimental setup in section 4. Ideally I feel a brief introduction should be given and a reference to relevant sections in the appendix if needed.\n\n__Overall Assessment and Suggestions for Improvement:__\n\nCurrently I have marked this work as borderline accept as it is novel and beats existing state of the art baselines (see strengths section above). However I feel that it could be improved with additional confidence interval analysis in the experiment section (see weaknesses above). There are also several limitations (noted in the weaknesses section) and the authors' own discussion regarding limitation to convex uncertainty sets (discussed on lines 329-335).\n\n__Minor Points:__\n\n- Lines 114-115 and 116-117 appear to repeat the same information. \n- On line 163, I believe it should read “$i$ for conciseness”.\n- Table 3 appears in the text before table 2. I imagine the label for these should be switched?", "was_revised": true, "final_justification": "After reading reviewing the authors' rebuttal - and absent any further modifications - I have increased my overall score to `Accept`. \n\nMy primary original concerns were related to experimental rigour (missing experimental variance over repeated random seeds) and potential limited use (discrete world space, simple detection distance metric not accounting for sight information, inference times clarification).\n\nSince then, the authors have addressed most of my stated weaknesses (including additional experimental results and clarifications). The issue of requiring a discrete world space still remains but is rationalized for practical efficiency reasons by the authors. Nonetheless, selection of the appropriate grid size without prior knowledge of the environment could pose an issue.\n\nPlease see my original review for a summary of the strengths and weaknesses of the work and author rebuttal comments for more details.", "rebuttal": "We sincerely thank you for the constructive and detailed feedback. We are encouraged that you found our work novel, well-written, and outperforming SoTA baselines. We address the specific weaknesses (W1, W2, W3) and questions (Q1, Q2, Q3) below.\n\n---\n\n### **W1: Experimental Rigor**\n We have now conducted experiments with 8 different random seeds and will update Table 1 with mean and standard deviation in the final version. (due to space limitations, we are currently able to include only a subset of the results.)\n|SMON$_{m=2}$: Test| | | | |\n|-|-|-|-|-|\n|**Methods**|**Success**|**Progress**|**SPL**|**PPL**|\n|OracleEgoMap|63.40 (±4.82)|71.92 (±2.18)|48.73 (±3.12)|56.12 (±1.62)|\n|Lyon|75.08 (±4.24)|83.44 (±1.81)|63.14 (±2.46)|68.33 (±1.73)|\n|NeuRO (Ours)|81.14 (±3.86)|85.26 (±1.53)|66.49 (±2.75)|71.10 (±1.42)|\n\nWe observed that Progress and PPL provide inherently more robust counterparts to Success and SPL, respectively. Also, the new results show that NeuRO not only achieves a higher mean success rate but also exhibits significantly lower variance, further reinforcing our central claim regarding the model’s robustness and generalization capabilities.\n\n---\n### **W2: Potential Limited Use**\nThank you for raising this point. We address it from three perspectives. Collectively, our analysis and experiments demonstrate that NeuRO is computationally efficient and effectively accounts for complex visual information in realistic environments, facilitating its applicability to real-time scenarios.\n1. **Optimization under Continuous Uncertainty**: See **Q3**\n2. **Incorporation of Sight Information**: See **Q1**.\n3. **Inference Efficiency**: See **Q2**.\n\n---\n\n### **W3: Experimental Clarity**\nWe thank you for pointing this out. Due to page limitations, we condensed the setup description. We will elaborate on these details in the revised version to improve clarity Specifically, we will:\n1. Include a glossary and symbol table in the appendix, along with a description of the basic setup (e.g., action FORWARD (0.25m), TURN-LEFT/RIGHT (30°)) for the MultiON task.\n2. Introduce prerequisite concepts such as conformal prediction at the beginning of the Method section.\n3. Add more intuitive explanations to clarify key modeling choices (e.g., using a discrete grid for efficient optimization while maintaining continuous decision variables for differentiability).\n\n---\n### **Q1 & W2: Detection Factor & Sight Information**\nWe appreciate the question and you are correct that $d(p_t)_v$ itself is a simplification. As for the line-of-sight information, we do explicitly incorporate it during the generation process of the uncertain transition matrix, where networks takes the raw depth image and BEV ego-map as inputs. We believe this incorporation contributes to our model’s strong performance even with a simple distance-based factor.\n\nTo validate this hypothesis, we conducted a new experiment to make this geometric reasoning more explicit within the optimization process itself, inspired by your suggestion. We modified our PICNN to also predict an obstacle map (just like the current goal map) and merged two map to construct a optimization grid with both object location belief and obstacles. Then, we redefined $d(p_t)_v$ based on the geodesic distance (calculated via Breadth-First Search) instead original Manhattan one to consider the predicted obstacles. The results (**Progress** metric) are shown below:\n|Progress |SMON$_{m=2}$|SMON$_{m=3}$|UMON$_{m=2}$|UMON$_{m=3}$|\n|-|-|-|-|-|\n|NeuRO|85.26 (±1.53)|72.08 (±3.20)|64.33 (±3.82)|52.11 (±5.47)|\n|NeuRO + sight info.|87.90 (±1.26)|75.25 (±3.14)|67.26 (±3.35)|54.88 (±5.82)|\n\nAs observed, this leads to a moderate but consistent improvement in navigation performance. We hypothesize this is because our perception module already captures parts of the occlusion information from the input depth data, so the contribution of the new explicit factor partially overlaps with what the network has already implicitly learned. This experiment affirms your intuition and further reinforces the physical grounding of our framework. We will include this insightful analysis in the revised version.\n\n----\n\n### **Q2 & W2: Inference Time**\nWe thank you for this crucial clarifying question. The \"Inference time\" reported in Table 3 refers to the training-time process of solving the optimization problem and computing the policy gradients via Eq. 7(b), NOT the actual runtime deployment.\n1. **Absolute Time**: The values in Table 3 are normalized for comparison (relative to $E=3, \\tau=4$; see line 523-524 in Appendix). The absolute solving times are presented in Appendix F (Table 5) and are modest (e.g., 1.084s for a large $E=20, \\tau=12$ setting). We will incorporate this clarification from the Appendix into the main paper. \n2. **Runtime Performance**: We attribute the primary computational cost to the gradient computation of the optimization model but not the solving process, because modern linear program (LP) solvers are already highly efficient for solving. To validate this analysis, we measured the runtime cost at test stage, where the agent only needs to solve the forward pass of the optimization problem to get the action offset without any gradient computation. This result confirms that NeuRO is fully capable of real-time performance. We will add this crucial table and clarification to the revised version.\n    | |Memory Consumption |Test Runtime (SMON$_{m=2}$; per step (ms))|\n    |-|-|-|\n    |Lyon|22.18MB|12.9|\n    |HTP-GCN|25.63MB|15.3|\n    |NeuRO|20.24MB|11.6|\n3. **Local Grid Scope**: The optimization time is manageable since our framework does not require a large optimization grid. This is because our optimization grid models the agent's local egocentric view, NOT the global map (Evidence: 1. The grid is re-formulated at every time step based on the current egocentric observation; 2. We always initialize agent position to the center of the optimization grid). This keeps the optimization problem small and tractable. In the MultiON task, the input ego-map spans a 5m view and each cell in the map represents a 0.8m*0.8m area, which means there are about 30 cells. Our generation process of the optimization grid can be understood as overlaying a goal position distribution onto this local map. Therefore, a grid size of $E=5$ ($V=25$ cells) is sufficient. In summary, our optimization grid simulates a local field of view, so the problem size remains bounded regardless of the overall environment scale.\n4. **Scalability**: Furthermore, we have analyzed the source of the optimization problem’s computational complexity in Appendix F, which scales with $O(E^2\\tau)$. To address this, we introduce a basis function expansion method in Appendix F to let network predict α and β. It effectively reduces this complexity to $O(K\\tau)$, where $K \\ll E^2$ is a constant. We empirically validate this approach, demonstrating its effectiveness in accelerating the optimization. However, for the sake of clarity and to maintain focus on the “network+optimization” framework itself in the main paper, we have detailed this extension in the appendix.\n\n---\n\n### **Q3 & W2: Continuous Space**\nThis is a very insightful question that touches upon a fundamental challenge in robust optimization, but modifying the optimization grid (the uncertainty part in our robust optimization problem) to a continuous space would transform our problem into a semi-infinite program (SIP). In general, solving SIPs to optimality is NP-hard and computationally intractable, especially within the tight time constraints of an agent's decision loop. While relaxations can be used to approximate SIPs, they often alter the optimization landscape and sacrifice the guarantee of finding the true robust optimum.\n\nTherefore, we intentionally adopt a discretized grid as a principled and pragmatic design choice to ensure the downstream robust optimization problem remains a tractable Linear Program. This guarantees that we can find the global optimum of the subproblem efficiently and, more importantly, that we can differentiate efficiently through it for end-to-end training. Also, from a practical standpoint, one could even view the real world map as a very fine-grained discrete grid, making our approach a practical approximation. We will add this discussion to the appendix to provide a richer context for our design choices."}, {"review_id": "UIxCPVvw9d", "reviewer": "Reviewer_XTEy", "summary": "This work proposes NEURO, a hybrid framework for visual navigation that converts network predictions into calibrated convex uncertainty sets via Partially Input Convex Neural Networks (PICNNs) and then employs robust optimization to plan actions under partial observability, achieving SOTA performance and generalization on multi-object navigation benchmarks.", "questions": "- Could the authors please report results averaged over multiple random seeds? Many results lack measures of statistical significance, such as variance or confidence intervals.\n- Why are some results of HTP-GCN in Table 1 missing?\n\nI'm willing to increase the score if the authors address the outlined questions and limitations.", "limitations": "See weaknesses above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths: \n- The presentation is clear and easy to follow.\n- Recasting multi-target object navigation as a pursuit-evasion problem is novel because previous research has relied almost exclusively on POMDP formulations; this alternative perspective can open new avenues for multi-target object navigation.\n- Calibrated uncertainty sets directly inform the planner, a step toward reliable agents that must quantify risk.\n- The method (NEURO) significantly outperforms baselines in the testing (unseen) scenarios, suggesting better generalization performance.\n- Provides the full codebase to reproduce the results.\n### Weaknesses:\n- Many results (e.g., in Table 1, Table 3, and Figure 3) lack measures of statistical significance such as variance or confidence intervals.\n- The experiments omit recent baselines, e.g., PoliFormer [1]\n- While this work focuses on multi-object navigation, including results for the single-object setting ($m=1$), would be helpful and strengthen the experiments section.\n\n[1] PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators, Zeng et al., CORL 2024.", "was_revised": true, "final_justification": "The authors’ response addresses all of my questions and concerns, and I have increased the score accordingly.", "rebuttal": "We sincerely thank you for the detailed feedback and for recognizing the novelty of our work. We sincerely hope the following clarifications and new experimental results will address the outlined weaknesses (W1, W2, W3) and questions (Q1, Q2).\n\n---\n\n### **W1 & Q1: Statistical Significance**\nWe have conducted new experiments with 8 random seeds to ensure statistical significance. The new results show our method is not only superior in mean performance but also exhibits lower variance. We will update all experimental results in all the tables in the final version. (due to space limitations, we are currently able to include only a subset of the results)\n|SMON$_{m=2}$: Test|||||\n|-|-|-|-|-|\n|**Methods**|**Success**|**Progress**|**SPL**|**PPL**|\n|OracleEgoMap|63.40 (±4.82)|71.92 (±2.18)|48.73 (±3.12)|56.12 (±1.62)|\n|Lyon|75.08 (±4.24)|83.44 (±1.81)|63.14 (±2.46)|68.33 (±1.73)|\n|NeuRO (Ours)|81.14 (±3.86)|85.26 (±1.53)|66.49 (±2.75)|71.10 (±1.42)|\n\nThe new results confirm that NeuRO achieves a higher mean success rate and also exhibits significantly lower variance. This finding is critical, as it indicates that our framework produces more stable and reliable policies, which directly supports our central claim of enhanced robustness and generalization.\n\n---\n\n### **W2: Recent Baselines**\nWe thank the reviewer for suggesting recent baselines. To provide a more comprehensive comparison in the evolving landscape of MultiON, where few works report results, we have made our best effort to reproduce other methods for the MultiON task, as some official implementations for this benchmark are not available (* denotes our implementation following the methodology outlined in the corresponding paper; due to space limitations, we are currently able to include only a subset of the results). As shown, NeuRO continues to outperform even this strong, transformer-based baseline. We will add this comparison to Table 1 in the revised manuscript.\n|SMON$_{m=2}$: Test|||||\n|-|-|-|-|-|\n|**Methods**|**Success**|**Progress**|**SPL**|**PPL**|\n|SAM* [1]|70.02 (±4.28)|73.24 (±2.14)|54.18 (±3.06)|61.50 (±2.21)|\n|PoliFormer* [2]|76.84 (±3.60)|82.87 (±1.46)|62.50 (±2.33)|67.84 (±2.03)|\n|Lyon|75.08 (±4.24)|83.44 (±1.81)|63.14 (±2.46)|68.33 (±1.73)|\n|HTP-GCN*|76.20 (±4.48)|84.92 (±1.77)|60.94 (±2.38)|67.08 (±1.60)|\n|ImplicitNav [3]|78.41 (±3.20)|82.10 (±1.36)|65.66 (±2.29)|68.72 (±1.32)|\n|NeuRO (Ours)|81.14 (±3.86)|85.26 (±1.53)|66.49 (±2.75)|71.10 (±1.42)|\n\n---\n\n### **W3: Single-Object Navigation**\nWe greatly appreciate this insightful suggestion. We have included our results on single-object navigation in Appendix H. We have now incorporated a broader range of navigation agents into the single-object navigation task in MultiON settings (# denotes the agent specialized for single-object tasks). The results are shown below. (due to space limitations, we only show some of the results). These results suggest that agents trained for multi-object navigation can effectively generalize to standard single-goal scenarios, highlighting NeuRO’s strong generalization. Further results about this point are included in Appendix Table 8.\n|Methods|Success|SPL|\n|-|-|-|\n|BEVBert # [4]|78.45 (±3.27)|64.82 (±3.81)|\n|OracleEgoMap|82.55 (±4.16)|63.42 (±2.28)|\n|Lyon|86.27 (±2.40)|67.03 (±2.61)|\n|PoliFormer# [2]|87.41 (±1.78)|66.57 (±2.51)|\n|NeuRO (Ours)|90.03 (±2.12)|78.50 (±1.84)|\n\n---\n\n### **Q2: HTP-GCN Results**\nWe apologize for this omission. The results were not included initially as the official code was unavailable. We tried reproduce it based on the official paper and now completed HTP-GCN’s results (See W2).\n\n-----\n\n[1] N. Gireesh, et al., Sequence-Agnostic Multi-Object Navigation, ArXiv, 2023\n\n[2] KH. Zheng, et al., PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators, CORL, 2024\n\n[3] P. Marza, et al., Multi-Object Navigation with dynamically learned neural implicit representations, ICCV, 2023\n\n[4] D. An, et al., BEVBert: Multimodal Map Pre-training for Language-guided Navigation, ICCV, 2023"}, {"review_id": "kL8JZVCw6G", "reviewer": "Reviewer_aSLL", "summary": "The paper proposes NEURO, a hybrid learning + optimization framework for Multi-Object Navigation (MultiON). A visual-perception module first encodes RGB-D observations, egocentric maps and goal embeddings into features. These features are fed to Partially Input Convex Neural Networks (PICNNs), which output calibrated convex uncertainty sets over object-transition probabilities via a conformal-prediction procedure. \nA downstream robust pursuit–evasion optimization problem then plans an action trajectory that maximizes discounted capture belief while respecting motion constraints. The optimization result is blended with the network’s own policy and its objective value is used as an auxiliary reward, enabling end-to-end training by differentiating through the Karush-Kuhn-Tucker conditions.\n\nExperiments on both sequential (S-MON) and unordered (U-MON) MultiON benchmarks show that NEURO outperforms prior methods on unseen environments and converges faster during training. \nAblations illustrate the effect of action-blending weight, grid/time-horizon scale, and conformal coverage.", "questions": "a. Real-robot feasibility – have you profiled NEURO to demonstrate closed-loop runs on an actual mobile robot?\n\nb. Sensitivity to PICNN design – how do calibration quality and task performance vary with PICNN capacity or regularization?", "limitations": "See above\n\n--------------\nReview Update: Concerns Addressed - score increased to 5", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths :\n\na. Conceptual novelty – first task-based, uncertainty-aware optimization layer tightly integrated with visual navigation; extends differentiable‐optimization literature to long-horizon POMDPs.\n\nb. Principled uncertainty handling – PICNN + conformal calibration guarantees (marginal) coverage of true dynamics, keeping the downstream RO convex and tractable.\n\nc. Empirical gains & efficiency – consistent SOTA on S-MON/U-MON and faster convergence with modest inference overhead\n\nd. Careful ablations – studies on \\lambda, grid size, coverage level, and task-specific vs. generic formulations clarify where gains come from.\n\nWeaknesses:\n\na. Convex approximation limits expressiveness; many real object-movement distributions are non-convex → possible sub-optimal plans. Authors should aim at quantifying this.\n\nb. Evaluation confined to simulation; no physical-robot or real-world visual distractors, lighting changes, sensor noise, adversarial objects. Which is very important in these robot papers.\n\nc. Scalability of RO – solving large grids (E ≥ 5, τ ≥ 6) already multiplies inference time; How does the method extend to very large houses or outdoor scenes uncertain.\n\nd. **Important:** The writing is unclear; the authors assume considerable prior knowledge, which makes the paper difficult for non-expert readers to follow.\n\nOverall, the strengths outweigh the weaknesses. While I would like to see the shortcomings addressed, I remain inclined to recommend acceptance of the work.", "was_revised": true, "final_justification": "The authors provided clarifications and addressed my concerns. I am raising my score.", "rebuttal": "We sincerely thank you for the thoughtful and encouraging feedback. We hope that the following clarifications will fully address the noted weaknesses (W1, W2, W3, W4) and questions (Q1, Q2), further solidifying the contributions of our work.\n\n-------\n\n### **W1: Convex Approximation**\nTo address this question, while the overall navigation task is inherently non-convex and complex, we conducted the following experiment to quantify this. We first extracted a subset of SMON$_{m=2}$ task (both objects belong to the same category, e.g., “two cylinder”), and measured the navigation performance & prediction accuracy (following the method in Section 4.3 “Conformal Coverage Level\") under two settings: \n1. The 2 target objects are placed in separate rooms, which means the location belief distribution of each room’s object is approximately a convex uncertainty; \n2. Both similar objects are in the same room, resulting in a mutually interfering, non-convex posterior. \n||Two Room| |Single Room| |\n|-|-|-|-|-|\n| |Avg. Sucess|Pred. Acc.|Avg. Sucess|Pred. Acc.|\n|Lyon|72.80|-|79.42|-|\n|NeuRO (α=0.05)|77.84|0.88 (±0.019)|82.25|0.83 (±0.022)|\n\nWe observe the navigation performance in the first setting is lower, since finding objects in separate rooms is harder. However, NeuRO narrows this gap as it is specifically designed for convex problems. Also, while its prediction accuracy slightly declines in the second, non-convex task, NeuRO still outperforms SoTA baselines, indicating that non-convexity does not compromise its effectiveness and demonstrating its robustness.\n\n***Moreover, we would like to clarify a technical point discussed in our original manuscript:***\n\nIn our appendix, we present a basis function expansion method that not only addresses scalability but also, as we found, provides a principled path for tackling non-convex problems. We would first like to first clarify that non-convex problems (e.g., visual navigation), being NP-hard, are generally intractable and lack theoretical guarantees for their solutions. Our proposed method in Appendix F serves as a way to alleviate this issue. Specifically, we replaces the belief representation layer of the optimization model with basis functions, leveraging their power and flexibility to approximate non-convex belief variables ($\\alpha$ and $\\beta$). Our experimental results validate this substitution, showing a moderate improvement in the agent's performance in vision-based navigation, an environment rich with non-convex inputs. This, of course, introduces additional training costs and data requirements from the extra network module. Furthermore, to this end, we can treat matrix $M$ as a given constraint, avoiding PICNN-based convex calibration by leveraging its relation with alpha-beta in Eq. (2b) or using general black-box networks (e.g., predicting uncertainty bounds as in our experiments).\n\nIn summary, we provides two complementary approaches: (i) the PICNN-based convex approximation; and (ii) the basis function expansion to fit more complex variables. While the second approach can be extended to non-convex scenarios, it incurs extra training costs, and solutions to non-convex problems generally lack theoretical guarantees. Given these considerations, we believe the first approach is more suitable for the data-scarce environments central to our paper's theme. Therefore, we chose to feature method (i) in the main text.\n\n-------\n\n### **W2 & Q1: Real-World Scalability**\nTo address the concern about \"visual distractors\" and \"sensor noise,\" we would like to first clarify that our experiments are conducted in the Habitat simulator, which features rich, photorealistic textures and varied lighting conditions, providing a strong baseline for real-world visual complexity. Furthermore, we conducted a new ablation study. We injected Dropout noise into the agent's RGB input during testing to simulate real-world sensor imperfections. The results show that NeuRO's performance degrades far more gracefully than the baseline.\n|SMON$_{m=2}$: Success|||\n|-|-|-|\n| |No Noise|Dropout Noise|\n|OracleEgoMap|63.40 (±4.82)|56.27 (±7.33)|\n|NeuRO|81.14 (±3.86)|78.25 (±5.24)|\n\nBeyond robustness to visual noise, we highlight three key aspects of our framework that support its real-robot feasibility:\n1. **Memory Efficiency**: NeuRO's memory consumption (20.24MB, largely consisting of the perception module) is less than other SoTA methods due to its parameter-free component of optimization model, thus not introducing prohibitive memory overhead (see the table in W3). \n2. **Runtime Performance**: See **W3**.\n3. **Non-convex Adaptability**: See **W1**.\nCollectively, these points demonstrate that our framework is not only effective but also practical and extensible for real-world applications.\n\n-------\n\n### **W3: Inference Time**\nWe thank the reviewer for this crucial clarifying question. To comprehensively address the reviewer’s concern, we offer the following clarifications: \n1. **Absolute Time**: The values in Table 3 are normalized for comparison (relative to $E=3, \\tau=4$; see line 523-524 in Appendix). The absolute solving times are presented in Appendix F (Table 5) and are modest (e.g., 1.084s for a large $E=20, \\tau=12$ setting). We will incorporate this clarification from the Appendix into the main paper. \n2. **Runtime Performance**: We attribute the primary computational cost to the gradient computation of the optimization model but not the solving process, because modern linear program (LP) solvers are already highly efficient for solving. To validate this analysis, we measured the runtime cost at test stage, where the agent only needs to solve the forward pass of the optimization problem to get the action offset without any gradient computation. This result confirms that NeuRO is fully capable of real-time performance. We will add this crucial table and clarification to the revised version.\n    | |Memory Consumption |Test Runtime (SMON$_{m=2}$; per step (ms))|\n    |-|-|-|\n    |Lyon|22.18MB|12.9|\n    |HTP-GCN|25.63MB|15.3|\n    |NeuRO|20.24MB|11.6|\n3. **Local Grid Scope**: The optimization time is manageable since our framework does not require a large optimization grid. This is because our optimization grid models the agent's local egocentric view, NOT the global map (Evidence: 1. The grid is re-formulated at every time step based on the current egocentric observation; 2. We always initialize agent position to the center of the optimization grid). This keeps the optimization problem small and tractable. In the MultiON task, the input ego-map spans a 5m view and each cell in the map represents a 0.8m*0.8m area, which means there are about 30 cells. Our generation process of the optimization grid can be understood as overlaying a goal position distribution onto this local map. Therefore, a grid size of $E=5$ ($V=25$ cells) is sufficient. In summary, our optimization grid simulates a local field of view, so the problem size E remains bounded regardless of the overall environment scale.\n4. **Scalability**: Furthermore, we have analyzed the source of the optimization problem’s computational complexity in Appendix F, which scales with $O(E^2\\tau)$. To address this, we introduce a basis function expansion method in Appendix F to let network predict α and β. It effectively reduces this complexity to $O(K\\tau)$, where $K \\ll E^2$ is a constant. We empirically validate this approach, demonstrating its effectiveness in accelerating the optimization. However, for the sake of clarity and to maintain focus on the “network+optimization” framework itself in the main paper, we have detailed this extension in the appendix.\n\n-------\n\n### **W4: Writing Clarity**\nWe sincerely appreciate this feedback. We acknowledge that the manuscript currently assumes a level of prior familiarity, and we will make the necessary adjustments. For example, \n1. **Add Justifications for Key Design Choices**: We will provide clear, intuitive explanations for our core modeling choices (E.g., we will explicitly clarify why we use a discrete grid for the uncertainty set (to avoid intractable semi-infinite programs)).\n2. **Provide Essential Background**: We will introduce a concise background section covering prerequisite knowledge, such as the core ideas behind Conformal Prediction.\n3. **Refine Notation and Symbols**: We will thoroughly revise our notation and add a comprehensive notation table to the appendix for easy reference.\n\n-------\n\n### **Q2: PICNN Design**\nWe have conducted new experiments on PICNN capacity, alongside with our existing ablation on the coverage level in Section 4.3, to provide a comprehensive sensitivity analysis. We analyze the sensitivity of our framework to the two primary design factors of the PICNN module—the conformal coverage level and network capacity—by measuring their impact on both prediction accuracy and final task performance. The results show that a larger PICNN improves both grid prediction accuracy and task performance, but with diminishing returns. We attribute this to the challenges of training a high-capacity model in a data-scarce regime.\n1. **Coverage Level $(1-\\alpha)$**: As shown in Section 4.3 (Table 2), task performance is directly and intuitively linked to calibration quality. A higher coverage level leads to more conservative but reliable uncertainty sets, improving performance, which aligns perfectly with our design motivation.\n2. **Network Capacity**: We have now conducted a new experiment to analyze the impact of PICNN capacity (dimension of hidden layers).\n|SMON$_{m=2}$||||\n|-|-|-|-|\n|Hidden Dim.|Avg. Success|Avg. SPL|Prediction Accuracy|\n|128|77.3|62.1|0.84 (±0.021)|\n|256|80.1|66.5|0.90 (±0.015)|\n|512|82.4|67.1|0.88 (±0.013)|"}, {"review_id": "CbkQKwHaFr", "reviewer": "Reviewer_QiSo", "summary": "This paper proposes NEURO, a hybrid framework that integrates deep neural perception with downstream robust optimization for visual navigation tasks in partially observable, data-scarce environments. The method leverages Partially Input Convex Neural Networks (PICNNs) to construct convex uncertainty sets, which are then used in a robust optimization formulation to guide planning and action selection. NEURO enables bidirectional interaction between learning and planning modules by incorporating optimization-derived feedback into policy learning. Experiments on MultiON tasks (both unordered and sequential) demonstrate improved performance and generalization over purely neural or heuristic baselines, with additional ablation studies validating key design choices.", "questions": "As outlined in the Weaknesses, my main concerns center around missing runtime/failure analysis and the sensitivity to belief estimation. Addressing these with additional experiments or clarifications would help resolve key doubts and could improve my overall assessment.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Innovative hybrid formulation: The integration of neural predictions with a robust optimization planner is novel and well-motivated, offering a principled way to address uncertainty in partially observable navigation tasks.\n2. Strong empirical results: NEURO consistently outperforms baselines on MultiON benchmarks under different target counts, especially in generalization to unseen environments.\n3. Comprehensive ablations: The paper includes detailed ablation studies on the action blending factor, spatial-temporal optimization scale, and uncertainty coverage level, demonstrating careful empirical analysis.\n4. Solid mathematical derivation: The core optimization formulation, uncertainty calibration, and dual problem transformation are all mathematically well-grounded and derived in detail.\n\nWeaknesses\n1. The paper lacks runtime and computational cost analysis, raising concerns about the feasibility of real-world robotic execution, especially given the optimization overhead.\n2. The paper does not include any analysis or discussion of failure cases.\n3. The comparison includes only one method from 2024; incorporating more recent baselines would strengthen the empirical validation.\n4. The optimization module depends on accurate belief estimation via predicted object transition matrices, which are not directly supervised. The paper does not analyze how sensitive the method is to inaccurate belief estimation, which is critical for robust planning in partially observable environments.\n5. The convexity assumption imposed by PICNNs may be overly idealized, as real-world visual uncertainty is often non-convex or multimodal. While the authors acknowledge this limitation in the conclusion, its practical impact (e.g., performance degradation in non-convex scenarios) is not empirically analyzed.\n6. The theoretical coverage guarantee for the uncertainty set (Proposition 1) relies on the i.i.d. assumption for calibration data, which may not hold in embodied navigation settings with strong temporal correlations between observations.", "was_revised": true, "final_justification": "The author's rebuttal has resolved my concerns about Convexity Assumption, and I raise my score from Borderline Reject to Borderline accept.", "rebuttal": "We sincerely thank you for the detailed and expert feedback. We are encouraged that you found our work innovative and our mathematical derivations solid. We hope that the following clarifications can address the noted weaknesses (W1, W2, W3, W4, W5, W6), further solidifying the contributions of our work.\n\n-----\n\n### **W1: Computational Cost Analysis**\nWe address this crucial point by analyzing memory footprint and runtime performance, which proves NeuRO’s efficiency for real-world deployment.\n1. **Memory Consumption**: NeuRO is lightweight. Its checkpoint size (20.24MB) is smaller than other SoTA methods due to its parameter-free optimization component.\n2. **Runtime Performance**: We demonstrate with the following points:\n- ***2.1 Absolute Time***: The values in Table 3 are normalized for comparison (relative to $E=3, \\tau=4$; see line 523-524 in Appendix). The absolute solving times are presented in Appendix F (Table 5) and are modest (e.g., 1.084s for a large $E=20, \\tau=12$ setting). We will incorporate this clarification from the Appendix into the main paper.  \n- ***2.2 Runtime Performance***: We attribute the primary computational cost to the gradient computation of the optimization model but not the solving process, because modern linear program solvers are already highly efficient for solving. To validate this analysis, we measured the runtime cost at test stage, where the agent only needs to solve the forward pass. This result confirms that NeuRO is fully capable of real-time performance. We will add this crucial table and clarification to the revised version.\n    |Memory Footprint|Test Runtime (SMON$_{m=2}$; per step (ms))|\n    |-|-|\n    |Lyon|22.18MB|12.9|\n    |HTP-GCN|25.63MB|15.3|\n    |NeuRO|20.24MB|11.6|\n- ***2.3 Local Grid Scope***: The optimization time is manageable since our framework does not require a large optimization grid. This is because our optimization grid models the agent's local egocentric view, NOT the global map (Evidence: e.g., The grid is re-formulated at every time step based on the current egocentric observation). This keeps the optimization problem small and tractable. In MultiON, the ego-map spans 5m with ~30 cells (0.8m × 0.8m resolution). Our generation process of the optimization grid can be understood as overlaying a goal position distribution onto this local map. Therefore, a grid size of $E=5$ ($V=25$ cells) is sufficient. In summary, our optimization grid simulates a local field of view, so the problem size $E$ remains bounded regardless of the overall environment scale.\n- ***2.4. Scalability***: To reduce the $O(E^2\\tau)$ complexity (the source of the optimization problem’s computational complexity) analyzed in Appendix F, we introduce a basis function expansion that enables the network to predict $\\alpha$ and $\\beta$ directly, reducing the complexity to $O(K\\tau)$ with a predefined constant K ≪ E². Empirical results confirm its effectiveness.\n\n----\n\n### **W2:  Lack of Failure Case Analysis**\nWe agree this is an important analysis, and we have now systematically studied our method's robustness from two perspectives:\n1. **Target Geometries (Appendix H.1)**: We analyzed performance on targets with less distinct geometries (e.g., short cylinders, ambiguous spheres). The results show that while all methods' performance degrades, NEURO's performance drops far more gracefully, demonstrating superior robustness to perceptually challenging objects.\n2. **Visually Similar Distractors**: We analyzed failure cases in S-MON (m=3) and found 62% were due to perceptual ambiguity (e.g., confusing a target sphere with a pan). This issue is fundamental, as the failure rate for this reason is comparable across methods (Baseline: ~66%; Lyon: ~60%). Notably, our core framework improves resilience to such distractors over the baseline, even without the specialized perception modules used by methods like Lyon.\n\n---\n\n### **W3: Limited Comparison**\nWe thank you for suggesting recent baselines. As few works report MultiON results and official implementations are often unavailable, we reproduced several methods based on their original papers (* indicates such methods). A subset of results is shown below due to space limits. As shown, NeuRO continues to outperform even this strong, transformer-based baseline. We will add this comparison to Table 1 in the revised manuscript.\n|SMON$_{m=2}$: Test|||||\n|-|-|-|-|-|\n|**Methods**|**Success**|**Progress**|**SPL**|**PPL**|\n|SAM* [1]|70.02 (±4.28)|73.24 (±2.14)|54.18 (±3.06)|61.50 (±2.21)|\n|PoliFormer* [2]|76.84 (±3.60)|82.87 (±1.46)|62.50 (±2.33)|67.84 (±2.03)|\n|HTP-GCN*|76.20 (±4.48)|84.92 (±1.77)|60.94 (±2.38)|67.08 (±1.60)|\n|ImplicitNav [3]|78.41 (±3.20)|82.10 (±1.36)|65.66 (±2.29)|68.72 (±1.32)|\n|NeuRO (Ours)|81.14 (±3.86)|85.26 (±1.53)|66.49 (±2.75)|71.10 (±1.42)|\n\n----\n### **W4: Sensitivity to Inaccurate Belief Estimation**\nWe demonstrate NEURO's resilience to inaccurate belief estimation through both its core mechanism and a new experiment.\n1. **Robust Calibration Mechanism**: Our framework is fundamentally designed to handle estimation inaccuracies. The conformal calibration procedure (Proposition 1) ensures that the generated uncertainty set $\\Omega(x)$ covers the true object dynamics with a prespecified probability. As shown in Table 2, this calibrated, conservative belief representation can filter out noise and make accurate predictions (about object location) under uncertainties. This mechanism is our framework's primary defense against the inevitable inaccuracies of belief estimation.\n2. **Robustness to Noisy Inputs**: To further validate this resilience, we conducted a new ablation study. We injected Dropout noise into the agent's RGB input to interfere belief estimation. The results show that NeuRO's performance degrades far more gracefully than the baseline:\n|SMON$_{m=2}$: Success|||\n|-|-|-|\n| |No Noise|Dropout Noise|\n|OracleEgoMap|63.40 (±4.82)|56.27 (±7.33)|\n|NeuRO|81.14 (±3.86)|78.25 (±5.24)|\n\n---\n### **W5: Convexity Assumption**\nTo address this question, while the navigation task is inherently non-convex and complex, we conducted the following experiment in response to quantify the practical impact. We extracted a subset of SMON$_{m=2}$ where both objects are of the same category (e.g., two cylinders), and measured the navigation performance & prediction accuracy (following the method in Section 4.3) under two settings:\n1. The 2 objects are placed in separate rooms, where each’s location belief distribution is approximately a convex uncertainty; \n2. Both objects are in the same room, resulting in a mutually interfering, non-convex posterior. \n\nWhile the first task is harder due to room traversal, NeuRO narrows the performance gap as it is specifically designed for convex problems. Also, while its prediction accuracy slightly declines in the second, non-convex task, NeuRO still outperforms SoTA baselines, indicating that non-convexity does not compromise its effectiveness and demonstrating its robustness.\n||Two Room| |Single Room| |\n|-|-|-|-|-|\n| |Avg. Sucess|Pred. Acc.|Avg. Sucess|Pred. Acc.|\n|Lyon|72.80|-|79.42|-|\n|NeuRO (α=0.05)|77.84|0.88 (±0.019)|82.25|0.83 (±0.022)|\n\n***Moreover, we would like to clarify a technical point discussed in our original manuscript:***\n\nIn our appendix, we present a basis function expansion method that not only addresses scalability but also, as we found, provides a principled path for tackling non-convex problems. We would first like to first clarify that non-convex problems (e.g., visual navigation) are generally intractable. Our proposed method in Appendix F serves as a way to alleviate this issue. Specifically, we replaces the belief representation layer of the optimization model with basis functions, leveraging their flexibility to approximate non-convex belief variables ($\\alpha$ and $\\beta$). Our experimental results validate this substitution, showing a moderate improvement in the agent's performance in vision-based navigation, an environment rich with non-convex inputs. To this end, we can also treat matrix $M$ as a given constraint, avoiding PICNN-based convex calibration by leveraging its relation with alpha-beta in Eq. (2b) or using general black-box networks (e.g., predicting uncertainty bounds as in our experiments).\n\nIn summary, we provides two approaches: (i) the PICNN-based convex approximation; and (ii) the basis function expansion to fit more complex variables. While the (ii) can be extended to non-convex scenarios, it incurs extra training costs, and solutions to non-convex problems generally lack theoretical guarantees. Thus, approach (i) is more suitable for the data-scarce environments central to our paper's theme.\n\n---\n### **W6: i.i.d. Assumption**\nWe sincerely apologize for the lack of clarity that led to this misunderstanding. We want to clarify that the i.i.d. assumption applies to data sampled for a given time step t, not across temporally correlated steps within an episode, with following two reasons:\n\n***Firstly***, this is supported by our formulation and is not violated by the nature of embodied navigation. First, there is a minor typo on Line 162-163; we intended to state that we omit the time index $t$ for conciseness, not the object index $i$. This is consistent with Eq. (5), where index $t$ is omitted.\n\n***Secondly***, because the optimization grid is re-formulated at every time step $t$, the conformal quantile q must also be determined based on data relevant to that specific time step. This means the calibration procedure relies on i.i.d. samples drawn for the same, given time $t$\n\n---\n\n[1] N. Gireesh, et al., Sequence-Agnostic Multi-Object Navigation, ArXiv, 2023\n\n[2] KH. Zheng, et al., PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators, CORL, 2024\n\n[3] P. Marza, et al., Multi-Object Navigation with dynamically learned neural implicit representations, ICCV, 2023"}]

Author rebuttal:

Meta-review: The submissions deals with visual navigation and data scarcity and proposes to add a task specific optimization routine, which deals with uncertainty in a dedicated way. All four reviews were quite appreciative and positive. The expert reviewers appreciated novelty (a fresh new perspective), sound methodology, the clear advantage of being able to handle uncertainty, performance on the Multi-ON task, comparisons with a large number of baselines, available code.

Some weaknesses have been pointed out: missing standard deviations, some limitations (grid; scalability; convexity), missing experiments with real robots, missing failure cases, and writing.

The rebuttal provided by the authors was helpful and could clear up many concerns.
There was a clear consensus on acceptance and the AC concurs.
Decision: Accept (poster)