Imitation Beyond Expectation Using Pluralistic Stochastic Dominance
Authors: Ali Farajzadeh
Danyal Saeed
Syed M Abbas
Rushit N. Shah
Aadirupa Saha
Brian D Ziebart
Venue: NeurIPS 2025 spotlight ()
Keywords: imitation learning
Primary area: reinforcement_learning
TL;DR: This paper develops stochastic dominance as an objective for imitation learning to provide stronger guarantees for demonstrators with differing preferences.

Abstract: Imitation learning seeks policies reflecting the values of demonstrated behaviors. Prevalent approaches learn to match or exceed the demonstrator's performance in expectation without knowing the demonstrator’s reward function. Unfortunately, this does not induce pluralistic imitators that learn to support qualitatively distinct demonstrations. We reformulate imitation learning using stochastic dominance over the demonstrations' reward distribution across a range of reward functions as our foundational aim. Our approach matches imitator policy samples (or support) with demonstrations using optimal transport theory to define an imitation learning objective over trajectory pairs. We demonstrate the benefits of pluralistic stochastic dominance (PSD) for imitation in both theory and practice.

Imitation Beyond Expectation Using Pluralistic Stochastic Dominance
Ali Farajzadeh, Danyal Saeed, Syed M. Abbas, Rushit Shah, Aadirupa Saha, Brian D. Ziebart
Department of Computer Science
University of Illinois Chicago
Chicago, II. 60607
fafaraj5, dsaaeed3 , sab.aaaa,, aawt:haaahhhaaaaauuaaaaa a bziebart)
Abstract
Imitation learning seeks to estimate policies reflecting the values of demonstrated behaviors. Prevalent approaches learn match exceed the demonstrator's performance in expectation without knowing the demonstrator's reward function. Unfortunately, this does not induce pluralistic imitators that learn support distinct demonstrations. We reformulate imitation learning using stochastic dominance over the demonstrations' reward distribution across a range of reward functions our foundational aim. Our approach matches imitator policy samples (or support) with demonstrations using optimal transport theory define an imitation learning objective over trajectory pairs. We demonstrate the benefits of pluralistic stochastic dominance (PSD) for imitation in both theory and practice.
Introduction
When learning from demonstrations, behaviors reflecting individ- preferences and capabilities are often demonstrated. Existing imitation learning methods struggle to preserve these distinct be- haviors while trying to improve beyond them. For example, inverse
Pareto
Frontier
50% Stochastic
Dominance
reinforcement learning (Abbeel & Ng. 2004 Ziebart 2010) and discriminative imitation (Ratliff al. 2006] Ho Ermon 2016)
Expected Dominance
methods seek to match or outperform (Syed & Schapire (007) the demonstrations under a range of reward functions in expectation. As shown in Figure [ this can be achieved by an imitator that never produces behavior that a demonstrator prefers over any of his or her more preferable demonstrations.
50%
Demon-
strations
We seek stronger distributional guarantee of pluralistic stochastic dominance (PSD(" which ensures the imitator higher probability of achieving any level of reward than the demonstration distribution (i.e., stochastic dominance) for all reward functions (i.e., pluralism) within some defined set. This requires the imitator to match or improve upon the distinct properties of exceptional demonstrations rather than focusing on the average of demonstrations randomizing between different modes (Figure I These guarantees support more complex applications of imitator policies (e.g., sam- pling many candidate trajectories and selecting the best) beyond the assumption that a single imitator trajectory is sampled and executed.
Figure 1: Dominance in pectation (green) guarantees better performance than the demonstration average for all conical sum reward functions. Pluralistic stochastic domi- nance (red) makes the distribu- tion of rewards preferable by guarantecing a higher proba- bility of achieving any reward.
by
Cooe available at
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Our approach is based on the observation that if all demonstrations can be matched to supported imitator behaviors that are better than the paired demonstration for all reward functions (Figure TT) then PSD is achieved (Armbruster & I Luedtke, 2015). We employ optimal transport theory (Rioux (et al., 2024) to perform this matching and margin-based upper bound (Ziebart et al.] 2022) to behavior pair. We then optimize the imitator policy using the matched pairs in the fully realizable and model-based imitator policy settings. We provide stochastic dominance generalization guarantees to establish the theoretical benefits of pursuing pluralistic stochastic dominance, and experimentally demonstrate better support for distinct behavior that improves beyond the paired demonstrations.
define a loss over the demonstration-iminn
The main contributions of this paper are three-fold: First, PSD improves upon imitation learning methods designed for specific sisk-sensitivities (Majumdar et al., 2017, Singh et al. 2018)) Santara et al. 2018 Lacotte et al. 2019) to provide guarantees simultaneously across all reasonable risk- sensitive performance measures (e.g., value-at-risk,, conditional value-at-risk, and range value-at-risk, detailed in Appendix [A.]]. These performance guarantees provide broader support for more complex use cases of imitation learning, such as selecting the best trajectory from set of samples. Second, PSD-based imitation provides an alternative justification to maximum entropy inverse reinforcement learning for stochastic imitation policies. Finally, we introduce novel evaluation metrics for imitation learning based on stochastic and Pareto dominance that can be applied settings with known reward bases, but not singular motivating reward function.
2 Background and related work
2.1 Inverse reinforcement learning
In imitation learning settings, reward functions defining desirable behavior are unknown. Instead, foundational formulations assume that reward features are available that define linear (Ng & Russell 2000 Abbeel & Ng. 2004) or conical sum (Syed & Schapire 2007) reward function.
Definition 2.1. Given reward features f: E + R for behavior trajectories & E E, the family of
feature-based reward functions is defined as re (E) = 0 f(E) with parameters 0 E R (linear) or
0 E 12 (conical sum).
An imitation policy that matches the feature moments of the demonstrator guarantees equal rewards in expectation expectation-including the demonstrator's unknown (linear) reward function (Abbeel & Ng 2004)):
v0 R BE-P, [((E)] = 15 [(() = Ee-P. [ro(E) = Be (E)
(I)
where P, denotes the distribution over trajectories based on the interaction between the policy m and the dynamics of the decision process, which we assume are deterministic..
Many imitation learning approaches can be viewed as matching various moments (rewards, on-
policyloff-policy state-action value functions) of the demonstrator (Swamy et al.
2021). including:
behavior cloning (Pomerleau) 1988); maximum margin planning (Ratliff al.]
2006); maximum
entropy inverse reinforcement learning (Ziebart 2010): DAGGER (Ross et al.
2011) generative
adversarial imitation learning (Ho & Ermon] 2016); and Value Dice (Kostrikov et al.1).
Entropy regularization methods for reinforcement learning (Neu et al.
2017 also known as softmax
decision policies (Sutton & Barto 2nnceeiee the diversity of the imitator's trajectories within these moment moment-matching techniques. However, these provide robust predictive guarantees for imitation learning (Ziebart) 2010) rather than performance guarantees for diverse demonstrators. Extensions of these methods attempt to model variations in preferences or quality of demonstrations with latent variables. These are then used condition policy models (e.g.. as mixture models) or focus imitation on more desirable demonstrations (Brown et al.) 22020b) Chen et al., 202T] Wu et al., 2019 Zhang jet al. (02T). We aim to avoid the computational challenges (e.g., difficult nonconvex optimizations) and/or strong assumptions underlying these approaches.
2.2 Outperformance and subdominance minimization
Our approach is closer in motivation to methods designed to outperform demonstrators. Early methods focus policies that outperform in terms of expected rewards (Definition 2.2)

---

Definition 2.2. Policy X1 has expected dominance over #2 if the expected trajectory reward under Ti least much as the expected trajectory reward under TT: Bei T1 (Ei)) Be (Fo for fixed 0.
MWAL (Syed Schapire 2007 and LPAL (Syed et al. 2008) guarantee outperforning the demon- strator in expectation under the assumption that the signs of the reward function weights are known (i.e., conical reward functions of Definition 2. guarantee better expected rewards:
In this setting, better expected reward features
40 E RRZ Be-P, ((()) = R
r(3)
Ee- ro((s] 2 Be-rs [o(tt]
(2)
Subdominance minimization (Ziebart al., 2022) extends this idea of outperformance by seeking uniform dominance (Definition 2.3) across conical sum reward functions by minimizing a convex bound over the probability of violating uniform dominance.
Definition 2.3. Policy и has uniform dominance over m2 if all trajectory samples from कı have at least much reward all samples from n2! Fe (ro(E) ro(E))) = 1 for fixed A.
Unfortunately, uniform dominance encourages deterministic policies that are similar to the expected dominance policy in settings like Figure
We pursue less strict notion of dominance in this paper: stochastic dominance (Definition 2.4 It is
based on having a better distribution of rewards.
Definition 2.4. Policy J has stochastic dominance over T2 if n has least much probability of exceeding any reward threshold: Vc E IP (re(Ei) 2 e) 2 (reae))(roEE2) 2 e) for fixed 0.
In terms of strictness, uniform dominance implies stochastic dominance, which implies expected dominance. However, uniform dominance often infeasible (c.g.. Figure [1), while stochastic dominance is always feasible (e.g., π #). In addition to expected reward bounds stochastic dominance guarantees broad risk measure improvements (Ogryczak & Ruszezynski 1999). We summarize some of these in Theorem 2.5
Theorem 2.5. Stochastic dominance of (o(T) 2 (T) for some fixed guarantees improved expected and risk-sensitive rewards for the imitator & ~ Px with respect to the demonstrator & Pa: A (c E [0, 1), E (c, 1)), [gwa(ro(EE) 2 Re (ro(E)), (ro).)ro(E)) and RVaRe,.(ro(E)) 2 RVaRe,.(ro(EE).
(rox(r (raa(ro(E)) 2 VaRa(ro(E)), CVaR,(ro(E)) 2
Prior imitation learning research employs risk sensitivity narrowly to address safety concerns by targeting specific tail risks (and specific quantile levels). Extensions of generative-aaversariall imita- tion learning (GAIL) (Ho & Ermon 2016) match specific demonstrator senstiiisti (Majundar al. 2017 Santara al. 2018; Lacotte al., 2019). Bayesian estimation methods (Brown et al. 2020a Javed et al. 2021) incorporate risk sensitivity to more robustly address uncertainty during contrast, we consider risk-sensitivity exhaustively- across family of reward functions and over all sensitivity thresholds, as guaranteed by stochastic dominance (Theorem to incentivize high-quality coverage of diverse demonstrations.
reward function estimation,
2.5
2.3
Optimal transport
Optimal transport theory considers the minimum cost transforming from one distribution, Px, to another Py under cost function e X' y Rzo. The Kantorovich (192 formulation defines this transformation using a joint probability measure 7 E Axxy with (z, y) representing the amount of probability mapped from to y and marginals that match the source and target distributions. For discrete distributions (with Px and Py supporting m and n values, respectively), this can be expressed as a linear program:
OT, (Px, min
Σ nh (. s.t. Vj. Σ Ti.j = Py Vi, Σ Mi = Px(xx; (3)
720
The optimization is solved exactly by the Hungarian algorithm in O ( time with error tolerance using specialized algorithms (Dvurechensky et al. 2018 in Ò (max((x)) /e2) time.
The optimal transport objective (commonly referred to as the Wasserstein distance for metrie costs) has been popularized as an alternative to the Jensen-Shanme divergence in gencrative-adveesania

---

learning (Arjovsky et al. 2017). Previous investigations for imitation learning tasks (iaao et al. 2019
Dadashi et al. 2021). include cross-domain imitation transfer (Nguyen et al., 2021 Fickinger et al. 2022), combining trajectory matching with behavioral cloning (Haldar et al. (023), and matching
reward-less trajectories with expert trajectories (Luo et al. 2023 Fach of these are distinct from our
approach and motivation.
We build upon key relationship between optimal transport and stochastic dominance in this work: Remark 2.6. IPy ž Px (Definition 2.4 if and only if there mapping from Px to Py that is non-decreasing in value: OT max(x-v, 0) x, IPy) = 0, where e(x, y) is positive only when x: > y.
3
Approach
3.1 Pluralistic stochastic dominance
We consider imitation learning with multiple demonstrators. Each has their own reward function, Ig. pre- sumed be from the family of conical sum cost functions (Definition 2.1 can
(a)
(
(c)
Pareto
2
Frontier
Hi
Imitator Samples
A!
deno
Feature moment methods guarantee that each demonstrator is least indifferent between the trajectory distributions of the demonstrators and the imitator in expectation (Swamy al. 202T). However, this does not guarantee any chance of produc- ing highly desirable behavior for any demonstrators. As consequence, if preferences are based on higher quan- tiles of reward distributions rather than expectations, the demonstration distribution can be highly preferable. We introduce pluralistic stochastic dominance (PSD) to ensure that the imitator policy is no less preferable to the demonstration distribution for all conical sum reward functions and all reward quantiles (Definition 3.T.
Demos
Matchings
[n]
(4)
demo
domo
f
C
Figure 2: (a) Features (f1, f2) of four demonstrations (black points) matched (yellow lines) with four imitator samples (red points); the cumulative features for fi (b) and f2 (c) for the sample distribution (P(4) and the full distribution (P(x=)). We seek to optimize the imitator policy using avail- able demonstrator/mmitaor samples to achieve full distribu- tion for the imitator that stochastically dominates the demon- strator. This can be verified for each by sorting the samples, e.g., for 0 [i] (b) and 0 [9] (c). However, this is in- tractable for continuous sets of 6; our upper bound instead employs a single matching (a).
Definition 3.1. Distribution Px provides (first-rder) pluralistic stochastic dominance over PN (for all conical sum reward functions and for margin function Bo), which we denote P. EPSD P=, iff:
R v(O E 20: c E RR, (ro((((o(E)) 2 c) 2 P:-== (ro(E) 2 c) + Ba-
Though exact replication (π = F) trivially achieves this, imitation learning often aims to be performant on withheld demonstrations, so exact replication of finite training demonstrations is not sufficient. Instead, imitator policies æ that achieve better reward distributions (and better generalization) are desired. Figure 2 a) provides an example of strict stochastic dominance.
We extend stochastic dominance to the pluralistic setting by taking the maximum of optimal transport problems (Remark 2.6) for each conical sum reward function (Definition 3.2p.
Definition 3.2. The pluralistic stochastic subdominance between imitator and demonstrator trajec- tory distributions (with margin function 3g) is given by the worst-case reward function:
or Pog (Fr.Pg)
max 1]
min 7
Not (ro(E)) (E)) + 80] s.t. Σ n.s (.(E)) Vi, Σ n. (=(())
(5)
Minimizing this entire set optimal transport problems to zero guarantees PSD (Theorem proof's of this theorem and others are provided in Appendix B
3.3 The

---

Theorem 3.3. Zero maximum optimal transport in Def. 3.2 (with _3 0) and pluralistic stochastic dominance are equivalent: maXge[0011
(IP (Ars):
IP EPSD Fa.
As result, PSD extends the risk- properties of stochastic dominance (Theorem 2.5 to the entire set of conical sum reward functions.
Corollary 3.4. PSD guarantees that exhibits better risk-sensitive performance than it under re for the set of measures in Theorem 2.3 and all conical sum reward functions, i.e., O.
While the inner minimization (over ?) in (5), OT (r(
(Px, Fx). is standard optimal transport linear program, the outer maximization (over 0) is of a convex function (of 0). This family of convex maximization programs is known be NP-hard (Raghavachari T969), suggesting computational
challenges for our specific instances, unfortunately (Dentcheva & Wolthagen 2015).
3.2 Matched subdominance minimization
Given the apparent computational challenges of exactly verifying pluralistic stochastic dominance (not only for ( but all 0 2 0), we instead derive a computationally efficient upper bound. We approach this by "pushing" the maximization of & deeper into the original PSD expression:
max DE &
min ac Σ, 26 ( Σ. E. Yj
ro(E)) - To (E(E) 0
1
sabdomn, (E(r)))))
This replaces set of optimal transport problems for each with one single optimal transport problem (Figure 2h) and makes numerous independent maximization problems that are easy to solve. Specifically, the resulting inner maximization is equivalent a particular instance of the subdominance (Ziebart et al. 2022) introduced for imitation learning via uniform dominance:
subdoma,a(E.C)
Σ ark (Ja(E) fl((t) + +
which measures how far trajectory is from Pareto-dominating & (by margin & with features weighted by o). We define our relaxed optimization problem as a linear program using the more general form of subdominance (Def. (.5).
Definition 3.5. Matched Subdominance Minimization given a and B is obtained from:
OT subdaam... (P Pa)
min Σis
7h "u.jubdom.,,aS..
(8)
00 Σ, P...) -.
Margins require strict improvement and avoid the trivial solution. The sets trajectories should be sufficiently large cover the distinct demonstrated behaviors. Given tens or hundreds of trajectories in each set, the optimal transport problem not critical computational bottleneck in practice.
As result of being an upper bound, stochastic dominance can be guaranteed (Theorem 3.6
Theorem 3.6. For any a > 0 and 3 2 O,
OT sabdom,, . (P P+)
P. PSD P=.
We note that this is special case of a recently established family of losses for which an optimal transport distance zero implies multivariate stochastic dominance (Rioux al.) 2024).
3.3 Policy learning algorithms
We consider two imitation learning settings: fully realizable with any distribution over trajectories possible to lear;; and policy model with parametric policy model, ma, optimized.
In the fully realizable setting, we consider a set of candidate trajectories, & (ideally from the Pareto frontier) and learn the imitator's s distribution over those trajectories (Def. 3.7
5

---

Definition 3.7. For given candidate set of trajectories, fixed weights, and distribution of demonstrations, the matched minimal subdominance imitator policy is obtained from:
min
y..subdom,,, (Ei, E5) + XReg(7) such that: Σ =a Vj, 1
(9)
where regularizer Reg(r) Ih or E, lhx, encourages more uniform assignments and imita- tion trajectory distributions, respectively. The imitator trajectory distribution is then obtained by marginalizing: ((,) = Σj %
Learned policy models enable generaliza- tion different tasks within the same envi- ronment or to other environments. We lever- age (deep) reinforcement learning methods (e.g., policy gradient optimization) using the subdominance- basse optimal transport solution to determine training signal for a policy model. This allows stochastically dominant policy optimization without first identifying a set of candidate trajectories (Def. 3.7). The model update procedure is described in Algorithm[] [
Algorithm Policy model update
Input: M imitator samples {Si), N demonstrations
(6)) policy/parameters Tæ. and learning rate 1)
Output: Updated policy/parameters "g
1: Set (5.) ,े
2: Solve OT suhdom given P=((s) and PP (E,) (Def. 3.2)
3: Construct training signals {a,} from OT solution
4: Update model parameters @ using variables a from
(10) or
+ Φ + 1= & log Pr,
Σ
Step 4 of the Algorithm parallels policy gradient methods (Williams 1992) with {ai} replacing other improvement signals. These are obtained from the OT matching, using demonstration normalization (IO) or weighted best match (I),, which emphasizes the best trajectories more:
Σ mi subdom(E::Es) Σ Ne. subdom(E:r,5)
(10)
3
a. = Σ Yis subdom(((.E))
argmin, subdom
5
Σ 7 subdom(Er,
(11)
Additionally, the a values of the subdominances for optimal transport 8 can either remain fixed, as implied by Algorithm [ or be simultaneously updated using stochastic gradient optimization.
3.4 Generalization analysis
We characterize distributional stochastic dominance guarantees for the population of demonstrations based on finite, IID training sample using the Dvvoetzky et al. (1966) inequality.
Theorem 3.8. Given the cumulative mass function (CMF) in the K K-dimensional reward feature space from shifting the empirical demonstration CMF (with N IID sampled trajectories): F E +f)
[F (f) - +] + ef(f oo), and its corresponding probability mass function, I2 (f), then:
OT subdoma. o (IP IP +
- P (Vf F (n) < P (f) 21 NK. 2N2 -NN2
This requires the convex hull of the Pareto frontier be supported by small number of points that each have at least imitator probability; and Pareto dominate least of P Cases in which F (f) assigns probability to "unrealizable" features f also addressed in Appendix
4 Experiments
4.1 Baseline imitators and evaluation metrics
As baseline methods for comparison, we evaluate: Maximum Entropy Inverse Reinforcement Learn- ing (MaxEnt IRL) (Ziebart, 2010);: Linear Programming Apprenticeship Learning (LPAL) (Syed et al.)
2008); Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016); an oracle version of InfoGAIL. (Li et al.) (017) trained on pre-determined mode clusters (InfoGAIL")) Risk-Averse Imitation I Learning (RAIL) (Santara al., (008);; and regret-based Bayesian Robust Optimization
for Imitation Learning (BROIL) (Brown et al. 2020a Javed al. 2021). Additional experimental

---

details are described in Appendix C We use withheld testing demonstrations to evaluate the diversity and quality of imitator policies in two ways:
Stochastic Dominance estimates whether the imitation reward distribution is better than the demon- stration reward distribution. We randomly select a of weight vectors to induce various reward functions. For each weight vector, we evaluate whether the imitation policy stochastically dominates the testing demonstrations (Def. 2.4) and report the rate of stochastic dominance for each approach.
Pareto Dominance estimates when imitation trajectories are unambiguously better than demon- strations. We use the exact imitator policy or randomly sample set of rollouts. We measure (ff((iia)) ž f( for each demonstration and report the minimum, average, and maximum.
4.2 Illustrative grid world experiments
We first consider Lava World, a deterministic grid environ- ment from the robust imitation literature (Brown al. (020a). Each trajectory starts from the same initial state and seeks to reach fixed goal state in the bottom-right corner of the grid. At each time step, the agent can move in any of the four cardinal directions. Trajectories are characterized by two features: the number of white and red cells traversed (Figure 3). The cost of a trajectory computed as a weighted sum of these features. trajectory terminates either when the agent arrives at the goal state or when a fixed time horizon (e.g., 10) reached.
+
Figure 3: Sample demonstration in the Lava World grid environment with white and red (lava) grid cells.
While the cost feature weights for this environment are unknown, we are provided with a set of demonstrated trajectories. To train and evaluate our approach, we first divide a set of trajectories (with unique features) reaching the goal within 10 timesteps into imitator candidates (when on or near the Pareto frontier) and demonstrations (when less optimal). We then further divide the demonstrations into two random, equally- sized subsets for training and testing.
Training Demonntraation/Rlllott Pairings
Testing DemonstrationnRollout Pairings
Demoretratins
White Cells (negated)
White Cells (negated)
-6-
-7-
-10
-12
-30
Red Cells (negated)
Red Cells (negated)
Figure 4: For training (left), demonstrations (bluc) are paired with rollouts (red) via 9 to construct the imitator distribution. For testing (right), withheld demonstrators paired with that imitator distribution via (8) Darker green pairings correspond to larger y values.
We employ our fully-realizable training approach (Definition 3.7 for PSD. We first prune the candidate trajectory set by removing trajectories that Pareto-dominated by others in the set. This ensures that only trajectories with potentially optimal rewards remain to define the imitator's policy. Subsequently, match the training set demonstrations with the pruned candidate trajectories by solving a quadratic program based on Eq. ( with fixed subdominance variables a = 1 and 8 = 0.5. and L2 regularization of the imitator trajectory distribution P(g) to promote greater uniformity over the set of candidates, resulting in improved generalization to unseen demonstrations. |Figure (left) shows this matching for particular training sample.
To verify generalized stochastic dominance after training, withheld demonstrations are matched to the imitator's 's trajectory distribution using Equation (8) If the objective of the matching problem
7

---

is zero (equivalently, the imitation trajectories all dominate their paired demonstration trajectories), then stochastic dominance is guaranteed, as Figure (right) shows.
0.50, -0.50 reward
0.99, 00.01 reward
-0.01, -0.99 reward
1.0
3.0
010
0.3
0.0
P(Reward >
P(Reward
0.0
P(Reward
0.-
2.6-
Demonenatioon
Dencrndrations
PSD
PSB
Maxcrt
Maxtre
Moxna LPN.
UPAL
Brail
Broi
values
values
values
Figure 5: Cumulative rewards of the trajectory distributions of the demonstrations, PSD, MaxEnt IRL. and Regret methods for three reward functions.
Figure F shows the excess reward distributions for three reward functions. Other imitation methods produce curves that worse for some portions of some reward functions, reflecting the trade-offs they make for better performance in other portions of these curves. In contrast, PSD produces excess reward curves that strictly better than the demonstration eurves for all three reward functions the train-test split of Figure indicating better mode coverage.
Comparative Dominance Analysis Under Different Training Data Proportions
Probabilty Cardition Tulfiliment Percenaage
90
70
60
80
PSD
60
MaxEnt
LPAL
30
40
50
60
80
90
100
Porcentage of Data Used for Training
Figure 6: Average dominance (with standard error) for different training data amounts.
Figure 6 shows the percentage randomly produced reward functions for which each imitation learning method stochastically dominates the withheld demonstration set, and how it changes with the amount of training demonstrations. PSD provides higher rate of stochastic dominance across the entire range of training data sizes compared other methods.
Table D provides summary statisties for how frequently demonstrations are Pareto dominated by the imitator's 's trajectory distribution (or withheld demonstrations as an additional baseline). PSD Pareto dominatess all demonstra-
Table 1:
Frequency of Pareto dominance over demonstrations Lava World.
Policy
min
avg
max
Demonstrations
0.08
0.16
0.50
tions with at least 8% probability. In contrast, all other imitators fail to have any probability of Pareto dominating
MaxEnt IRL
0.00
0.33
1.00
LPAL
0.00
0.46
1.00
at least one demonstration. Additionally, PSD provides
BROIL
0.00
0.15
1.00
the highest average Pareto dominance, with nearly half of
PSD
0.08
0.47
0.92
the imitator trajectories dominating (i.c., being unambigu- ously better) than the demonstrations.

---

4.3 Policy model optimization
Our second set of experiments considers policy model optimization (Alg. D in the Point Bot (Javed et al., 202T) and Reacher (Todorov et al., 2012) envi- ronments, Point Bot is continuous robotic task for navigating point mass (subject to noisy, velocity-ban -aacct air resistance) in two-dimensional plane from a starting position to pre-defined and stationary goal, ostensibly without passing through a gray region (obstacles). The robot moves by applying a force cardinal direction. Reacher is robotic arm with two rigid links and two joints (Figure 7) The end of one link is fixed to the center of the environment. In our multi-modal variant, the goal is to move the robot's 's end effector to one of two targets (red or yellow) by applying appropriate sequences of torques.
Figure 7: Reacher environment.
For the Point Bot environment, the imitators learn from the set of human- cumma-deemontrrattee trajectories shown in Figure (leff))) Some demonstrations completely avoid the obstacles (gray areas), others partially avoid obstacles, while many appear entirely oblivious of obstacles. The trajectories are characterized by the number of timesteps in gray areas, the number of timesteps in white areas, and the sum of distances to the goal location over the trajectory. To facilitate multi-modal policy learning, we increase the number of layers of the policy model from two to four (each with 64 fully connected hidden nodes) compared to prior work (Javed et al. 2021). For PSD, we train this model using the demonstration normalization variant of Algorithm
(a) Demonstrations
(b) Behavior Cloning
(c) GAIL
(d) InfoGAIL*
(e) PSD
LILE Figure 8: Point Bot training demonstrations starting from lower left origin and moving to the goal in the upper right (a); and sample trajectories from policies learned using each method (b)-(e).
Behavior cloning produces trajectories that are similar to the demonstrations, but often of lower quality. Specifically, the sample trajectories in Figure 8 encounter more obstacles and sometimes fail efficiently reach the goal. We initialize the other imitation learning methods from this behavior cloning policy as a starting point. GAIL (Figure (C) suffers from mode collapse, ultimately producing trajectories that all encounter obstacles. InfoGAlL* (Figure (dt) is given a binary mode membership dbbstacle-avoiding or obstaclc-oblivious) of each trajectory and learns a separate model for each mode. By matching each demonstrated mode's 's means, the resulting trajectories tend be suboptimal compared with the best demonstrations of the corresponding mode. PSD imitation (Figure ( produces trajectories that are of higher quality while still covering the spectrum of trade-offs between grays cells and white cells/distance of the demonstrations. Significant support is provided to these two modes: obstacle avoidance and obstacle obliviousness. However, some trajectories also cover trade-offs between the two modes (e.g., avoiding the first obstacle, but not the second). This illustrates the flexibility PSD to operate in settings without clearly specified latent spaces of modes (e.g.. binary-valued modes).
For the Reacher environment, demonstrations are synthetically produced using reinforcement learning-more specifically, the soft actor-critic (SAC) algorithm (Haarnoja et al.) [2018) learn policies for two different targets, the yellow circle and the red circle. A SAC-learned policy (for the yellow or the red goal) and a uniformly random policy are sampled with complementary probabilities to produce the trajectories of the demonstration set with an equal number of red goal trajectories and yellow goal trajectories. The features we incorporate are the sum of distances from each of the targets over the entire trajectory. The policy model that the imitators train a Gaussian multi-layer perceptron with four layers of 64 neurons. For PSD, we train this policy model using the weighted best match variant of Algorithm [ and also report PSD with optimized a parameters (PSD-a") "), as described in 3.3

---

Table 2: Frequency of imitator Pareto and stochastic dominance of demonstrations.
Point Bot
Reacher
Pareto
Stochastic
Pareto
Stochastic
Policy
min
avg
max
avg
min
avg
max
avg
Demonstrations
0,000
0.222
0.444
0.000
0,000
0.144
0.333
0.000
Behavior Cloning 0.001
0.180 0.353
0.000
0.001
0.154
0.290
0.000
GAIL
0.000
0.015
0.031
0.002
0.000
0.005
0.023
0.425
RAIL
0,000 0,004
0.031
0.000
0.000
0.000
0.000
0.000
InfoGAIL."
0,000 0.227 0.496
0.000
0.409
0.474
0.500
0.071
PSD 0,070
0.326 0.493
0.420
0.452
0.498
0.547
0.561
PSD-a"
0.080 0.387 0.642
0.657
0.466
0.500
0.534
0.662
Table provides statistics for how well demonstrations are supported by the imitator policy (Pareto dominance) using 1000 policy rollouts, and whether the reward distributions of the imitator are strictly better than the demonstration reward distribution (stochastic dominance) averaged over 1000 random reward functions for both Point Bot and Reacher. Similarly to the fully realizable experiments, there some demonstrations that are very difficult for the baseline methods to outperform (near zero minimum Pareto dominance values). The Point Bot demonstrations pose a significant challenge because some are distinct from the two main demonstrated modes, causing InfoGAIL* * to also perform poorly in terms of minimum Pareto dominance. In contrast, the PSD policy produces trajectories that provide coverage of all demonstrations. More broadly, PSD excels across all metrics because tends to produce trajectories that are: in proportion with the demonstrated behavior modes and often of higher quality than the demonstrations comprising that mode. PSD's high frequency of stochastic dominance illustrates the effectiveness of our policy gradient optimization in achieving the PSD objective. The other imitation methods unfortunately unable maintain the modes of the demonstrations and generally exhibit poor performance across all of these metrics, with a few exceptions, as a result.
5 Discussion and conclusions
This paper introduces stochastic dominance important property of distributional alignment (Sorensen et al. 2024) for imitation learning when demonstrations reflect the differing preferences of distinct demonstrators. Stochastic dominance provides stronger guarantees for demonstrators than expectation-matehiee imitation methods: reward distributions for each demonstrator that are at least as good as the demonstrated distribution-desppee not knowing each demonstrator's exact reward function-for all common risk-sensitive measures. This avoids policies that are compromises between competing objectives by maintaining stochasticity. Though directly achieving stochastic dominance appears computationally difficult, we establish relaxation using optimal transport theory that leads exact algorithms in the fully-realizable setting and policy gradient algorithms when training policy model. Through qualitative and quantitative analyses we show that our imitation learning approach provides support to all demonstrated behavior modes, while aiming to produce better quality behavior within those modes, leveraging concepts of both Pareto and stochastic dominance.
There are multiple important directions for future research. We have focused on deterministic dynam- ics in this paper. While our policy model optimizations naturally extend to stochastic environments, additional analyses and experimental validation remain future work. Next, though hand-engineered reward features are reasonable for engineered systems (e.g., self-driving vehicles, robotics), many imitation learning methods learn reward functions without such features being available. Integrat- ing reward feature learning in a manner that leverages potential multi-modality of demonstrations using our framework is an important future direction to avoid the limitation of known reward fea- tures. Finally, one key challenge is that policy optimization based on distributional criteria appears more challenging than maximizing expected rewards. Exploration of both on-policy and off off-policy reinforcement learning in this context is likely needed for scaling to larger environments.
10

---

Acknowledgments
This work was supported by the National Science Foundation under award #2312955.
References
Abbeel, P. and Ng. Apprenticeship learning via inverse reinforcement learning. Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Arjovsky, M., Chintala, S., and Bottou, L., Wasserstein generative adversarial networks. In Interna- tional conference on machine learning, pp. 214-223. PMLR, 2017.
Armbruster, B. and Luedtke, J. Models and formulations for multivariate dominance-conntrained stochastic programs. Transactions, 47(1):1-14, 2015.
Brown, D., Niekum, S., and Petrik, M. Bayesian robust optimization for imitation learning. Advances in Neural Information Processing Systemx, 33:2479-2491 2020a.
Brown, D. S., Goo, W., and Niekum, S. Better-haan-demonsrraaoo imitation learning via automatically- ranked demonstrations. In Conference robot learning, pp. 330-359. PMLR, 2020b.
Chen, L., Paleja, R., and Gombolay, M. Learning from suboptimal demonstration via sulf-supervised reward regression. In Conference on robot learming, pp. 1262-1277. PMLR, 2021.
Dadashi, R., Hussenot, L.,, Geist, M., and Pietquin, O. Primal Wasserstein imitation learning. In
International Conference on Learning Representations, 2021.
Dentcheva, D. and Wolfhagen, F. Optimization with multivariate stochastic dominance constraints.
SIAM Journal on Optimization, (5(1)5564 2015.
Dvoretzky, A., Kiefer, J., and Wolfowitz, J. Asymptotic minimax character the sample distribution function and of the classical multinomial estimator. The Annals of Mathematical Statistics, pp. 642-669, 1956.
Dvurechensky, P., Gasnikov, A., and Kroshnin, A. Computational optimal transport: Complexity by accelerated gradient descent better than by sinkhorn's 's algorithm. In Proceedings of the 35th International Conference on Machine Learning, pp. 1367-1376. PMLR, 2018.
Fickinger, A., Cohen, S., Russell, S., and Amos, B. Cross-domain imitation learning via optimal transport. In International Conference on Learning Representations, 2022.
Haarnoja, T.. Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. Pmlr, 2018.
Off Off-policy maximum entropy
Haldar, S., Mathur, V., Yarats, D., and Pinto, L. Watch and match: Supercharging imitation with regularized optimal transport. In Conference on Robot Learning, pp. 32-43. PMLR, 2023.
Ho, J. and Ermon, S. Generative adversarial imitation learning. Advances in Neural Information Processing Systems, 29, 2016.
Javed, Z., Brown, D. S., Sharma, S., Zhu, J., Balakrishna, A., Petrik, M., Dragan, A., and Goldberg, K. Policy gradient Bayesian robust optimization for imitation learning. In International Conference on Machine Learning, pp. 4785-4796. PMLR, 2021.
Kantorovich, L. V. On the translocation of masses. In Dokl. Akad. Nauk. USSR (NS), volume 37, pp- 199-201, 1942.
Kopa, M. and Petrová, B. Strong and weak multivariate first-order stochastic dominance. SSRN 3144058, 2018.
Kostrikov, I., Nachum, O., and Tompson, Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2019.

---

Lacotte, J., Ghavamzadeh, M., Chow, Y., and Pavone, M. Risk-sensitive generative adversarial imitation learning. In International Conference on Artificial Intelligence and Statistics, pp. 2154- 21 2163, 2019.
Leamed-Miller, E. and DeStefano, J. A probabilistic upper bound on differential entropy. IEEE Transactions on Information Theory, 54(11):5223-5233 2008.
Li, Y., Song, J., and Ermon, S. Infogail: Interpretable imitation learning from visual demonstrations. Advances in neural information processing systems, 30, 2017.
Luo, Y., Cohen, S., Grefenstette, E., and Deisenroth, M. P. Optimal transport for offline imitation learning. The Eleventh International Conference on Learning Representations, 2023.
Majumdar, A., Singh, S., Mandlekar, A., and Pavone, M. Risk-sensitive inverse reinforcement
learning via coherent risk models. In Robotics: Science and Systems, volume 16, pp. 117, 2017.
Naaman, M. On the tight constant in the multivariate dvanezzky-keeeer-wooiivit inequality. Statistics & Probability Letters, 173:10908: 2021.
Neu, G., Jonsson, A., and Gómez, V. A unified view of entropy-regularized Markov decision processes, arXiv preprint urXiv:1705: 7705.0778, 2017.
Ng, A. Y. and Russell, S. Algorithms for inverse reinforcement learning. International Conference Machine Learning, volume 1, pp. 2000.
Nguyen, T., Le, T., Dam, N., Tran, Q. H., Nguyen, T., and Phung, Tidot: A teacher imitation learning approach for domain adaptation with optimal transport. In International Joint Conference on Artificial Intelligence 2021, pp. 2862-2868, 2021.
Ogryczak, W. and Ruszezyúski, A. From stochastic dominance mean-risk models: Semideviations as risk measures. European journal of operational research, 1 16(1))33-50, 1999.
Pomerleau, D. A. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
Raghavachari, M. On connections between zero-one integer programming and concave programming under linear constraints. Operations Research, 17(4):680-684, 1969.
Ratliff, N. D., Bagnell, A., and Zinkevich, M. A. Maximum margin planning. In International Conference Machine Learning, pp. 729-736, 2006.
Rioux, G., Nitsure, A., Rigotti, M., Greenewald, K., and Mroueh, Y. Multivariate stochastic dominance via optimal transport and applications to models benchmarking. arXiv preprint arXiv:2406.06445, 2024.
Ross, S., Gordon, G.. and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pp. 627-635, 2011.
Santara, A., Naik, A., Ravindran, B., Das, D., Mudigere, D., Avancha, S., and Kaul, B. RAIL: Risk- Averse Imitation Learning. In International Conference on Autonomous Agents and Multi-Agent Systems, pp. 2062-2063, 2018
Singh, S., Lacotte, J., Majumdar, A., and Pavone, M. Risk-sensitive inverse reinforcement learning via semi-and non-parametric methods. The International Journal of Robotics Research, 37(13-14):
1713-1740, 2018.
Sorensen, T., Moore, J.. Fisher, J., Gordon, M., Mireshghallah, N., Rytting, C. M., Ye, A., Jiang, L.,, Lu, X., Dziri, N., Althoff, T., and Choi, Y. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05007. 2024.
Sutton, R. and Barto, A. Reinforcement Learning: Introduction. Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
12

---

Swamy, G., Choudbury. S., Bagnell, J. A.. and Wu, S. Of moments and matching: A game theerttc framework for closing the imitation gap. In International Conference on Machine Learning, pp- 10022-10032, 2021.
Syed, U. and Schapire, R. E. A game-theoretic approach to apprenticeship learning, Advances in Neural Information Processing Systems, 20, 2007.
Syed, U., Bowling, M., and Schapire, R. E. Apprenticeship learning using linear programming. In International Conference Machine Learning, pp- 1032-1039, 2008.
Todorov, E., Erez, T., and Tassa, Mujoco: A physics engine for model-based control. 2012 IEEE/RSJ international conference on intelligent robots and systems, 5026-5033, IEEE, 2012.
Williams, R. J. Simple statistical folllwing algorithms for connectionist reinforcement learning. Machine learning, 1-229-256, 1992.
Wu, Y.-H., Charoenphakdee, N., Bao, H., Tangkaratt, V., and Sugiyama, M. Imitation learning from imperfect demonstration. In International Conference Machine Learning, pp.
PMLR, 2019.
Xiao, H., Herman, M., Wagner, J., Ziesche, S., Etesami, J., and Linh, H. Wasserstein adversarial imitation learning. arXiv preprint arXiv: 19tp://900005115 13, 2019.
Zhang. S., Cao, Z., Sadigh, D., and Sui, Y. Confidence-aware imitation learning from demonstrations with varying optimality. Advances in Neural Information Processing 3 34:12340-12350, 2021.
Ziebart, B., Choudhury, S., Yan, X., and Vernaza, Towards uniformly superhuman autonomy via
subdominance minimization. In International Conference on Machine Learning, pp. 27654-27670. PMLR, 2022.
Ziebart, B. D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA]-
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1 (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked While "[Yes] " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providded a proper justification is given (e.g., "error bars expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way,
the checklist one of the factors in their evaluation.
reported because it would be too computationally
"[No] " or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer often more nuanced, so please just use your best, judgment and write justification elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims are motivated in theory in $ 3 with experimental results provided in $ $ This paper's 's contribution is compared to other popular methods with
summary of results Table and Table 2 where we measure stochastic dominance and Pareto dominance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
14

---

Answer: [Yes]
Justification: Though we do not create a separate section to enumerate limitations, all our assumptions (e.g., about the conical sum cost functions of given cost features) are stated when we discuss the motivation of our approach. Additionally, we discuss expanding beyond these limitations potential future work in our conclusions section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
All theoretical claims have their proofs provided in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
15

---

We have outlined in the paper the information that we believe is necessary to reproduce the results. Furthermore, we plan on releasing code which would enable exact replication.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code with instructions will be uploaded in supplementary material, and also later publicly released with paper publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/nips.cc/ppull httpsccccpppbccc/eeeeceec ptuuuddeee/odeeubbiisssooooooo.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
16

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Although our contribution is mainly theoretical, a summary of these details is available in Appendix [ However, we understand this might not be enough for complex experiments, For this reason, we will publish the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Although error bars are reported for only the fully-rcalizable setting, the results in Tables T and 2 are the average of multiple runs described in
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
17

---

Answer: [No]
Justification: Detailed compute times and requirements are not provided. But experiments are simple enough to typical modern laptops stated in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e-g.. preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/ http://eeerrip.ccppppbbeee wwtpccccpppcceeeeeeeeeeecceccece htt.ai//ehhhcsciiiinnnees
Answer: [Yes]
Justification: The work does not utilize human subjects and uses simulated environments. Public code was used that is referenced. Public datasets were not used. Our code will be released along with the demonstrations it was tested on.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: No direct negative societal impact expected.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
18

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not utilize models or datasets with high risk of misuse. General safety concerns that apply to any Al algorithm model apply.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Although license terms explicitly mentioned, the original creators repositories upon which we build our work are credited along with links to original repositories, where the detailed license terms available. All repositories that work primarily builds upon are under MIT License.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, saperswithcodde com /atasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The only new assets are some demonstrator trajectories that will be released alongwith the code.
Guidelines:
The answer NA means that the paper does not release new assets.
19

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We only human trajectories for some of our experiments. These were generated by the authors themselves 'playing' the simulated RL environment.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Study does involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
LLMs not involved as core important part of study.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
20

---

Please refer to our LLM policy httpss//Deerriigccc///oorrrrrres
for what should or should be described.
21

---

A Additional Background Material
A.1 Risk-sensitivity
Limiting the probability and magnitude of undesirable outcomes is crucial in many applications. Popular metries for assessing these include value risk (Def. A.Ip. conditional value risk (Def. A.2). and range value at risk (Def. [A.3).
Definition A.1. The Value-at-Rine
(VaR) for a random reward variable X and given quantile
E [0, 1]. is the inverse of the cumulative density function F(X) (if exists) 112 or the generalized inverse (e.g., for discrete variables)
13):
Fx (c)
(12)
VaR, = vx())
inf P(X S x) c).
(13)
Definition A.2. The Conditional Value-at-Rish (CVaR), measures the expected value within the tail of reward distribution for given quantile CV)R, Ex [x] S ve())] Definition A.3. The Range Value-at-Risk (RVaR) discards both tails (defined by e and d) of the reward distribution and computes the expectation for the range in between: RVaR.,d = Ex [X] (x( X S vx())
A.2 Stochastic Dominance
We review three multivariate extensions of stochastic dominance (Definition 2.4
Definition A.4 (Distribution-based stochastic dominance (Kopa & Petrová 2018)). Random
variable X stochastically dominates (in distribution) random variable if and only if
Vc E K P(X 3 c) S P(Y c).
Definition A.5 (Utility-basee based stochastic dominance (Armbruster & Luedtke 2015)). Random variable
X stochastically dominates random variable Y if and only if Elu(X)] 2 EY] for any non- decreasing utility function u IR + R.
Definition A.6 (Linear stochastic dominance (Dentcheva & Wolfhagen 2015 Random
variable X stochastically dominates (in distribution) random variable Y if and only if 40 2 0, 0 X has univariate stochastic dominance over 0 YY.
The linear stochastic dominance definition motivates our primal formulation using the traditional imitation learning assumption of linear reward functions Abbeel & Ng (2004). However, the utility- based definition the strongest of these three (Armbruster & Luedtke 2015 Kopa & Petrová 2018). From an imitation learning perspective, it guarantees that a demonstrator or user with an unknown non-decreasing utility function will prefer the distribution of random rewards from variable X over the distribution of random rewards from variable Y. Our dual relaxation aligns with this definition.
A.3 Optimal Transport
We provide proof sketch for Remark 2.6 which establishes a direct correspondence between stochastic dominance and optimal transport. For simplicity, we assume one-to-one mapping between distributions, but the argument easily extends when the mapping is not one-to-one.
provides notation for defining a mapping between distributions for X and Y, namely using the pairing (ve(X), voYY)) Ve e [0, 1]. which maps between values at the same quantile.
Proof sketch for Remark 2.6 The (generalized) inverse ve of Definition A.
Y stochastically domínates X, then (X) < (Y Vc E [0, 1]. This corresponds to an optimal transport solution with value 0 for cost functions that are when x S y.
In the other direction, the optimal transport solution has a value of 0, then for some complete set of pairings (2),
c2), Mey (X) S (Y). If ci * C2 for some of these pairings, then given one pairing (Ca, (x) such that ca > Cz. there must be another pairing (ca. Cy) such that Cs < Cy and Cb < Co by the pigeonhole principle, resulting in Ve, (X) Ve, (X) < Ve,, (Y) Voz (Y). Re- pairing these as Cca:Cy) and (cs. Cx) does not increase any cost an optimal transport solution since Ve, (X) < Ve, (Y) and Ve, (X) < Vc, (Y), and brings the pairing closer to sorted matching. This re- ne-pairing procedure can be continued until the resulting pairing is exactly (v, (X), (Y)) Ve E [o, 1).
This pairing is exactly the definition of stochastic dominance (Definition 2.4
22

---

B Proofs of Theorems and Lemmas
Proof of Theorem
3.3 We first prove that |ax OT AAul+ (I'x, Px) 0
Px (r)(E)) x PSD
Pa (ro(E)). First, consider the optimal assignment for a specific 0 E [0. 10.1] i|
in Definition 3.2
(0)) Since the corresponding objective value assumed be zero, this implies
that:
(0) > (.)) 2 (E,)
(14)
( (E) 2 c] = mk, (o(E)) 2 c) [Tr((E)) 2 V R,
(15)
where [x]] is an indicator function that evaluates to 1 if expression æ is true and otherwise. We leverage this equality (15) in (b) below prove the implication:
O( E R 10 1 Px(ro(E) 2 c) e
Yig (O)
(16)
mi (ro 2 2 (o(E)) 2 c]
(
ni, (0)1 (o(E)) c]
A (ro(( 2 C).
where inequality (a) results from adding additional condition. This inequality is then generalized to all positive values of 0:
v(o,00 0) E R20 [0, (ara((()) 2 aC) 2 Pa (ara(E) oC)
1O E E 20 (ro(E) 2 Pa 20).
R
In the other direction (P+()(()) EPSD (r(r((E) = max OT (P P+) = 0), our proof is constructive. For any 0 E B sort the trajectories & and E, according to r-()). Then choose mi (0) that matches &; and E, according to the sorted order with the weight based on the remaining
unmatched probabilities of (C,) and (E(E))- Since (ro(() C) 2 (ro((() 2 C), then (ro(E)) ro(S)) 0, OT [ (Ix,,, P=) 0. This holds for all 0 E [0, n
Vi.j
Proof of Thewrem? 2.5 Leveraging the results of Theorem
3.3 we consider the (0, for which
Ens Thet (T((E)) ro((s)] 0. This implies that:
V(i,j), n. (0) r(En)) 2 Ra (0) (E())
(17)
Σ (0) r(En)) 2 Σ ne.s rE(E))
(18)
1
bj
= Eçar. [r(E)] 2 Be-p= (ro(t)
(19)
Further, using the definition of pluralistic stochastic dominance (Definition 3.1. we have: P. PSD v0 E
Pa
K
C
E IR:
20:
Pr PSD P& iff: v(O E RKx C E R),
(20)
(r(((()) 2 C) 2 (=((o(() 2 C) (ro(EE) < C² s P+ (ro (5) C).
(21)
(22)
For convenience, we define these probabilities as p and
p p=Pr(to(t) s < (T) < 2C) = q (,(r((E))= =C
(23)
v)(ro(E))= C and
(24)
23

---

but < q and as VaR monotonically increases with increasing confidence level (if a' 2 then
Val (X) 2 Va (X)):
(To(E)) = C' 2 C
(25)
As pluralistic stochastic dominance holds for all C, VaR guarantee holds for all a
aa (ro 2 (ro(())(E))
Va E [0.
(26)
The proof for CVaR trivially follows from 26 If a function is always smaller than the other, its definite integral and average (with restricted domain) will also be smaller:
Pa (ro((E)) - 1 S. (To(E)) d7 _ [ (r((E( dy pa
1
(27)
(VaRa(1o(E)) 2 (ro
Ya [0,
(28)
And for the same reason, if we have different limits of the definite integral, the inequality still holds:
1 [e un (ro(E)) dy 2 (r,s(ro(())) = B -
3 a [ en (ro(E) dr = (r,e(ro(E))
1
(29)
RVaRa, s 2 RVaRR,,(ro(E))
0 o<asB<l
(30)
Lemma B.1. The matched subdominance minimization value (Def. 3.5) upper bounds the worst-case
reward difference optimal transport value: OT zubdomn,, (P Px) 2 maX0e(0.1) OT ( (I Pa).
Proof. Starting from Definition
3.2
(a)
min Σ 8E[0,1] 7
Wi 0 (.f(E)) 8. (((E))
I min max
Σ Vi 0 (e.(())) 8 (f((E)
(31)
x2 80
(b)
min
Vi.j max DE 0.1
(0.(()) 0 f((a))
(32)
min
Th.j su.jsuboom ((E:),
(33)
(d)
Su.juubdom ,,(s, Ey),
min 2
where: (a) follows from the maxmin- minmax inequality; (b) makes the maximizing choice of independently for each pair (i,j) with Vio > 0: (c) results from the subdominance being the worst- case difference rewards for the imitator; and (d) is nondecreasing as subdominance margin increases.
Proof of Theorem 3.6 Since the optimal matched subdominance (Definition 3.5) upper bounds the optimal pluralistic sensitive matching (Definition 3.2) via Lemma B.I an objective value of zero for the former implies objective value of zero for the latter. Theorem 3.3 then implies pluralistic stochastic dominance.
Figure 9 shows when minimizing the matched subdominance is unnecessary for pluralistic stochastic dominance (i.e., P. PSD F+
OT subdoms o (P,, P&
0).
The right-sided multivariate Dvoretzky- Bieezzkk-KKieer---ee -iofe--ooooo--o inequality (Dvoret zky et al. 1956 Naaman 2021) provides probabilistic bounds on the deviation between the empirical multivariate cumulative mass function (CMF) with N samples (FN) and the true population CMF (F) from the right as (See proof of Lemma 4.1 in NNamann 202
Proof of Theurem 3.8

---

Imitator Samples
Imitator Samples
Imitator/Samples
Imitator Samples
|mitatorrSamples
Demonstrations
Demonstrations
Demonstrations
f
Figure 9: (left) Imitation not being pluralistic stochastic dominator of demonstrations (and the maximizing the improvement violation); (center) imitation being a pluralistic stochastic dominator, but paired subdominance nonzero; (right) imitation being a pluralistic stochastic dominator and paired subdominance is zero.
Empirical CMF: F
Shifted Empirical CMF: F
P
f P
~v P
Pareto frontier
Pareto frontier
feature 2
feature 2
feature 1
feature
Figure 10: (Left) A sample population of demonstration feature vectors (in blue) in two dimensions overlaying the heatmap its empirical CMF, and (Right) the heatmap the empirical CMF shifted to the right by 0.05, along with feature vector samples from its corresponding PMF obtained via finite differences (in green).
P(D =)== =P sup (F (f) (F)- >6) < <NKe-
(34)
PEERK
which is the same saying
(F (f) ((f) SE) 5x)21 21-NKe 2N6
P
sup
(35)
FERK
since suPrEr (FE (f) F(f)) Se + V RK (F) - F(f)) S we have:
P((FA) (f) -c) F(f)) 21 1 NKKe Vf E R
(36)
Our approach considers the worst-case distribution within these bounds and characterizes when stochastic dominance the worst-case can be guaranteed. This perspective inspired by entropic analysis under these bounds (Learned- Miller & DeStefano 2008).
Consider the worst-case CMF based this bound:
(f)
F (f) c. cl[f 00),
(37)
25

---

where is reduced from every f E IK and added to the largest possible value of The corresponding probability mass function, F (f) may be obtained from the CMF via finite differences.
If (f) > O, but f is not Pareto-dominated by any realizable demonstrator trajectory, we call f unrealizable. Let u A (f unrealizable) denote the probability under P of cost features that are not possible to Pareto-dominate (or equal) by any realizable demonstrator trajectory. The sum of these unrealizable probabilities is an integer multiple of which we add one more (the one subtracted from the CMF at every input value). We assume that these unrealizable probabilities u get assigned, or "matched", to the worst-case realizable demonstrator trajectory Swe during optimal transport. See Figure I0 for an example of the discussed quantities in two dimensions with = 50 and 0.05 for the case when
0.
From (36 we see that the random variable with CMF F (f) stochastically dominates the random variable with CMF F by margin of e with probability at least (1 N Ke K
Now if max OT ud:..: .P - _ = for the remaining distributions after making the worst-case assignment of unrealizable probability to Ewe, in other words, if our approach achieves pluralistic stochastic dominance over the trajectory distribution of the worst-case CMP, i.e., P, PSDD F (by Theorem 3.6). then by (36) we have pluralistic stochastic dominance over the demonstrator population trajectory distribution Pa with probability at least (1 Ne -2Nc2, -
). i.e., I(P,, PSD
F*) 2 (1 11--NA Ne -
C Additional Experimental Details
We provide additional experimental details in this section, including expanded interpretations of our evaluation measures (Section C.T). implementation details (Section C.2) and supplementary experimental evaluations (Section
C.1 Evaluation Measure Interpretations
Since assessing stochastic dominance across the entire set of conical sum reward functions directly is computationally challenging (Section 3.1 we instead assess the dominance of the imitator over the demonstrator from two different perspectives.
Our stochastic dominance measure considers the frequency of stochastic dominance over randomly sampled cost functions rather than the guarantee for the entire set of reward functions. For each sample reward function, stochastic dominance guarantees:
That for all non-decreasing utility functions applied to the sampled reward function, the imitator is preferable to the demonstrations (Def(A.)) and
For any reward threshold, the imitator has an equal or higher probability of exceeding that threshold compared to the demonstration distribution 2.44..4)
Our Pareto dominance evaluation measure considers uniform dominance (Definition 2.3) between sampled imitator trajectories and the demonstration set of trajectories, indicating how frequently the imitator trajectory is preferred for conical sum reward functions. Large values across the entire set of samples indicate good alignment with the demonstrations. Low values do not prevent stochastic dominance (as shown in Figure g center), but they tend to indicate some degree of misalignment. We report the minimum, average, and maximum of these samples to assess how well the imitator trajectory samples align with demonstrations.
C.2 Implementation Details
Our implementation builds upon OpenAl Spinning Up (Brown et al. 2020a repositories.
PG- (G-BROIL (Javed et al. 2021 and BROIL
broil
https///ggtaabbcccm//ammmmm////3
26

---

For Lava world experiments, for reporting the results comparing different approaches with different amounts of training data and frequency of imitator Pareto dominance, we have randomly split the whole set containing 24 demonstrations into two equal splits of training and testing 100 times, and the represented results are averaged. We have used the time horizon of 10 for trajectories. For solving the Quadratic Program (QP), we have used MOSEK optimizer, and have set the regularization parameter to 0.001. For subdominance calculation during training, we have set the parameter to 0.5. The threshold for achieving the goal in Point Bot was originally in the default setting of 1, however, we have increased to 10. The initial features for this environment are the number of gray and white cells, however, we added another feature that takes into account the distance from the target.
We use REINFORCE algorithm for policy optimization. For our policy network, we have used a Gaussian Multi-Layee Perceptron (MLP) with 4 hidden layers each having 64 neurons with the Tanh activation function. The network receives the agent's observations and produces a mean and standard deviation for each action dimension, and the agent takes actions by sampling from this Gaussian distribution. For optimization, we used the Adam optimizer with learning rate of 3e 5, and for the subdominance calculation, we set the 3 parameter to 0.001. We solve the QP using MOSEK optimizer. Training goes on for 2000 iterations, and the best model saved according to the lowest QP objective value. We use 10 demonstrations for training, they come from two main modes, with each mode having demonstrations. During each iteration, we rollout 30 trajectories. For PSD-a", we have used Adam optimizer with a learning rate of 5e 4 for learning alpha values. Alpha values are initialized uniformly, sum to 1, and are always at least equal to 0.1. Training goes on for 4000 iterations and similarly the best model is saved.
The reacher environment we use has two targets with fixed positions, one labeled with a yellow circle and the other red. We use two sets of demonstrations, each containing 15 trajectories, for two different modes of behavior. The modes include moving the robot's ss end effector to the yellow or red circle. For Reacher, we have used the same policy optimization algorithm we used for Point bot, REINFORCE. The policy network architecture and QP optimizer are also the same. We have used Adam optimizer with a learning rate of 3e-3, a ,3 parameter value of 0.001, and with the MOSEK regularization parameter set to 1000. During each iteration, we rollout 60 trajectories. Training goes on for 400 iterations and the best model is saved based on the lowest QP objective value. For PSD-a", we have used Adam optimizer with a learning rate of 1e for learning alpha values. Similar Point Bot, alpha values are also initialized uniformly, sum to 1, and at least 0.1. Training goes on for 1000 iterations and similarly the best model is saved.
For Point Bot, baseline experiments (GAIL, RAIL, InfoGAIL, BC) were run on several different personal computers and the slowest one took less than 5 hours to converge (e.g. on a laptop with 2.6GHz 10-core CPU, 32GB RAM). For Reacher, GAIL seemed to converge much earlier than Pointbot, taking close to an hour to converge (showing minimal improvements with longer training). RAIL trained for 2-3 hours Reacher but failed to display any improvement in metrics. Experiments for PSD were run on an in-house server with GPU accelcration (cquipped with two Nvidia GTX 1080 Ti GPUs), taking close to hour and 1.5 hours each for convergence with Pointbot and Reacher, respectively.
C.3 Supplemental evaluations
We provide additional experimental results in this section.
27

---

MinSubPG
Figure 11: Trajectory samples from policy learned using subdominance minimization Ziebart et al. 2022
Figure shows the resulting trajectory samples from a policy learned using uniform subdominance minimization Ziebart ct al. 2022) in the Point Bot experiment of Section 4.3 Like GAIL (Ho &
Ermon 2016) (Figure ([). it converges to a single mode of behavior. this case, the mode is "compromise" between the values reflected by the demonstrations. Unfortunately. since other modes are ignored, the resulting imitation policy does not provide good performance guarantees relative to the entire demonstration distribution.
Impact of Noisy Demonstrations
Demonstrations
BC
PSD
Noisy demonstration trajectories (left), trajectory samples from a behavioral-cloned policy (center) and trajectory samples from policy learned using PSD (right).
Figure 12:
We next investigate imitation learning using a "noisier" set of demonstrations (Figure 12 left). Learning from these noisier demonstrations exacerbates the suboptimalities of behavior cloning (Figure 12 center), with far greater amounts of trajectory in the gray portions of the environment for many of the trajectories compared to the demonstrations (Figure T2] left) and behavior cloning from less noisy demonstrations (Figure Sp). The PSD policy (Figure 12] right) produces better tradc-off's of distance and obstacle with its trajectories, though with noticeable suboptimality compared the PSD policy learned from less noisy data (Figure (B). This suggests that incorporating a larger margin may be needed when learning from noisier demonstrations.
28


Reviews:
[{"review_id": "5uld7oXx7L", "reviewer": "Reviewer_3HFR", "summary": "The authors propose a method for imitation learning based on stochastic dominance over the distribution of possible rewards linearly induced by the demonstrations. To do so, the authors use an optimal transport relaxation. The authors evaluate their approach on a lava world, point bot and reacher.", "questions": "Question:\n- Several of the inverse RL baselines rely on interaction with the environment. My understanding from the paper is that PSD does not require further data collection. Would allowing interaction with the environment allow for stronger theoretical guarantees?\n\nScore increase:\n- More extensive experimental analysis, particularly on a less toy example where multiple reward objectives are desirable.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The proposed approach relies on a unique approach and outperforming the demonstrators across a wide range of reward functions is a powerful result with solid theoretical grounding. \n\nWeaknesses\n- One concern I had was the gap between the proposed motivation of the paper and the experiments. Generally, if we are assuming a mixture of behavior in our demonstrations, then why not simply match the distribution of demonstrations (e.g., with the very popular diffusion policy approach (Chi et al., 2023))? While there are practical cases where we might be interested in multiple reward functions (the authors mention risk-sensitivities), the experiments do not examine these settings. This is of concern, given that the reward function is assumed to be linear with the features. As such, it’s unclear to me whether the paper is actually more effective than the baselines at the types of problems it's attempting to address, rather than just toy constructions with randomly sampled rewards. \n- It’s fairly clear but worth mentioning that the proposed approach introduces a more complex optimization problem than in typical imitation learning. Consequently, it’s unlikely to have immediate practical implications.", "was_revised": true, "final_justification": "I stand by my original recommendation of accept. While I do think there is further opportunity to strengthen the paper with additional experiments, I believe the paper is theoretically grounded and offers an interesting new approach to the field. This is consistent with the reviews from the other reviewers.", "rebuttal": "Thank you for your positive review and insightful points/questions.\n\nWeakness #1–Matching behavior mixtures and diffusion policy: A major trade-off between diffusion-based approaches and PSD is that by requiring cost features, PSD provides performance guarantees over the corresponding space of cost functions. The denoising objectives of diffusion models aim to match demonstrations, including replicating any consistent suboptimalities of the demonstrator. This prevents stochastic dominance from being achieved.\n\nNote that under the perspective of PSD, the distinct sets of demonstrations in Figure 8a (ignoring/avoiding the obstacle) and different demonstrated modes more generally correspond to different reward functions, rather than being (nearly) “tied” in reward in a much more complex reward function (under the energy-based policy model perspective).\n\nWeakness #2–Complexity/practicality: We believe that the added computational complexity of solving an optimal transport problem to match rollouts with demonstrations is not very significant for existing imitation learning pipelines. However, our requirement of many training demonstrations in identical settings/tasks is a significant practical challenge in domains like autonomous driving in which only a single “real-world” demonstration is typically available.\n\nQuestion #1–Online/offline learning: To clarify, we employ a candidate set of trajectories in the fully realizable experiments and do not require any additional environment interaction. However, for the policy model optimization, we apply Algorithm 1 with on-policy samples from the imitator and a fixed set of demonstrations in our experiments. Offline/off-policy extensions when further interactions with a simulator/environment are expensive are possible by employing importance weighting in Step 1 of the algorithm. We do not explore this experimentally in this paper though. We expect that our method can be made more data efficient in the interactive imitation learning setting (e.g., with the learner able to query the demonstrator/expert in learner-chosen states/histories) by better focusing on the demonstrator’s trajectory distribution nearer to the Pareto frontier.\n\nWe plan to integrate these clarifications into our revision and would appreciate any additional questions or suggestions."}, {"review_id": "9cewUz1seZ", "reviewer": "Reviewer_EQ92", "summary": "The paper proposes an approach to imitation learning by introducing Pluralistic Stochastic Dominance (PSD) as an alternative to traditional expectation-based or risk-sensitive objectives. Rather than matching average performance, PSD ensures that the learned policy provides a better distribution over reward outcomes than the demonstrator for all conical sum reward functions. This is achieved via an optimal transport-based formulation that matches demonstration and imitation trajectories while enforcing a form of reward dominance. \nThe paper provides a theoretical formulation of PSD and its connection to stochastic dominance, a practical relaxation of the intractable optimization using matched subdominance minimization. The paper also presents learning algorithms for both fully realizable and policy-model-based settings and a suite of empirical evaluations (grid world, Point Bot, Reacher) showing that PSD-based methods outperform or match baseline approaches in both Pareto and stochastic dominance metrics.", "questions": "1. How does the approach handle or adapt to situations where no explicit reward features are available, as in high-dimensional visual domains? Would it be feasible to integrate learned representations into the PSD framework?\n\n2. While the outer maximization over reward weights is convex, it is noted to be NP-hard. Can the authors provide more insights into how this scales in practice for high-dimensional feature spaces?\n\n3. The method relies on a subdominance margin (e.g., β = 0.5 in experiments). How sensitive are the results to this choice, and what guidance can be offered for tuning this parameter?\n\n4. InfoGAIL* is included as a strong baseline. Could the authors provide further discussion on whether PSD could be augmented with latent variable models for additional gains, or does PSD render such modeling unnecessary?\n\n5. Since policy optimization relies on subdominance-derived signals rather than reward gradients, how stable is learning over time? Any failure cases or instability patterns observed?\n\nEvaluation Criteria Shift:\nA rebuttal addressing questions 1, 2, and 3, especially regarding how PSD could generalize beyond known reward features and scale computationally, would significantly increase confidence in broader applicability and lead to a higher score.", "limitations": "The paper explicitly discusses core limitations:\n- it assumes access to known reward features (limiting generalization).\n- current methods are restricted to deterministic dynamics; extending to stochastic settings is left for future work.\n- the computational hardness of exact PSD verification is acknowledged and addressed via approximations.\nFurther discussion of sample efficiency, especially under high-dimensional state/action spaces, and the feasibility of deployment in complex real-world environments would improve the section.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: Strong theoretical grounding: PSD is rigorously defined and connected to prior work in optimal transport and stochastic dominance. Derivation of tractable approximations via convex optimization is well-justified and novel. Experiments are well-controlled and convincingly demonstrate the benefits of the approach across multiple settings and metrics.\nWhile the theory is sound, the computational burden of the proposed methods is only partially addressed, particularly the outer maximization over reward functions. The policy optimization algorithm relies on handcrafted reward features, which limits applicability in more general, high-dimensional settings.\n\nClarity: The paper is clearly written and well-organized, with useful diagrams (e.g., Figures 1, 2, 4, and 8) illustrating core concepts. The step-by-step exposition of theoretical constructs, optimization strategy, and experimental setup is effective.\nSome key terms (e.g., “supporting behaviors”, “matched subdominance”) are introduced with heavy notation before intuitive descriptions are provided. The transition from full PSD definitions to the practical relaxation could be clarified with a higher-level overview or example before diving into math-heavy formulations.\n\nSignificance: The concept of pluralistic dominance fills a critical gap in imitation learning by addressing the preservation of diverse high-quality behaviors rather than average performance. Potential for significant impact in applications requiring robustness or multimodal imitation, such as robotics, autonomous driving, and human-agent interaction.\nGeneralization beyond hand-crafted reward features is mentioned as future work but limits current real-world applicability. Experimental validation is compelling but relatively limited in scale; larger-scale benchmarks would strengthen the claim of practical impact.\n\nOriginality: Introduction of pluralistic stochastic dominance as a criterion in imitation learning is novel and conceptually rich. The proposed method differentiates itself clearly from previous approaches like GAIL, MaxEnt IRL, and BROIL in both objective and outcomes.\nSome connections to mixture model-based methods (e.g., InfoGAIL) could be more deeply explored, especially given their relevance to handling multimodal demonstrations.", "was_revised": true, "final_justification": "I believe my questions have been addressed, I confirm my score, and I have completed the mandatory acknowledgement.", "rebuttal": "Thank you for your detailed review and insightful questions.\n\n\nQuestion #1–Cost representation learning: Learned cost features could immediately be fine-tuned (e.g., adversarially) since our objective is differentiable with respect to cost feature model parameters. Using stochastic dominance as the basis for representation learning more broadly is of great interest, but beyond the scope of the current work.\n\nQuestion #2–Computational hardness: We avoid the computational challenges of the original optimization (5) using the relaxation in (6). \n\nQuestion #3–β choice: Larger β parameters encourage the algorithm to try to outperform the demonstrations more. Appropriate parameter choice depends on the scale of the feature values, the range of possible feature values in the environment, and the quality of the demonstrations. For example, if β is small and demonstrations are very suboptimal, the training loss can be driven to zero without “pushing” the learned policy towards the Pareto frontier. From this perspective, β should be adapted so that each cost feature contributes a balanced, non-zero amount to the overall training loss.\n\nQuestion #4. The optimal transport solution of PSD generally renders latent variables unnecessary. We view this as a significant advantage of PSD, since modeling this latent structure is often challenging. Under PSD, demonstrations could be split into separate batches to decrease the optimal transport complexity if latent task variables could be reliably inferred.\n\nQuestion #5. Like reward-based (stochastic) policy optimization, our approach suffers from the high variances of each learning step, but the optimization generally trends towards improvement over time, and the model with the best training performance during training is generally reliable.   \n\nWe plan to incorporate this discussion into our revision and would appreciate any additional questions or suggestions."}, {"review_id": "2Qc2kEgwYl", "reviewer": "Reviewer_X6Xw", "summary": "This paper identifies and addresses a key deficiency in imitation learning (IL): the tendency for algorithms to learn policies that match the *average* performance of demonstrators, potentially leading to behaviors that are a compromise between different expert strategies and are not truly preferred by any single demonstrator. The authors propose a more robust and intuitive objective: **Pluralistic Stochastic Dominance (PSD)**. The goal is to learn an imitator policy whose distribution of rewards is provably better than the demonstrators' distribution, not just in expectation, but across all reward quantiles and for an entire class of plausible reward functions (conical sum rewards).\n\nTo make this concept practical, the work forges a strong connection between PSD and optimal transport (OT). It shows that achieving PSD is equivalent to minimizing a worst-case OT cost. Since solving this directly is intractable, the authors derive a computationally efficient, convex upper bound they term \"Matched Subdominance Minimization,\" which can be solved as a linear program. The paper presents algorithms for both a tabular \"fully realizable\" setting and a deep reinforcement learning setting where the objective serves as a loss for policy gradient updates. The experimental results on several environments convincingly show that the PSD approach successfully preserves distinct behavioral modes from the demonstration set while improving upon them, outperforming a suite of strong IL baselines on novel and well-suited evaluation metrics.", "questions": "1. A point of clarification regarding the implementation details for the fully-realizable setting. Line 268 mentions that the training demonstrations are matched with candidate trajectories by \"solving a quadratic program based on Eq. (10)\". However, there is no Equation (10) in the paper. Is this referring to the optimization problem in Definition 3.7 (line 203), perhaps with the L2 regularization on the transport plan `γ` making it a quadratic program?\n\n2. Second, concerning the policy model optimization (Algorithm 1), the OT problem is solved in the inner loop to generate training signals. Could the authors comment on the practical computational overhead of this step, especially as the number of demonstration $N$ and imitator $M$ samples grows?", "limitations": "The choice of experimental environments is logical and serves to clearly illustrate the core properties of the PSD framework. However, to fully establish the scalability and robustness of the policy optimization algorithm, it would be compelling to see results on more standard, high-dimensional continuous control benchmarks. For instance, tasks from the MuJoCo suite (e.g., Humanoid) would challenge the policy optimization with more complex dynamics and action spaces, providing stronger evidence of its general applicability in deep RL settings.\n\nFurthermore, the true power of pluralistic imitation would be most apparent on large-scale, heterogeneous datasets that are becoming standard in offline RL and IL research. While the paper's experiments use synthetically generated demonstrations with clear modes, real-world datasets are often more \"messy.\" It would be highly impactful to evaluate PSD on such data. For instance, the **D4RL benchmark** contains mixed-quality data from diverse sources (e.g., scripted policies, human players, and partially trained agents), which is a perfect use-case for a method designed to improve upon, but not average over, demonstrations. More recent efforts like **MimicGen** or the **IKEA ASM** datasets provide demonstrations from multiple humans with inherent variability in strategy and skill. Testing PSD in these settings would provide powerful evidence of its ability to handle the complexity of real-world demonstration data, moving beyond the clean modes in the current experiments and truly validating its \"pluralistic\" promise.\n\nFu, J., Kumar, A., Nachum, O., Tucker, G., & Levine, S. (2020). D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219.\n\nMandlekar, A., et al. (2023). Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17596.\n\nBen-Shabat, Y., Yu, X., Saleh, F., Campbell, D., Rodriguez-Opazo, C., Li, H., & Gould, S. (2021). The ikea asm dataset: Understanding people assembling furniture through actions, objects and pose. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 847-859).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "#### Strength\n\nThis shift from matching moments to dominating distributions is a significant conceptual contribution to the field.\n\nThe theoretical development is rigorous and elegant. The authors do an excellent job of connecting the high-level goal of PSD to a practical algorithmic solution through the lens of optimal transport. The relaxation from the intractable worst-case OT problem to the tractable \"Matched Subdominance Minimization\" is clever and provides the key that makes this framework usable. The presentation is exceptionally clear, with Figures 1 and 2 being particularly effective at providing intuition for the core concepts. The experimental validation is also a high point; the authors not only compare against relevant and strong baselines but also propose new evaluation metrics (stochastic and Pareto dominance rates) that directly measure what the paper claims to achieve, which is a hallmark of thoughtful research.\n\n#### Weaknesses\n\nDespite these significant strengths, the work has a few limitations that warrant discussion. The most critical is the framework's reliance on pre-defined, hand-engineered reward features. This assumption, while common in the classic IRL literature, is a major departure from many modern IL approaches (like GAIL) that can learn directly from high-dimensional states (e.g., pixels). This limits the immediate applicability of the method to domains where such features are not readily available. The authors rightly acknowledge this as an area for future work. Additionally, the methodology relies on solving an optimal transport problem in the learning loop, which may present scalability challenges for very large numbers of demonstration and imitator trajectories compared to methods that rely on simpler adversarial training.", "was_revised": true, "final_justification": "This paper introduces a novel and insightful method for imitation learning that goes beyond fitting the average of demonstrations. The derivation and analysis are rigorous, and properly supported by carefully designed experiments. This work receives positive feedback from all reviewers.\n\nThus I keep my rating at 5 - Accept", "rebuttal": "Thank you for your careful reading of our paper and insightful questions.\n\nWeakness #1–Available cost features: We agree that requiring cost features limits the applicability of our approach compared to methods like GAIL or diffusion-based techniques. However, we also note that in many “real” applications of imitation learning for engineered systems (e.g., autonomous vehicles, robotics), cost features are available and widely employed. \n\n\nQuestion #1–Missing quadratic equation: Thank you for pointing out this error in the writing. You are indeed correct that this is meant to refer to the equation in Def. 3.7, and L2 regularization yields a QP.\n\nQuestion #2–Optimal transport complexity: For our experiments (n=10, m=30), we use an off-the-shelf QP solver (MOSEK), which has negligible run time (<20ms) compared to the m=30 rollouts. In theory, for uniform distributions, the Hungarian algorithm is O(m^3) time for the exact solution, but we find MOSEK to scale at O(mn) in practice from m*n of 10^3 to 10^6. If applied at larger scales, specialized algorithms with ε error rate guarantees require Õ(mn/ε^2) time complexity (Dvurechensky et al. 2018). However, we would favor taking randomized subsets of the demonstrations (for very large n) as training batches to avoid this potential computational bottleneck, assuming the true set of distinct modes is only moderately large. Importantly, apart from the O(mnk) complexity of computing the pairwise costs, none of these complexities depends on the cost feature set size (k).\n\nWe appreciate the pointers to environments/datasets that are well-aligned with the motivations of PSD and plan to investigate these in future work. We will aim to alleviate concerns about OT being a bottleneck based on your concerns."}]

Author rebuttal:

Meta-review: The authors proposed a method for imitation learning based on stochastic dominance over the distribution of possible rewards linearly induced by the demonstrations. To do so, the authors use an optimal transport relaxation. The authors evaluate their approach on a lava world, point bot and reacher.

The reviewers found that the paper has made significant contributions. In particular, they found that the shift from matching moments to dominating distributions is a significant conceptual contribution to the field. The paper is also strongly theoretically grounded. PSD is rigorously defined and connected to prior work in optimal transport and stochastic dominance. Derivation of tractable approximations via convex optimization is well-justified and novel. The experiments are also strong: the proposed approach outperforms the demonstrators across a wide range of reward functions. Finally, the paper is clearly written and well-organized,

The reviewers would encourage the authors to expand the applicability of the proposed framework: currently, the framework is reliant on a set of pre-defined, hand-engineered reward features.
Decision: Accept (spotlight)