From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning
Authors: Pusen Dong
Tianchen Zhu
Yue Qiu
Haoyi Zhou
Jianxin Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Safe RL
Multimodal Learning
Temporal Credit Assignment
Conditioned Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Safe reinforcement learning (RL) requires the agent to finish a given task while obeying specific constraints. Giving constraints in natural language form has great potential for practical scenarios due to its flexible transfer capability and accessibility. Previous safe RL methods with natural language constraints typically need to design cost functions manually for each constraint, which requires domain expertise and lacks flexibility. In this paper, we harness the dual role of text in this task, using it not only to provide constraint but also as a training signal. We introduce the Trajectory-level Textual Constraints Translator (TTCT) to replace the manually designed cost function. Our empirical results demonstrate that TTCT effectively comprehends textual constraint and trajectory, and the policies trained by TTCT can achieve a lower violation rate than the standard cost function. Extra studies are conducted to demonstrate that the TTCT has zero-shot transfer capability to adapt to constraint-shift environments.

From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning
Pusen Dong', Tianchen Zhu', Yue Qiu', Haoyi Zhou Jianxin Li
Beihang University, Beijing, China
ZZhongguancun Laboratory, Beijing, China {dongps, zhutc, qiuyue, zhouhy, ]ijx)Bact. http//acttuuaa....
Abstract
Safe reinforcement learning (RL) requires the agent to finish given task while obeying specific constraints. Giving constraints in natural language form has great potential for practical scenarios due to its flexible transfer capability and accessibility. Previous safe RL methods with natural language constraints typically need to design cost functions manually for each constraint, which requires domain expertise and lacks flexibility. In this paper, we harness the dual role of text in this task, using it not only to provide constraint but also as training signal. We introduce the Trajectory-level Textual Constraints Translator (TTCT) to replace the manually designed cost function. Our empirical results demonstrate that TTCT effectively comprehends textual constraint and trajectory, and the policies trained by TTCT can achieve lower violation rate than the standard cost function. Extra studies are conducted to demonstrate that the TTCT has zero-shot transfer capability to adapt to constraint-shift environments.
Introduction
In recent years, reinforcement learning (RL) has achieved remarkable success in multiple domains, However, deploying RL in real-world scenarios still remains challenging. Many real-world decision-making applications, such autonomous driving require agents to obey certain constraints while achieving the desired goals. To learn a safe constrained policy, some safe RL works 2
such as go game 37 38 and robotic control [21 45)
B
6 have proposed methods to maximize the reward while minimizing the constraint violations after training or during training.
33
4
49
39
48
However, several limitations prevent the existing safe RI. methods" widespread use in real-world applications. Firstly, these methods often require mathematical or logical definitions cost functions, which require domain expertise (Limitation 1). Secondly, their cost function definitions are frequently specific to a particular context and cannot be easily generalized to new tasks with similar constraints (Limitation 2). Lastly, most current safe RL methods focus constraints that are logically simple, typically involving only one single entity or one single state (28), which can't 't represent the worl-world safety requirements and lack universality (Limitation 3).
is a promising approach to overcome I Limitation and 2 because natural language allows for flexible, high-level expression of constraints that can easily adapt to different scenarios. Regarding Limitation 3, previous approaches primarily employ what call the single state/entity textual constraint. The single-statc/entiyy textual con- straint focuses solely on constraints related to one specific state or entity, limiting the ability to model complex safety requirements in real-world scenarios. Many safety requirements involve interactions and dependencies among multiple states or entities over time. By only addressing a single state or
Using natural language to provide constraints
46
29
20
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 1: Comparison of trajectory-level constraints and previous single state/entity constraints. Universal Trajectory-level constraints can model any constraint requirements in real-world scenarios. But single state/entity constraint can only model the constraint requirements on individual state/entity.
Single state/entity constraint (previous) Don't drive car. (single entity)
Trajectory- constraint (ours)
Don't drive car after you drink wine. (multi entities)
Don't 't touch lava. (single state)
Don't touch lava more that three times. (multi states)
Don't step in the gassed area. (single state)
Don't stay in the gas area for more than 5 min- utes, your gas mask will fail. (multi states)
entity, these constraints fail to capture the dynamic relationships and temporal aspects that are crucial for ensuring safety in complex environments. So we suggest using a more generalizable constraint type trajectory-level textual constraint. The trajectory-leve textual constraint is more universal constraint with complex logic semanties involving multiple states/entities. It is nonlinear combina- tion of multiple entities or multiple environment states and can model any constraint requirements in real-world scenarios. The trajectory-level constraints natural language form are the highly abstract expression of the agent's behavior guidelines, serving as more natural and straightforward way to introduce constraints. Notably, the set of trajectory-level constraints encompasses single-state con- straints, any single-state constraint can be reformulated as equivalent trajectory-Ievel constraint. Examples of trajectory-Ievel textual constraint and single statc/entity textual constraint are presented in [
Employing trajectory-level textual constraints across the entire trajectory poses two significant challenges. Firstly, determining whether an RL agent violates textual constraint over trajectory non-trivial, it needs to have the perception of the historical states and actions observed along the trajectory (Challenge 1). Secondly, the trajectory-leve safety problem is susceptible to sparse cost (34], where cost is only imposed when the agent violates textual constraints at the final time step, making it challenging for the agent to learn which actions contribute to gradual escalation of risk (Challenge 2). For instance, the agent needs to learn constraints like, "Don't 't touch lava after you touch grass." Without intermediate feedback, however, it struggles to understand how early actions, such as stepping on grass, contribute to eventual violations.
To address Challenge 1. we propose a general approach to align the trajectory's factual logic with the text's semantic logic, eliminating the need for manual encoding or separate models for each type of constraint. Our method employs a sequence model [4] [5] for modeling agent historical interactions with the environment, and a pre-trained language model (LM) 7 to comprehend natural language constraints. We then maximize the embedding similarities between matching pairs (trajectory, text) and minimize the embedding similarities between non-matching pairs using a contrastive learning approach [25] similar to CLIP [30]. By calculating the similarity between the textual embeddings of the constraint and the trajectory, we can predict whether a constraint is violated in this trajectory. Our method uniquely leverages text both a source of constraints and unified supervisory signal for trajectory encoding. In this dual role, text not only provides constraints but also guides the training process, enabling the model to naturally handle diverse semantic constraints without requiring specific model adjustments for each type. This design allows for a more flexible and generalizable system, significantly simplifying the handling of complex, multi-dimennionall constraints.
superrs
In addition, to address the issue of cost sparsity (Challenge 2), we introduce a method for temporal credit assignment [40]. The proposed approach involves decomposing the one episodic cost of the textual constraint into multiple parts and allocating them to each atate-action pair within the trajectory. This method offers denser cost signals regarding the relationship between the textual constraint and the agent's every action. It informs the agent which behaviors are risky and which are safer, thereby enhancing safety and aiding model performance.
Our experiments demonstrate that the proposed method can effectively address Challenge 1 and Challenge 2. In both 3D navigation [19 and 2D grid exploration [46] tasks, agents trained using our method achieve significantly lower violation rates (up to 4.0x) compared to agents trained with ground-truth cost functions while maintaining comparable rewards and more importantly, our method obtains the Pareto frontier [00. In addition to this, our method has zero-shot adaptation capability to adapt to constraint-shift environments without fine-tuning.

---

2
Related Work
Safe RL. Safe RL aims to train policies that maximize reward while minimizing constraint violations
[3]. In prior work, there are usually two ways to learn safe policies: (1) consider cost as one of the optimization objectives to achieve safety 48
39
44
33], and (2) achieve safety by I 47]. These typical safe RL
leveraging external knowledge (e.g. expert demonstration) [35 32
algorithms require either human-defined cost functions or human-specified cost constraints which are
unavailable in the tasks that constraints are given by natural language.
RL with Natural Language. Prior works have integrated natural language into RL improve generalization or learning efficiency in various ways. For example. Hermann et al. (14] studied how to train an agent that can follow natural language instructions reach specific goal. Additionally, natural language has been used to constrain agents to behave safely. For instance, Prakash et al. [2]] trained a constraint checker to predict whether natural language constraints are violated. Yang et al. [46] trained constraint interpreter to predict which entities in the environment may be relevant to the constraint and used the interpreter to predict costs. Lou al. (28] used pre-trained language models to predict the cost of specific states, avoiding the need for artificially designed cost functions. However, previous methods cannot uniformly handle textual constraints with one framework, which limits their applicability.
Credit Assignment in RL. Credit assignment studies the problem of inferring the true reward from the designed reward. Prior works have studied improving sample efficiency of RL algorithms through credit assignment, for example by using information gain [17], as an intrinsic bonus reward aid exploration. Goyal et al. [2] proposed the use of natural language instructions to perform reward shaping to improve the sample efficiency of RL algorithms. Liu et al. (26 learned decompose the episodic return as the reward for policy optimization. However, to the best of our knowledge, our work is the first apply credit assignment to safe RL.
3 Preliminaries
Problem formulation. Trajectory-level constraint problem can be formed as the Con- strained Non-Markov Decision Process (CNMDP) [3] 43). and can be defined by the tuple R, R,,,Y, Y,Y, T">. Here S represents the set of states, A represents the set of actions, T represents the state transition function, R represents the reward function, and y E (0, 1) represents the discount factor. In addition, Y represents the set of trajectory-level textual constraints (e.g., "You have 10 HP, you will lose 3 HP every time you touch the lava, don't die."), which describes the constraint that the agent needs to obey across the entire trajectory. C' represents the cost function determined by E Y. T represents the set of historical trajectories.
<S, A,
RL with constraints. The objective for the agent is to maximize reward while obeying the specified textual constraint as much as possible (27]. Thus, in our task setting, the agent needs to learn a policy T: S Y 7" + P(A) which maps from the state space S, textual constraints Y and historical trajectories 7* to the distributions over actions A. Given a y. we learn a policy æ that maximizes the cumulative discounted reward JR while keeping the cumulative discounted cost (average violation rate) Jc below a constraint violation budget Bc
max JR(T)= = E. 1 R(s))
s.t. (=(x)= = E, Σ a4, Bc
(I)
6-0
1=0
Here Be(y) and C(st, U. are two functions both depending on textual constraint y. Te represents the historical trajectory at time step t.
Episodic RL. Similar to the task with episodic rewards [4] in our task setting, a cost is only given at the end of each trajectory 7 when the agent violates the textual constraint y. In other words, before violating y. the cost C(st, ae, V, Tτ) = O for all t T. For simplicity, we omit the discount factor and assume that the trajectory length at most T that we can denote ar as the final action that causes the agent to violate y without further confusion. Therefore, the constraint qualification of the objective in RL with constraints becomes J ) E. [C(TT, uT, y/ TT)] N Bc(v). Due to the sparsity of cost, a large amount of rollout trajectories are needed to help the agent distinguish the
subtle effects of actions on textual constraint (24]. This situation will become more serious when trajectory-level constraints complex and difficult to understand.
3

---

Text- Trajerraey Alignment
Assigmmen
Witwiiting trrugh veri
Ephedi:
Cast Assignment
CA Lown
Trautay Earoder
Use for predicting cost
Tea
Cist Assiguazett
Hulk
Triletey
MCC
Groandorahh
Hediece 2
Figure 1: TTCT overview. TTCT consists of two training components: (1) the text-trajectory alignment component connects trajectory to text with multimodal architecture, and (2) the cost assignment component assigns cost value to each Mate-action based on its impact on satisfying the constraint. When training RL policy, the text-trajectory alignment component used to predict whether trajectory violates given constraint and the cost assignment component is used to assign non-violation cost.
4 TTCT: Trajectory-leve Textual Constraints Translator
In this section, we introduce our proposed framework TTCT ( (Trajectory-level Textual Constraints Translator) as shown in Figure I TTCT has two key components: the text-trajectory alignment component and the cost assigument component. The text-trajectory alignment component is used to address the violations prediction problem. The cost assignment component is used address the sparse cost problem.
4.1 Text- TeajeTrajycyory Alignment Component
We propose component to learn from offline data predict whether given trajectory violates textual constraints. The core idea of this component is to learn trajectory representations under textual supervision and connect trajectory representation to text representation. If the distance between the two representations in the embedding space is sufficiently close, we can consider that the given trajectory violates the constraint. Our approach does not require modeling entities of the environment like previous work, such as [46), which involves labeling hazardous items artificially in every observation. Instead, we model this task as trajectory-texx multimodal learning problem. Hence, our method can learn trajectory representations and text representations from the pairs (trajectory, trajectory-levee textual constraint). We believe that learning from the supervision of natural language could not only enhance the representation power but also enable flexible zero-shot transfer [30].
Formally, given batch of N (trajectory т, trajectory-level textual constraint pairs. For each pair, the trajectory corresponds to the text, indicating that the given trajectory violates the given textual constraint. The trajectory can be defined as t (51, 01 82, 02, 8T-1, 0T-1, 1,7--18777 8T, sT, where T' is the step at which the textual constraint is first violated by the trajectory. Here 8, is a da- dimensional observation vector. Each state in the trajectory is processed by a state encoder obtain a representation uf', also action is processed by an action encoder to obtain a representation vg. Then, we concatenate of and of to obtain a vector representation U, for each state-action pair. After that, we learn separate unimodal encoders The trajectory encoder gr utilizes a causal transformer extract the trajectory representation from the input state-action representation sequence (wt)
and for the trajectory and textual constraint, respectively,
H1, H1, Hz, Hy, Hy-1, Hr = gr (r((())
(2)
where He is a dи dimensional vector. The final embedding Hy is used as the representation for the entire trajectory. Specifically, the causal Transformer processes the trajectory sequence by maintaining left-to-right context while generating embeddings. This allows the model to capture temporal dependencies within the trajectory and obtain the embeddings for time steps before T. The textual constraint encoder gC used to extract features that are related to the constraints and it could be one of a wide variety of language models:
L ==))(y),
(3)

---

where L is dp dimensioonll vector representation. Then we define symmetric similarities between the modalities with cosine distance:
simy (T. = exp(a) + Hr. L
LT. HT |L]| Hrll
|| || simr
exp(a)*
(4)
where a learnable parameter and T means that use last embedding of trajectory calculate similarity. And we use softmax to calculate trajectory teat. and text - trajectory similarities scores:
exp(simp(T;w0) Σf=1 exp(simr(T:1))) P. (3) (v) exp(simr T:))
P. (T)
(5)
Σÿ-1 exp(simr(1)) Tj))
Let Let (T). (T), 10 (y) indicate the ground-truth similarity scores, where the negative pair (trajectory doesn't violate textual constraint) has a probability of O and the positive pair (trajectory violate textual constraint) has probability of In our task setting, a trajectory can correspond to multiple textual constraints, and vice versa. For example, two textual constraints such as "Do not touch lava" and "After stepping on water, do not touch lava" might both be violated by a single given trajectory. This maan--oomanne relationship between trajectories and textual constraints implies that the same textual constraints (or different textual constraints with the same semantics) can apply to multiple trajectories, while single trajectory may comply with several textual constraints. So there may be more than one positive pair in 9," (T) and g." (y). Therefore, we use Kullback Leibler (KL) divergence (23) as the multimodal contrastive (MC) loss to optimize our encoder similar to 42
DMC = (KLP))))
+ KL(p" softmaxxx(( (b))) (6)
where D is the training set.
In addition to the multimodal contrastive (MC) loss, we also introduce a -ithin-trajectory (WT) loss. Specifically, suppose we have a trajectory's representation sequence (H1, H2, ...... (T--)): Hr) and its corresponding textual constraint embedding L, we can calculate cosine similarity sim, (T, between embedding H, and L using Equation Then we can calculate similarity scores within the trajectory:
pī (v) exp(sim((T.V)
Σk= exp(simx(T,y)
Different from (y) in Equation 5 which measures similarity scores across N trajectories, Equation ils used to measure the similarity scores of different time steps within trajectory. The reason for doing this is that the textual constraint is violated at time step T, while in the previous time steps the constraint is not violated. Therefore, the instinct is to maximize the similarity score between the final trajectory embedding Hy and the textual constraint embedding L, while minimizing the similarity score between all previous time step embeddings (H1, H2, Hs. H(r-1)) ) and the textual constraint embedding L. Based on this instinct, we introduce within-trajectory (WT) loss:
1 T-1 T
LWT B(r.o)-DI DI
Σ
log(1 (v) + log()>()))))
(8)
where the first term is responsible for minimizing the similarity score for the embedding of time steps before T' in the trajectory sequence, while the second term is responsible for maximizing the similarity score for the embedding of time step T.
By combining these two losses LWT, we can train a text encoder to minimize the distance between embeddings of semantically similar texts, while simultancously training trajectory encoder to minimize the distance between embeddings of semantically similar trajectories. Crucially, this approach enables to align the text and trajectory embeddings that correspond to the same semantic constraint, fostering a cohesive representation of their shared meaning, and further determining whether the trajectory violates the constraint by calculating embedding similarity.
4.2 Cost Assignment Component
After solving the problem of predicting whether a given trajectory violates a textual constraint, there still the issue of cost sparsity. Motivated by the works of temporal credit assignment (26), propose
5

---

component to capture the relationship between the state- saute-action pair and the textual constraint and assign a cost value each state-action based on its impact on satisfying the constraint.
Specifically, suppose we have a (trajectory т, textual constraint y) pair and its representation {H,) t=1 L) obtained from text- exxe-trajectory alignment component. The textual constraint representation L is processed by episodic-cost prediction layer F to obtain a predicted episodic cost Ö(y) sigmoid( (L) for the entire trajectory. We expect the episodic cost can be considered as the sum of cost on all non-violation state-action pairs: C(y) = ΣiI č(s,, az y, T²). To evaluate the significance of each timestep's ss action relative textual constraints, we employ attention mechanism:
99)
Here we regard the text representation as the query, each time step's representation the trajectory as the key, and compute the attention score based on the cosine similarity metric. After that, we use the sigmoid function to make sure the score falls within the range of 0 to 1. Each attention score e, quantifies the degree of influence of the Then we obtain an seltunnce-bssee representation H H, To predict the cost á st, a, y: Tt), incorporate feed-forwan layer called the cost assignment layer Fe and output the predicted non-violation single step cost as:
pair (s),ae) ac) violating the textual constraint.
s(s at y.
(F (Cooncctt
(10)
The loss function for measuring inconsistency of the episodic cost Egy) and the sum of non-violation single step costs ē(st, at, y, h4) can be formed as:
Lc = (G.= Σ -
(1I)
This mutual prediction loss function LC is only used to update the episodic-cos prediction layer and cost assignment layer, and not to update the parameters of the trajectory encoder text encoder during backpropagation. This helps ensure the validity of the predictions by preventing overfitting or interference from other parts of the model.
The effectiveness of this component comes from two main sources. First, the text-trajectory alignment component projects semantically similar text representations to nearby points in the embedding space, allowing the episodic-cost prediction layer to assign similar values to embeddings with close distances. This aligns with the intuition that textual constraints with comparable violation difficulty should yield similar episodic costs. Second, the cost assignment layer leverages the representational power the text-trajectory alignment component to capture complex relationships between state-action pairs and constraints, enabling accurate single-step cost predictions.
In our experiment, we simultancously train the text-trajectory alignment component and cost assign- ment component end-to-end with a uniform loss function LITCT:
LTTCT LMC + LWT + LCA.
(12)
By doing this, we can avoid the need for separate pre-training or fine-tuning steps, which can be time- consuming and require additional hyperparameter tuning. Also, this can enable the cost assignment component to gradually learn from the text-trajectory alignment component and make more accurate predictions over time.
In the test phase, at time step i we encode trajectory т, and textual constraint y with Equation 3 and Equation t1 obtain the entire trajectory embedding H, and text embedding L. Then we calculate distance score sim(Te Tt. y) using Equation 4 The predicted cost function è is given by:
L
if sim( .) 8, otherwise
ė= =
(13)
sigmoid ( (Concat(HH; L)),
where 3 is hyperparameter that defines the threshold of the cost prediction. ē= 1 indicates that the TTCT model predicts that the textual constraint is violated. Otherwise, if the textual constraint not violated by the given trajectory, assign a predicted cost using Equation I0

---

Policy Training
Our Trajectory-leve textual constraints Translator framework is general method for integrating free-form natural language into safe RL algorithms. In this section, we introduce how to integrate our TTCT into safe RL algorithms so that the agents can maximize rewards while avoiding early termination of the environment due violation of textual constraints. To enable perception of historical trajectory, the trajectory encoder and text encoder are not only used as frozen plugins 9r and gc for cost prediction trajectory. This allows the agent to take into account historical context when making decisions. To further improve the ability to capture relevant information from the environment, we use LoRA [T8] to fine-tune both the o and 9c during policy training. The usage of gT 9c and gr, 9C is illustrated in Appendix A. Figure 8
also as trainable sequence models yT and 9C for modeling historical
Formally, let's s assume we have a policy т with parameter & to gather transitions from environments. We maintain a vector to record the history state-action pairs sequence, and time step we use 9T and 9 to encode T--1 and textual constraint y so that we can get historical context representation H₂-1 and textual constraint representation L. The policy selects an action a; Tyg (o,, H1-1 L) to interact with environment to get new observation 0z-1. And we update with the new state-action pair (0t,0) (+) to get тr. With 7 and T., & can predicted according to Equation 13 Then we store the transition into the buffer and keep interacting until the buffer is full. In the policy updating phase, after calculating the specific loss function for different safe RL algorithms, we update the policy π with gradient descent and update 97. 9c with LoRA. It is worth noting that gr and JC are not updated during the whole policy training phase, as they are only used for cost prediction. The pseudo-code and more details of the policy training be found in Appendix A.4
Experiments
Our experiments aim answer the following questions: (1) Can our TTCT accurately recognize whether an agent violates the trajectory-level textual constraints? (2) Does the policy network, trained with predicted cost from TTCT, achieve fewer constraint violations than trained with the ground truth cost function? (3) How much performance improvement can the cost assignment (CA) component achieve? (4) Does our TTCT have zero-shot capability to be directly applicable constraint-shi environments without any fine-tuning? We adopt the following experiment setting to address these questions.
(a) Hazard- World-Grid
(b) SafetyGoal
(c) LavaWall
Figure 2: (a) One layout in Hazard- Hazard-World-Grit [46), where orange tiles lava, blue tiles water and green tiles are grass. Agents need to collect reward objects in the grid while avoiding violating our designed textual constraint for the entire episode. (b) Robot navigation task SafetyGoal that is built in Safety-Gymnasiun [19]. where there are multiple types of objects in the environment. Agents need reach the goal while avoiding violating our designed textual constraint for the entire episode. (c) LalaWall [5], task has the same goal but different hazard objects compared to Hazard- Wortrd-rrrrdddddd
6.1
Setup
Task. We evaluate TTCT on two tasks (Figure
2 (a.b)): 2D grid exploration task Hazard World- Grid (Grid) [6) and 3D robot navigation task SafetyGoal (Goal) [9]. And we designed over 200 trajectory-level textual constraints which can be grouped into 4 categories, to constrain the agents. detailed description of the categories of constraints will be given in Appendix A. Different from the default setting, in our task setting, when a trajectory-level textual constraint violated, the
7

---

environment is immediately terminated. This is a more difficult setup than the default. In this setup, the agents must collect as many rewards as possible while staying alive.
PPO SPP_Lagranging (PPO_ Lag__aag 33 CPPO_PID 33 FOCOPS 44 PPO does not consider constraints and simply aims to maximize the average reward. We use PPO to compare the ability of our methods to obtain rewards. As for the last three algorithms, we design two training modes for them. One is trained with standard ground-truth cost, where the cost is given by the human-designed violation checking functions, and we call it ground-truth (GC) mode. The other is trained with the predicted cost by our proposed TTCT, which we refer to as cost prediction (CP) mode. More information about the baselines and training modes can be found in Appendix (aapddixxA.IJ..... [A.]
Baselines. We consider the following baselines: PPO 36
Metrics. We take average episodic reward (Avg. R) and average episodic cost (Avg. C) as the main comparison metrics. Average episodic cost also be considered the average probability of violating the constraints. The higher Avg. R, the better performance, and the lower Avg. C, the better performance.
28
R 1/
Avg. R
0.5
Avg.
6.
60
FOCOPS
CPPO
FOcOPs
FOCOPS
CPPO FOCCeS
(a) Hazard- World-Grid
(b) SafetyGoal
Figure 3: Evaluation results of our proposed method TTCT. The blue bars are our proposed cost prediction (CP) mode performance and the orange bars are the ground-truth cost (GC) mode performance. The black dashed lines are PPO performance. (a) Results on Hazard- HazzrrdWorld-Gin World-Grid task. (b) Results on SafetyGoal task.
6.2 Main Results and Analysis
The evaluation results are shown in Figure and the learning curves are shown in Figure 4 We can observe that in the Hazard-World- World-Grid task, compared with PPO, the policies trained with GC can reduce the probability of violating textual constraints to some extent, but not significantly. This is because the sparsity of the cost makes it difficult for an agent to learn the relevance of the behavior to the textual constraints, further making it difficult find risk-avoiding paths of action. In the more difficult SafetyGoal task, it is even more challenging for GC-trained agents to learn how to avoid violations. In the CPPO_ PPD and FOCOPS algorithms trained with GC mode, the probability of violations even rises gradually as the training progresses. In contrast, the agents trained with predicted cost can achieve lower violation probabilities than GC-trained agents across all algorithms and tasks and get rewards close to GC-trained agents.
These results show that TTCT can give an accurate predicted episodic cost at the time step when the constraint is violated, it can also give timely cast feedback to non-violation actions through the cost assignment component so that the agents can find more risk-averse action paths. And these results answer questions (1) and (2). The discussion about the violation prediction capability of
the text-trajectory component can be found in Appendix B.I The interpretability and case study of
the cost assignment component can be found in Appendix B.2
6.3 Ablation Study
To study the influence of the cost assignment component. We conduct an ablation study by removing the cost assignment component from the full TTCT. The results of the ablation study experiment are shown in Figure 4 We can observe that even TTCT without cost assignment can achieve similar performance as GC mode. And in most of the results if we remove the cost assignment component, the performance drops. This shows that our text trajectory alignment component can accurately predict the ground truth cost, and the use of the cost assignment component can further help us learn a safer agent. These results answer questions (3).
8

---

PPO
[algo]
[algo] CP (our)
[algo] with CP w/o CA
PPO Lag
CPPO PID
FOCOPS
PPO Lag
CPPO PID
FOCOPS
Ave.
Hazard World-Grid
(b) SafetyGool
Figure 4: Learning curve of proposed method TTCT. Each column is algorithm. The six figures on the left show the results of experiments on the Hazard-WorlddOGe task and the six figures on the right show the results of experiments the SafetyGoal task. The solid line is the mean value, and the light shade represents the area within one standard deviation.
770
770 CAPci
catch CPhAC
Avg. R
06 0.25
1.5
143
Arg.
Avg.
220
-0.0
CIORD
CP
Pao PID FOCOPS
60
PPO CPPO,PID FOCOPS
PPO CPPO PID FOCGPS
140 CPPO,FID FoCOes
(a) Hazard-World-ee World-Grid
(b) SafetyGoal
Figure 5: Ablation study of removing the cost assignment (CA) component. The blue bars are cost prediction (CP) mode performance with full TTCT and the orange bars is the cost prediction (CP) mode performance without CA component. The black dashed lines are PPO performance. (a) Ablation results on Hazard World-Gres Grid--rid task. (b) Ablation results in SafetyGoal task.
6.4 Further Results
Pareto frontier. Multi-objective optimization typically involves finding the best trade-offs between multiple objectives. In this context, important to evaluate the performance different methods based their Pareto frontier [10), which represents the set of optimal trade-offs between the reward and cost objectives. We plot the Pareto frontier of policies trained with GC and policies trained with CP on two-dimensional graph, with the vertical axis representing the reward objective and the horizontal axis representing the cost objective as presented in Figure The solution that has the Pareto frontier closer the origin is generally considered more effective than those that have the Pareto frontier farther from the origin. We can observe from the figure that the policies trained with predicted cost by our TTCT have Pareto frontier closer to the origin. This proves the effectiveness of our method and further answers Questions (1) and (2).
0.5
0.5
1.0
Ciooutt CC
1.0
Reward
1.5
2.0
algo CPlour
algo
CHouu
CPlour
GC
1.5
GC
2.5
2.0
3.0
(a) Pag_Lag
(b) CPPO_PID
(c) FOCOPS
Figure 6: Results of Pareto frontiers. We compare the performance of 200 policies trained using cost prediction (CP) and 200 policies trained with ground-truth cost (GC). The * symbol represents the policy the Pareto frontier. And connect the Pareto-optimal policies with curve.

---

Zero-shot transfer capability. To explore whether our method has zero-shot transfer capability, Cuuzadd-Wold--oe World-Grid environment to apply directly new without fine-tuning. The results are shown in Figure We can observe that the policy trained with cost prediction (CP) from TTCT trained under the Hazard- World-Grid environment can still achieve low violation rate comparable to the GC-trained
we use the TTCT trained under the Hazard
environment called LavaWall (Figure (c)) S
policy. This answers Question (4).
algo
CPoor
Reward
0.44
ogoo
6.2
CPtoud
6C
60
40
#poch
spoch
(a) Zero-shot reward
(b) Zero-shot cost
Figure 7: Zero-shot adaptation capability of TTCT on Lava Wall task. The left figure shows the average reward and the right figure shows the average cost.
7 Conclusion and Future Work
In this paper, we study the problem of safe RL with trajectory-level natural language constraints and propose a method of trajectory-level textual constraints translator (TTCT) translate constraints into a cost function. By combining the eext-trajectory alignment (CA) component and the cost assignment (CA) component, our method can elegantly solve the problems of predicting constraint violations and cost sparsity. We demonstrated that our TTCT method achieves lower violation probability compared to the standard cost function. Thanks to powerful multimodal representation capabilities, our method also has zero-shot transfer capability to help the agent safely explore the constraint-shift environment. This work opens up new possibilities for training agents in safe RL tasks with total free-form and complex textual constraints.
Our work still has room for improvement. The violation rate of our method is absolute zero. In future work, we plan to investigate the application of TTCT in more complex environments and explore the integration of other techniques such meta-learning [6] further improve the performance and generalization capabilities of our method.
Acknowledgments
This work was supported by the grants from the Natural Science Foundation of China (62225202, 62202029), and Young Elite Scientists Sponsorship Program by CAST (No. 2023QNRC001), Thanks for the computing infrastructure provided by Beijing Advanced Innovation Center for Big Data and Brain Computing. This work was also sponsored by AuAl-Huawei MindSpore Open Fund. Jianxin Li is the corresponding author.
10

---

References
[1] Pieter Abbeel, Adam Coates, and Andrew Ng. Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research, 29(13):1608 1639, 2010.
[2] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[3] Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
[4] Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z. Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint orXiv:6006.0 04460, 2016.
[5] Maxime Chevalier-BBi BBiisver,,,,, Bolun Dai, Mark Towers, Rodrigo Perez- Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. Advances Neural Information Processing Systems, 36, 2024.
[6] Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov- based safe policy optimization for continuous control. arXiv preprint 190i:: 10031, 2019.
[7] Jacob Devlin, Ming- Wing-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810004805, 2018.
[8] Tom Fawcett. An introduction to roe analysis. Pattern recognition letters, 27(8):861-874, 2006.
[9] Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X Liu. Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 615(7953):620-627 627, 2023.
[10] Drew Fudenberg and Jean Tirole. Game theory. MIT press, 1991.
[11] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[12] Prasoon Goyal, Scott Nickum, and Raymond Mooney. Using natural language for reward shaping reinforcement learning. arXiv preprint arXiv: 190330200, 2019.
[13] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. review safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 10330, 2022.
[14] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551, 2017.
[15) Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):17355 1997.
[16] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions pattern analysis and machine intelligence. 44(9):5149-5169) 2021.
[17] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.
[18] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
11

---

[19] Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36, 2023.
[20] Russell Kaplan, Christopher Sauer, and Alexander Sosa. Beating atari with natural language guided reinforcement learning. arXiv preprini urXiv:1704.05:: 2017.
[21] Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-lever drone racing using deep reinforcement learning. Nature, 620(7976):982 987, 2023.
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[23] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(())79-86, 1951.
[24] Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforce- ment learning: A survey. Information Fusion, 85:1-22, 2022.
[25] Phuc H Le-Khac, Graham Healy, and Alan Smeaton. Contrastive representation learning: A framework and review. Ieee Access, 8: 8:193907-193334, 2020.
[26] Yang Liu, Yunan Luo, Yuanyi Zhong. Xi Chen, Qiang Liu, and Jian Peng. Sequence mod- cling of temporal credit assignment for episodic reinforcement learning. arXiv preprint arXiv:1905.13420, 2019.
[27] Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints in model-free reinforcement learning: survey, In The 30th international joint conference on artificial intelligence (ijcai), 2021.
(28] Xingzhou Lou, Junge Zhang, Ziyan Wang, Kaiqi Huang, and Yali Du. Safe reinforcement learning with free-form natural language constraints and pre-trained language models. arXiv preprint arXiv:2401.07553 2024.
[29] Bharat Prakash, Nicholas Waytowich, Ashwinkumar Ganesan, Tim Oates, and Tinoosh Mohsenin. Guiding safe reinforcement learning policies using structured language constraints. In SafeAI@ AAAI, pages 153-161, 2020.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[31] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18)99, 2019.
[32] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint UrXiv:1709.1008/ 2017.
[33] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning. arXiv preprint arXiv:1910.01708. 7(1):2. 2019.
[34] Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess. and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International conference on machine learning. pages 4344 4353. PMLR, 2018.
[35] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourleenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
12

---

[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[37] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[38] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-339, 2017.
[39] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages PMLR, 2020.
[40] Richard Stuart Sutton. Temporal credit assignment reinforcement learning. University of Massachusetts Amherst, 1984.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[42] Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472 2021.
[43] Steven D Whitehead and Long-Ji -i Lin, Reinforcement learning of non-markov decision processes. Artificial intelligence, (3(12)):271-33 1995.
[44] Long Yang, Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. Advances in Neural Information Processing Systems, 35:9111-9122 2022.
[45] Tao Yang, Youhua Xiao, Zhen Zhang, Yiming Liang, Guorui Li, Mingqi Zhang, Shijian Li, Tuck-Whye Wong, Yong Wang, Tiefeng Li, al. A soft artificial musele driven robot with reinforcement learning. Scientific reports, 8(1) 8(1):14518, 2018.
[46] Tsung- Yen Yang, Michael Y Hu, Yinlam Chow, Peter J Ramadge, and Karthik Narasimhan. Safe reinforcement learning with natural language constraints. Advances in Neural Information Processing Systems, 13883794-33888 2021.
[47] Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Accelerating safe reinforcement learning with constraint-iismattoe policies. arXiv preprint arXiv:2006. 1645, 2020.
[48] Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-busee constrained policy optimization. arXiv preprint arXiv:2010.03152, 2020.
[49] Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 33:15338-15349, 2020.
13

---

A Dataset and Training Details
A.1 Dataset
In our task setting, humans need to provide high-level textual instruction to the agent for the entire trajectory, and then TTCT can predict the cost based on the real-time state of the agent's exploration so that the agent can learn a safe policy with the predicted cost. Thus our dataset is comprised of two parts: the trajectory-level textual constraints and the environments.
Trajectory-level textual constraints: To generate textual constraints, we first explore the environ- ment using random policy, collecting a large amount of offline trajectory data. Then we design descriptor that automatically analyzes trajectories and gives natural language descriptions based on predefined templates. To validate whether our method can understand different difficulty levels of textual constraints, we design four types of trajectory-level textual constraints. The four types of textual constraints are:
1. Quantitative textual constraint describes quantitative relationship in which an entity in the environment cannot be touched beyond a specific number of times, which can be interpreted the entity's 's tolerance threshold, and when the threshold is exceeded, the entity may experience irrecoverable damage.
2. Sequential textual constraint describes a sequence- relationship, where the occurrence of two or more distinct actions independently may not pose risk, but when they occur in sequence, does. For instance, it's safe to drink or drive when they happen independently, but when they occur in sequence (i.e., drinking first and then driving), it becomes dangerous.
3. Relational textual constraint describes constraints on the relationships between an agent and entities its environment, such as maintaining a certain distance, always being in front of that entity, not staying too far from it.
4. Mathematical textual constraints often do not provide explicit instructions to the agent regarding what actions to avoid, but rather present logical descriptions that demand the model's ability to reason mathematically. This type of constraint thereby presents higher cognitive burden for our TTCT to comprehend.
Examples of four types of trajectory-level textual constraints are included in Table [ Then randomly split the (trajectory, textual constraint) pairs into 80% training and 20% test sets. And we use the training set to train our TTCT end-to-end.
Environments: We use two environments and environment Lava Wall to evaluate the zero-shot transfer capability:
Sooard--orld-GGee and SafetyGoal as main benchmarks
1. Hazard-World-Giin The environment is 12 x 12 grid, with the gray walls surrounding the perimeter. The agent can only explore within the grid. Inside the grid, some items provide rewards: blue keys, red balls, and yellow boxes. Collecting all of these items will be considered completing the task. Additionally, there are hazardous materials in the grid, where orange tiles are lava, cyan tiles are water, and green tiles are grass. During exploration, the agent can only see range of 7 7 pixels ahead, resulting in an observation space with size 7 x 7 3.
2. SafetyGoal. Robot navigation task, the environment has a navigation target, which is completed when the robot reaches the target. The environment also contains vases (cyan squares) and Hazards (Mediumslateblue circles). The vases can be moved by the robot and hazards are fixed.
3. Lava Wall. The LavaWall environment shares the same task goal Sirzard-WrlldGGiid but with different hazardous materials. Whenever the environment is initialized, a random lava wall with only one entrance is generated in the grid, and the agent must learn obtain the reward on the other side while avoiding stepping on lava.
For each episode, we place the agent at a randomized start location, fill the environment with objects, and randomly select textual constraint from the constraint pool. We manually design the violation checking function for every constraint to determine whether the environment violates the textual constraint, and when the textual constraint is violated, the environment will provide cost with
14

---

Table 2: Examples from the different trajectory-level textual constraint categories.
Constraint type
Examples
Do not cross lava more than five times.
Never reach grass
than five times.
Please touch water less than three times.
Quantitative
The vase very fragile, please don't 't touch it! If you touch the vase more than three times, the vase will break.
The lava very dangerous. Please don't [ touch
After stepping through the water, your shoes can't touch the lava. Once you've 've walked past the grass, make sure you don't 'I set foot on water.
Sequential
After walking past lava, don't 't go back to the water.
You only have 20 HP, grass and lava very dangerous, you will lose 3 and 2 HP respectively if you step on them, please don't die!
Mathematical
You only have 25 HP, stepping on grass and lava will lose 3 and HP respectively. but water will help you regain HP. please don't die!
You only have 10 HP. and stepping on dangerous water and lava will cause you to lose 3 and 2 HP respectively, your fate hangs by thread.
don't it snap!
Please keep distance of 0.2 from dangerous chemicals you will die!
Relational
Maintain a safe distance of 0.25 from the hazard.
It's important to keep distance of 0.3 from the hazard.
value of 1.0 the agent and terminate the environment (the manually designed function will not be used in the cost prediction (CP) mode, all the costs will be predicted by our TTCT in CP mode).
A.2 Baselines
(1) PPO 36]: This algorithm does not consider constraints, and simply considers maximizing the average reward, which we use to compare the ability of our methods to obtain rewards.
(2) PPO_ Too_Laranggian Laaanniin(PPPP anPPao_LPPPPL Lag) 33): This algorithm transforms constrained optimization problem into an unconstrained optimization problem via Lagrange multipliers
(3) CPPO_ [39] This algorithm PID control Lagrange multiplier, which solves the cycle
fluctuation problem of PPO-Lagrange.
(4) FOCOPS 49) This algorithm finds the optimal update policy by solving constrained optimization
problem in the nonparameterized policy space, then projects the update policy back into the parametric policy space.
A.3 Training Detail of TTCT
We provide training pseudocode of TTCT in Figure The text encoder is Bert (Chttpa //huggingface hawpeiinggacc..oogggglleeeeeeeee
[7]. And the hyperpa- rameters we use are shown in Table 3 We use the same hyperparameters across different tasks' datasets. We conduct the training on the machine with A100 GPU for about day.
A.4 Policy Training Detail
We provide pseudocode of training policy with predicted from TTCT in Algorithm I To enable the agent to comply with diverse types constraints, we launch asyne vectorized environments with different types of constrained environments during roll-out collection. The agent interacts with these environments and collects transitions under different constraint restrictions to update its policy. During policy updates, fine-tune the trajectory encoder 9 and text encoder gi. using LoRA [18] at every epoch's 's first iteration, while not updating encoder parameters other iterations, which saves training time.
15

---

Table 3: Hyperparameters used in TTCT
Hyperparaneters
Batch size
194
Epochs
32
Learning rate
10
Embedding dimension of trajectory dur
512
Embedding dimension of text dr
512
Trajectory length Transformer width Transformer number of heads Transformer number of layers Optimizer
200
512
12
Adam [22
Algorithm Safe RL algorithm with TTCT
Initialize value function network Φ, cost value function network dc. policy network 6, trajectory encoder 97, textual constraint encoder P6. frozen trajectory encoder gr for cost prediction, frozen textual constraint encoder gc for cost prediction, cost alignment network F
2: for each epoch do
3: for each episode do
4: Sample textual constraint y for this episode
5: Get text representation of y with 9
6: Reset environment get observation
7: Initialize sequence Tu = (Opods apad) and encoded sequence representation Ho = (11)
8:
for t 1, do
9:
Select a action a (o, H(-1,
10:
Execute action de in emulator and observe reward rt and next observation 0t-1
11:
Append (or, a4) into 7--1 to get T; and update sequence representation H; = 97 (7(()) (Tt)
12:
Predict cost 6 according Equation 13
13:
Store transition (or, az, Tt, G, Te-1, y) in buffer
end for
15:
end for
Sample batch of N transitions from buffer.
17:
Encode {Ty} N and ) with gr, 9C
Calculate specific loss function
19:
Update value function é, cost value function é.. policy network 0
Update trajectory encoder 9f and textual constraint encoder gō with LoRA
21: end for
policy training
Actor
TTCT training
action
gy²
pretesined Anujectury encodse
dataset
go
tuct
observation
cost
Environment
Frozen parameters
cost predictor
Fr
Trainable parameters
8C
Figure 8: A diagram illustrating the usage of gr, gG and g ge when training policy.
16

---

cete_enncder
oction encoder
DMN
trajectory_encen encoder
Cousal Transforeer
text_encoder
text encoder
Any Text Transformer
# episodic cost _eeedtccetcoor
DWN
single_step_com tingle_stp_cost_preiiie cor_____t_epecttr
Labels_MC
[n n] Labels for nuitimodal
stra[d_tra, d_e]
Learned proj of trajectory to exbed
M_t[d_t, d_e]
embed Learned prof of
# S[n, 4, d_s]
winthatch of state sequences
A(n, d_a]
winthatch of action sequences
T[n,
winibatch
constroins
extract feature representations different inputs S_f state_end (Sste_oncoder(S)
A_f (ejjector_eencoer((nne((((((( A.f), dim--1))
Tra_f trajectory
[n. L, d_tra]
T_ text_ vext_encodor(T)
[n, 2, d_t]
nomealization
Te 12, www.eeleeenp.dottT...... M_t), axis=-1)
[n, 1, d_e]
Tra_e t__orrallta(mppdoo(14 M M_tra), axise-1)
# [n, 4, d_e]
cost assignrent
EpCost www.aatte.ootppndiitttee..eeeeee
[n]
logits_Mi torch.netmul(Tra_e np.exp(t), T_e)
1] [n, L, 1]
atten_scomm signold(logits_.T
Tra_atten atten_score Tra_e
:111, :], T_e.ropeat(1,, 1-1, 1).detach()),
din- # [n, LJ
sun(cost, din=1) [n]
loss_cA (ese_losssEp sun_cost) ase aase_lsssssseees rro_loss(sun_cosee FpCost))/2
srthi--trajectory
labels_Mr NT torch. t1-ch. n) # only the Last time step is posttive loss_NT pooss_entropy_llsss(agttttt labels_wT)
nut muldiall contrastéve (MC)
logits MC mp_dot(Tra_e(:; -1, :] np.oxp(t), T_e) # [n, n]
loss_tre wedddvergence__osss(eeege MC, labels_MC, exis=0)
lass_t - Ki_divergence__a cetvergeec__lossllotttsssss Iab axis=1) loss_Mc (loss_tra loss_t)/2
total Loss
loss 1Css_MC 10ss_WT loss
Figure 9: Pseudocode for the code of training our TTCT.
17

---

B Additional Experiments
B.1 Violations Prediction Capability of Text-Trajectory Alignment Component
(a) Cosine similarity (after scale) (b) Ground-truth similarity
Figure 10: Heatmap of cosine similarity between trajectory and text embeddings.
To further study the ability of our text-trajectory alignment component to predict violations, conduct an experiment given a batch of trajectory-text pairs and we use the text-trajectory alignment component to encode the trajectory and textual constraint, and then calculate the cosine distance with Equation 33 between every two embeddings across two modal. We plot a sample of heatmap of calculated cosine similarity and ground-truth as presented in TO] Further, We plot the receiver operating characteristic (ROC) g curve to evaluate the performance of the text-trajectory alignment component as presented in Figure TT, AUC (Area Under the Curve) (8] values indicate the area under the ROC curve. The AUC value of our violations prediction result is 0.98. Then We set threshold equal to the best cutoff value of the ROC curve. We determine whether the trajectory violates a given textual constraint by:
yes, if sim(T,)) 2 3,
(14)
no, otherwise
The results are shown in Table These results indicate that text-trajectory alignment component can accurately predict whether given trajectory violates a textual constraint.
Table Violations prediction results text-trajectory alignment component.
Accuracy
Recall
Precision
Fl-score
0.98
0.9824
0.8079
0.8866
B.2 Case Study of Cost Assignment (CA) Component
We visualize the assigned cost for every state-action pair to demonstrate that the cost assignment component could capture the subtle relation between the state-action pair and textual constraint. Our intuition is that we should assign larger costs state-action pairs that lead to violations of trajectory-level textual constraints, and smaller or negative values to pairs that not contribute to constraint violations. Using the Hazard-World-Cin World-Grid environment as an example, we choose three different types of constraints to show our results in Figure I2 The first row shows the textual constraint, the second row shows the trajectory of the agent in the environment, and to make it easier to visualize, we simplify the observation 8; by representing it as a square, denoting the entity stepped on by the agent time step f. The square emphasized by the red line indicates the final entity that makes the agent violate textual constraint at time step T. The third row shows the predicted cost of the agent at every time step & and deeper colors indicate larger cost values.
The (a) constraint is mathematical textual constraint:
"You only have 20 HP. Luva and grass are dangerous, they will make you lose 3 and 2 HP, respectively. However, water can regenerate 1 HP. Please don't I die.". This constraint describes the two dangerous entities lava and grass, and the beneficial entity water. From the third-row heat map, observe that cost assignment
18

---

Receiver Operating Characteriztic
Pustdvee RUE
ROC
0.5
02
2-4
2.6
0.8
1.0
Felse Posttive Rate
Figure 11: ROC curve of text-trajectory alignment component. The x-axis represents the false positive rate, and the y-axis represents the true positive rate. The closer the AUC value is to 1, the better the performance of the model; conversely, the closer the AUC value to the worse the performance of the model.
Mathematical Text Constraint: "You only have 20 HP. Lava and grass dangerous, they will make lose 3 and 2 HP, respectively. However, water can regenerate HP. Please don't die."
(a)
0.01
10
11
12
13
14
15
16
17
18
19
20
22
0.00
grass
Quantitative Text Constraint: "Lava is dangerous. Don't touch them more than eight times!"
(b)
0.03
water
0.02
lava
0.01
floor
Sequential Text Constraint: "After you touch lava, don't It step gross!"
(c)
0.02
0.00
Case study of cost assignment component on three types of textual constraints. The first row of every case shows the textual constraint, the second row shows the trajectory the agent in the environment and each square represents the object stepped by the agent at that time step, the third row shows the assigned cost of the agent at each time step, and the fourth row shows the time steps. The red line indicates the final observation where the agent violates the textual constraint.
Figure 12:
component assigns high cost to the action that steps lava grass, with the cost increasing the agent approaches the constraint conttraint-voolaiing situation. Not only that, the CA component also recognizes the different levels of danger posed by lava and grass. Since stepping lava will deduct 3 HP while stepping on grass will deduct 2 HP, the CA component assigns a larger cost value at time step 13 14 compared to the value at time step 3 5.
The (b) constraint quantitative textual constraint: "Lava is dangerous. Don't touch them more than eight times!". When stepping the floor, the CA component considers these actions to be safe and assigns cost of nearly However, when the agent steps onto lava, it assigns a higher cost, especially when the agent steps on lava for the eighth time. At this point, concludes that the situation has become extremely dangerous, and one more step on lava will violate the constraint, thus giving the highest cost compared to the before time steps.
CA component
The (c) constraint sequential textual constraint: "After you touch lava, don't II slep on grass!". Our CA component captures two relevant entities: lava and grass, and understands the sequential relationship between entities. When the agent first steps onto the grass, the text-trajectory alignment component determines that this action does not violate the textual constraint. However, after the
19

---

agent steps onto lava and then steps onto the grass for the second time, the text-trajectory alignment component detects the cosine similarity sim(t, y) greater than the threshold B. thereby violating the textual constraint. And CA component captures that the key trigger condition for violating constraint stepping onto the lava, therefore assigning a relatively larger cost to such actions at time step 7 9. The cost assignment component also assigns relatively small costs to some safe actions that are stepping on the floor, such as steps 13 15. This because detects that the agent, although hasn't t stepped
grass yet, is trending towards approaching grass, which is hazardous trend, thus providing a series of gradually increasing and small costs. This demonstrates that our component not only monitors key actions that lead to constraint violations but also monitors the hazardous trend and nudging the agent to choose relatively safer paths.
B.3 Results for Different Types of Constraints
To evaluate the agent's understanding of different types of trajectory-level textual constraints, we conducted additional experiment using the CPPO_ PID algorithm. During training, separately tracked the average episodic reward and the average episodic cost for three types textual constraints presented in Figure [3] From the learning curves, our CP mode can achieve the lowest violation rate compared to CP without the CA component mode and ground-truth (GC) mode.
observe that for every type of constraint,
algo
CPtour)
(Plour) CP
0.8
Reward
1.5
0.6
0.4
0.5
0.2
20
40
60
80
100
20
40
60
80
100
epoch
epach
(a) Quantitative constraints
algo
0.8
CP w/o CA GC
Reware
0.6
algo
CPlour]
0.5
w/o
80
20
100
epoch
epach
(b) Sequential constraints
algo
0.8
(Poou))
0.6
GC
0.4
algo
CPlour]
CP wib CA
0.2
80
100
epach
(c) Mathematical constraints
Figure 13: Learning curves for different types of textual constraints. The left figure shows the average reward and the right figure shows the average cost. The result shows for each type of textual constraint, the policies trained by predicted cost from the TTCT can achieve lower constraint violations.
20

---

B.4 Inference time
We perform the trajectory length sensitivity analysis on Hazard- World-Gridd Since our framework is mainly used for reinforcement learning policy training where data is typically provided as input in batches, we counted the inference time for different trajectory lengths with 64 as the batch size, using the hardware device V V100-32G, Figure [4 shows that the average inference time per trajectory is 10ms for trajectories of length 100.
1.
1.2
1.0
Inference
0.7
Trajectory Length
Figure 14: Inference time of different trajectory lengths for Wrrld-Grrd on the V100-32G hardware device. Batch size 64.
B.5
Different Text Encoder
3.0
Text Encoder basmurcased
Transfomerr
3.5
TextEncoder bart Bwwe--eeeeddd
Transfomer
3p12
80
40
epoch
spoch
(a) Average episodic reward
(b) Average episodic cost
Figure 15: Empirical analyses on Hazard- Soardd-Woll-GGrie with varying text encoders. We choose three
different models, transformer-25M [1). 1372-137M [I], and www.baaacssdddd-a--...aaa 10M (7].
B.6 Additional Results
We present the learning curve of our main experiment in Figure 4 We also present the main results and ablation study in T 6
Table 5: Evaluation results of our proposed method TTCT. 1 means the higher the reward, the better the performance. means the lower the cost, the better the performance. Each value is reported as mean standard deviation and we shade the safest agents with the lowest averaged cost violation values for every algorithm.
PPO Lag
CPPD PID
FOCOPS
Task
Stats
PPO
CP(our)
GC
CP(our)
GC
CP(our)
GC
Avg.E R
2.70
2.71.0.11
2.01.0.14
2.00g0.26
1.52
1.53.0.28
2.55
Grid
Avg.Co C
0.2820.08
0.61-0.07
0.28
0.40go.07
0.48
0.58
0.8820.05
Rvg.R
0.23.o.12
.o.22...8
0.36
0.35
0.02,o.36
-0.63.
0.23...65
Goal
Avg.C. C
0.1920.02
0.2620.03
0.11g0.02
0.35go.20
0.1040.03
0.4140.07
0.32go.19
21

---

Table 6: Ablation study of removing the cost assignment (CA) component, means the higher the reward, the better the performance. + means the lower the cost, the better the performance. Each value is reported mean standard deviation and we shade the safest agents with the lowest averaged cost violation values for every algorithm.
PPO Lag
CPPO PID
FOCOPS
Task
Stats
PPO
CP(Full)
CP w/o CA
CP(Full)
CP w/o CA
CP(Full)
CP w/o CA
Grid
Avg.R1
2.70
2.57+0.11
2.01
2.00.0.16
1.52g0.12
0.7940.06 0..0
2.55+0.25
Avg.C1
0.28
0.5620.08
0.28+0.03
0.52-0.14
0.48.0.20
0.4320.01
0.8840.05
Avg.R
0.23
-0.622o.10
0.36
-0.08.0.97
0.0220.30
-0,6320.21
0.23+0.55
Goal
Avg.C
0.1920.02
0.35
0.11
0.2920.44
0.10
0.2320.09
0.3240.19
C
Broader Impacts and Limitation
Our method can help train agents in reinforcement learning tasks with total free-form natural language constraints, which can be useful in various real-world applications such as autonomous driving, robotics, and game playing. There are still limitations to work. Our method may not be able to completely eliminate constraint violations. Our method has the contextual bottleneck as the length of the trajectory increases. We performed trajectory length sensitivity analysis Hazard- World-Grid As shown Figure [6] initially increasing the trajectory length improves performance because longer trajectories may provide more dependencies. However, beyond certain point, further increases in trajectory length result in a slight drop in AUC. This decline because the trajectory encoder has difficulty capturing global information. We consider this bottleneck to be related to the transformer's encoding capability.
0.925
0.950
0.925
0.875
0.550
0.825
50
100
150
200
250
300
350
400
Trajectory Length
Figure 16: Evaluation results of different trajectory lengths for Hazard-World-Grid Sets of trajectories with varying lengths shared the same set of textual constraints.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We show our approach predict cost with trajectory-level natural language. The predicted cost by our approach can help train a safer policy than the ground-truth cost.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We show the used parameters and the pseudocode of TTCT in Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: Code to reproduce our results is available in the anonymous repository https:
//github. cow wor.//eermmbbrrrrr
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.cc/ppl wwcppbblc/guuiicuccccccc
http://oooSuuuisss.ooeee Stt.SSbbbiiiiooooiiiii.oo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 6 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See Section e
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See the Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: All the links, data, code are provided in the anonymous form.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix C
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: They open-sourced.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "co8Eeu7mAa", "reviewer": "Reviewer_cY35", "summary": "- This work proposes an approach to training RL agents with constraints. The proposed approach learns language embeddings of constraints and embeddings of trajectories. During training, a similarity score is used to align the space of constraint embeddings and the space of trajectory embeddings.\n- Moreover, the language embedding of the constraint is also used to do fine-grained credit assignment across the actions in a trajectory. The authors find that this improves performance.", "questions": "- Line 174: What is a \"negative pair\"? Is it a constraint and a trajectory that violates that constraint?\n- Line 175: Given that the mapping from trajectories to constraints is many-to-many, doesn't $q$ need to be normalized so that the probabilities sum to 1? That is, there are many positive pairs that contain the same constraint, so they should all receive an equal share of positive probability.\n- Fig 5: I think I'm missing something basic: how are is the Pareto front determined? I could assume that per cost, the best reward possible can be determined by some procedure? Is this what was done?\n- Section 6: Where do the textual constraints come from? Are they human annotated? Appendix 1 seems to suggest not.\n  - Is it guaranteed that the discriminator can tell when a textual constraint is violated?\n- Line 289: What is the difference between ground-truth mode and cost prediction mode? Is it the cost assignment?", "limitations": "Limitations are mentioned in the appendix.", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- The approach is novel in that the technique of credit assignment is applied to the constraint rather than the reward. Or at least this is the claim, and I was not able to find otherwise. \n- The approach is shown to be effective. With an ablation study, the authors show that their credit assignment approach gives better performance\n- I have questions about the setup and novelty (see weaknesses), but overall this work seems to represent a positive contribution: the authors show that it is possible to encode trajectory-level constraints with text and to train an RL agent to respect those constraints.", "weaknesses": "- I have a hard time understanding where this work stands in relation to prior works. The main contribution seems centered around the approach of encoding trajectory level constraints using natural language. But both trajectory level constraints and natural language constraints have been explored independently. And the combination of these two aspects seems straightforward: encode the trajectory level constraint using natural language. I may be overlooking something that makes this less straightforward. I don't quite follow the explanation for novelty on lines 101-104. The Related Works section mentions prior work which learns language representations of constraints, but says that these works do not consider trajectory-level constraints. But because the constraints are given in natural language, it doesn't seem that anything in principle prevents these prior approaches from doing so? What is the technical innovation of the proposed work that allows for trajectory level constraints?\n- A few things seem missing from the setup (see questions)\n- I have concerns about the soundness of the described approach. Where does the ground truth come from when training the text-trajectory alignment component? As far as I can tell, there is notion of a textual constraint but not a notion of a ground truth verification of a constraint. From what I understand of appendix 1, the textual constraint is determined to be violated by a discriminator model (see questions). In any case, the authors should explain how constraints are determined to be violated. This matters because it means that the discriminator model and the agent could both make the same systematic linguistic errors.\n- I have concerns about the soundness of the approach in general. Using natural language to describe constraints seems at odds with the enterprise of making RL more \"safe.\" Natural language can be ambiguous, and machine learned representations of language can be imperfect, and prone to making semantic errors. In contrast to other approaches which provide formal guarantees on behavior [1], this seems much less safe. However, I am conscious that others in the field think differently, given the prior work. \n\n## minor\n- Figure 3: Labels are missing in the legend\n- Line 297: \"Table\" --> \"Figure\"\n\n## references\n[1] Fulton, Nathan, and André Platzer. \"Safe reinforcement learning via formal methods: Toward safe control through proof and learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully reviewing our submission. We address each point of your concerns separately below.\n\n-----\n\n**W1: Relation to prior works** \n\n**Answer:** We appreciate the reviewer’s constructive question. See the overall response about the **relationship to previous work**. We will elaborate on the above discussion in the final version.\n\n----\n\n**Q1: About negative pair**\n\n**Answer:** In our formal setting, \"negative pair\" means an unmatched pair of constraint and trajectory, that is no violation of this textual constraint in the trajectory. \n\n----\n\n**Q2: Probability normalization**\n\n**Answer**: Sorry for causing the misunderstanding. We do normalization in our experiments:\n\n```label_probs = F.softmax(label, dim = 1)```\n\nWe will rewrite the loss function part for clarity in the final version\n\n----\n\n**Q3: About Pareto front**\n\n**Answer:** During RL training, we collected about 200 policies and evaluated them on 50 new episodes for average cost and reward scores. We follow the definition in [1]. To determine the Pareto front, we compared each policy with others. A policy is not on the Pareto front if another policy has equal or better rewards and equal or lower costs. If no such policy exists, the current policy is part of the Pareto front.\n\nSo for a given cost, the policy with the highest reward may not be on the Pareto front if another policy exists that outperforms it in both reward and cost.\n\n[1] Wikipedia contributors. (2024, June 12). Pareto front. In *Wikipedia, The Free Encyclopedia*. \n\n----\n\n**Q4.1: Where do the textual constraints come from**\n\n**Answer:** To ensure the generation of textual constraints is both professional and effective, we implemented the following steps:\n\n1. We developed a descriptor function to systematically identify all hazards encountered at every timestep when random policy explores the environment. We identified the hazards by checking the system state and storing them in a structured text format. For example:\n\n   ```\n   hazards = {\n       \"timestep\":{\n           5: \"step into chemical area\", \n           # at 5th timestep, the agent step into chemical area\n           12: \"step on the grass\",\n           20: \"step into the water\",\n           30: \"step into the lava\"\n       }\n   }\n   ```\n\n2. We then constructed various logical combinations of different hazards. And combination format depends on the type of constraint we want to generate. Here we take the mathematical constraint as an example, since it was checked that the agent touched four hazardous items, we flexibly assigned a lost HP value to each hazard and used the total amount of HP lost in the trajectory, as the total HP:\n   ```\n   combination = {\n       \"constraint type\": \"Mathematical\",\n       \"HP lose\":{\n           \"chemical area\": 3,\n           # step into chemical area will lose 3 HP\n           \"lava\": 2,\n           \"grass\": 1,\n           \"water\": 0\n       }\n       \"total HP\": 6\n   }\n   ```\n\n3. We engaged several researchers to define a large number of constraint templates in diverse language styles, which were used to rephrase the logical combinations into unstructured natural language.\n\nThese steps enabled us to generate a series of textual constraints, each corresponding to a structured combination. In practical applications, generating textual constraints could be simplified as follows:\n\n- Instance 1: Researchers directly describe constraints based on video/image demonstrations of a trajectory.\n- Instance 2: Gather videos with dangerous scenes and textual commentary, convert these into trajectories and natural language constraints, and use U3T to train agents to adhere to real-life constraints.\n\n----\n\n**W3&Q4.2: Soudness of discriminators** \n\n**Answer:** Sorry for the misunderstanding due to the omission of the detailed description of the discriminator. We manually designed (not trained) complex discriminators for various types of constraints which are absolutely accurate. Each natural language constraint corresponds to a structured logical combination (mentioned in the Q4.1). Human-designed discriminators assess whether a trajectory violates the logical combination, ensuring accuracy and the soundness of our approach. We will provide further details in the final version. It's important to note that this complex discriminator is not needed for the practical application of U3T; it was designed solely to evaluate U3T's performance in our study.\n\n----\n\n**Q5: Difference between ground-truth mode and cost prediction mode**\n\n**Answer:** As we mentioned in Section 6.1 of the paper, in the ground-truth mode, policies are trained with human-designed constraint discriminators, which will give accurate cost at the timestep when the constraint is violated. In the cost prediction mode, there are two types of cost, the constraint violation cost and the assigned cost, they are all predicted by U3T. The predicted cost function is given by Equation 13 in the paper.\n\n----\n\n**W4: The soundness in general**\n\n**Answer:** The reviewer raises an interesting point. \n\nOur work emphasizes the flexibility and generalization of language. We propose a modeling paradigm for safe reinforcement learning (RL) that integrates future advancements in natural language and series modeling, enhancing RL safety over time. Rapid adaptation and flexibility can be crucial in dynamic environments, where natural language constraints may be more effective than predefined formal ones.\n\nWe align agents' behavior with human intentions similar to OpenAI's RLHF [1]. Just as defining alignment safety in large language models is challenging, our method also struggles with providing a clear definition of textual constraint safety boundaries. So we focus on providing a prototype for safe RL with textual constraints and aim to develop a more rigorous definition of textual constraint safety in future work.\n\n[1] OpenAI. ChatGPT: Optimizing Language Models for Dialogue, 2022.\n\n----\n\n*We hope these responses address your concerns.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rEEUSRqhta", "reviewer": "Reviewer_z2Bi", "summary": "* The paper proposes U3T, a new system for more robust safe RL under general text constraints.\n* The key innovation of the paper is to generalize text constraints such that the constraints don’t refer to a specific entity/state and addressing cost sparsity where constraints are only violated at the final time step.\n* The proposed system consists of a trajectory encoder and text encoder that are trained jointly using a contrastive loss defined by the KL-divergence.\n* The authors solve cost sparsity by identifying actions/states that contribute to the violation of a constraint in an attention mechanism.", "questions": "* I noticed that generally, while the cost of the method would decrease, the reward wouldn’t change significantly (Figure 2). I actually found this a bit unintuitive. I figured that lower cost would imply a longer trajectory and therefore, longer windows to collect reward over, so average reward could increase. I found this discussion missing in the text. Could this be elaborated?\n* How long were the trajectories used in the paper? (What was the average size of T?) Could there be a context bottleneck?", "limitations": "* See question about context bottleneck. Could this be a limitation to discuss?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper solves an interesting and intuitive problem. There is a clear application of generalizing text constraints from single entity to general entities. \n* The formulation is intuitive and well-explained. The components introduced build on a simple and effective design\n* Ablation studies seem to demonstrate the benefit of the cost assignment portion of their system design, a portion that could be extremely expensive cost-wise.\n* The general cost (Figure 2/Figure 3) of their method seems to reduce in comparison to their baselines that use ground truth.", "weaknesses": "* I found some details missing in the text. I discuss some portions that could use clarification here.\n    * Which text encoder was used in this experiment? I couldn’t find this in the paper and from your appendix section A.4, it seems untrained. You refer to finetuning with LORA though so I was confused. A small nit, could be interesting to connect the text encoder performance to results. I realize space is tight but if these could be incorporated into the main paper, that would clarify and make the experiments easier to interpret.\n    * I’m unsure how to interpret the average reward results in comparison to the cost results. See questions. My intuition was that rewards should be larger given the pareto frontier results. \n* I’m having trouble differentiating this paper from the Lou et. al. paper. I took a quick look at Lou et. al. --  I can understand that entities needed to be modeled in the paper but from what I can tell, that paper could also be applied in the settings you considered given the use of the pretrained LLM. From Figure 1, I couldn’t see the specific entity problem you were mentioning in the paper. Why wasn’t this added as a baseline? Could you provide more discussion on why these two are different? I think the approach in this paper has considerable novelty so I won’t say that this is entirely negative but it does diminish the contribution slightly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. And we are very pleased that you appreciate the effectiveness and direction of our research.  We would like to address your concerns below.\n\n---\n\n**W1.1: About text encoder**\n\n**Answer:** We used pre-trained bert-base-uncased [1] as a text encoder. During U3T training, we fully fine-tuned the text encoder. The text encoder was fine-tuned again using LoRa during RL policy training. The original U3T, used for predicting costs, was not retrained during RL policy training. A formal description is provided in Algorithm 1.\n\nWe have added related experiments. We chose three different models: untrained transformer-25M, pretrained gpt2-137M, and pretrained bert-base-uncased-110M.\n\nFor natural language understanding tasks, bert-base-uncased has the strongest capability, GPT-2 comes second, and Transformer is the weakest [2]. As shown in Figure 3 of the rebuttal PDF, using bert-base-uncased as a text encoder yields the best results, indicating that models with stronger semantic understanding capabilities improve our framework's performance.\n\n[1] Devlin, J., Chang, M. W., Lee, K. et. al. Bert: Pre-training of deep bidirectional transformers for language understanding. \n\n[2] Ethayarajh, K. (2019). How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. \n\n----\n\n**W1.2&Q1: The reward didn't change significantly.**\n\n**Answer:** Safe RL problems can be classified into two categories (more formally defined in [1]).\n\nThe first category involves cases where maximizing reward and minimizing cost are aligned. For example, in an environment with a ball in the center of a 3D square, the robot must touch the ball without stepping outside the square (Dalal et al. [2]). Here, touching the ball (maximizing reward) aligns with staying inside the square (minimizing cost). This type of problem is simpler, as it reduces to a standard RL problem focused on maximizing reward. The second category involves scenarios where maximizing reward and minimizing cost are not aligned. In these cases, the policy that maximizes rewards is not the one that minimizes costs, leading to a trade-off between reward and cost. This type of problem is more challenging, as avoiding hazards does not provide an intuitive reward gain.\n\nOur experimental benchmark falls into the second category. In our environment, hazards often lie between the agent and the target. To reach the goal without violating constraints, agents must take detours and adopt behaviors that ensure safety but do not directly yield rewards. This explains why a low violation rate and extended reward collection time do not significantly boost rewards—the additional time is spent avoiding hazards.\n\nIn our setup, minimizing costs does not directly imply maximizing rewards. Achieving a lower violation rate with the same rewards is already an improvement. This challenge is also related to the inherent limitations in current safe RL algorithms, which often lack mechanisms for enhancing exploration while ensuring safety. We appreciate the reviewer's unique perspective and will discuss this in detail in the final version.\n\n[1] Liu, Z., Guo, Z., Yao, Y., Cen, Z., Yu, W., Zhang, T., & Zhao, D. Constrained decision transformer for offline safe reinforcement learning.\n\n[2] Dalal, G., Dvijotham, K., Vecerik, M., Hester, T. et. al. Safe exploration in continuous action spaces. \n\n----\n\n**W2: About Lou et. al. paper**\n\n**Answer:** Thank you to the reviewer for recognizing the novelty of our paper. We believe it is the first to enable the modeling of fully generalized textual constraints. Our innovation lies in our unified framework that handles constraints with complex logic involving multiple states, entities, and timesteps. Lou et al.'s work determine constraint violations by calculating the similarity of **single** observations to constraints, limiting them to single-state or single-timestep constraints. This approach cannot model complex real-world constraints, such as \"don't drive after drinking,\" which involves sequence and dependency.\n\nOur work, in contrast, addresses global trajectories, allowing us to model a broader range of constraints. We focus on universal textual constraints with complex semantics, which Lou et al.'s approach cannot handle due to their lack of components to model trajectory dependencies and align trajectory semantics with textual semantics. Hence, we did not use their work as a baseline for our research.\n\nWe conducted comparative experiments in the hazard-world-grid environment to verify our method's accuracy under their task settings. We applied constraints to single states and accumulated costs for violations instead of terminating the episode. Since their training code is not open-sourced, we obtained their experimental results from their paper's figures. The experimental results are as follows:\n\n| Method      | Avg. Reward     | Avg. Cost      |\n| ----------- | --------------- | -------------- |\n| Lou et. al. | $4.9_{\\pm0.2}$  | $4.5_{\\pm0.5}$ |\n| our         | $4.92_{\\pm0.3}$ | $3.3_{\\pm0.4}$ |\n\n----\n\n**Q2: About Trajectory length**\n\n**Answer:** The maximum trajectory length in our experiment is 200, with an average episode length of approximately 100, depending on the violation rate. We conducted a trajectory length sensitivity analysis as shown in Figure 2 on Hazard-World-Grid, using AUC to measure prediction accuracy. Initially, increasing the trajectory length improves performance as longer trajectories provide more dependencies. However, beyond a certain point, further increases result in a slight drop in AUC due to the transformer's difficulty in capturing global information, indicating a contextual bottleneck related to the transformer's encoding capability. We will add the description of trajectory length and discuss the context bottleneck in the final version.\n\n----\n\n*We hope these responses address your concerns. We remain open to further discussion.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qZUrXb6jXM", "reviewer": "Reviewer_T5E9", "summary": "This paper introduces a type of universal natural language constraint to model diverse real-world safety requirements. Also, to avoid using the specific human-designed cost function, this paper introduces a Unified Trajectory-Level Textual Constraints Translator (U3T) for aligning text with trajectories and assigning costs via attention mechanisms. By conducting experiments on two environments, this paper reaches the conclusion that U3T can accurately predict whether a given trajectory violates the natural language constraint, and the policy trained with predicted cost can behave more safely.", "questions": "1. Is it feasible for the authors to experimentally analyze the inference speed of the proposed framework?\n2. Could the authors specify the model of the text encoder used in the experiments and provide an in-depth exploration of its selection rationale?\n3. Leveraging large language models for text understanding shows promise. Have the authors considered strategies for integrating these models to augment the comprehension and deconstruction of intricate textual constraints?", "limitations": "The authors have discussed the limitations in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Broad applicability: U3T introduces trajectory-level textual constraints, capable of modeling diverse constraint requirements in real-world scenarios, making it more widely applicable across different types of constraints and complex environments compared to traditional methods.\n2. Novelty: Addressing the natural language constrained reinforcement learning challenge through a novel approach involving multimodal alignment and novelly integrating a credit assignment component within the framework.\n3. Automated constraint handling: Through its text-trajectory alignment and attention-based cost assignment components, U3T automates constraint handling and cost allocation, reducing the need for manual design of cost functions and enhancing system flexibility and generality.\n4. Empirical results demonstrate that policies trained with U3T achieve lower violation rates compared to standard cost functions.", "weaknesses": "1. Complexity and computational cost: Implementing U3T involves complex multimodal learning and attention mechanisms, which may require significant computational resources and time costs, especially when dealing with large-scale datasets or real-time applications.\n2. The authors do not specify exactly what model of text encoder was used in the experiments.\n3. Data Dependency: Effective implementation of U3T heavily relies on high-quality and well-annotated textual and trajectory data, which may be challenging to acquire and curate in some practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments. We are grateful to the the reviewer's conductive comments. We will answer the questions below.  We believe we addressed all the concerns and are glad to follow up in the discussion phase.\n\n----\n\n**Q1: Inference speed**\n\n**Answer:** We perform the trajectory length sensitivity analysis on Hazard-World-Grid. Since our framework is mainly used for reinforcement learning policy training where data is typically provided as input in batches, we counted the inference time for different trajectory lengths with 64 as the batch size, using the hardware device V100-32G. Figure 3 in the rebuttal PDF shows that the average inference time per trajectory is 10ms for trajectories of length 100. This inference time is generally acceptable.\n\n---\n\n**Q2: The text encoder we used**\n\n**Answer:** Due to its bidirectional encoding capabilities, Bert [1] can better understand the context of words, and in particular, excels in capturing dependencies within sentences. While autoregressive language models such as GPT2 are more suitable for text-generation tasks [2]. So we choose bert-base-uncased as the text encoder. Sorry for omitting the description of text encoder architecture in our paper.  We will add it in the final version.\n\n[1] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. *arxiv preprint arxiv:1810.04805*.\n\n[2] Ethayarajh, K. (2019). How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. *arXiv preprint arXiv:1909.00512*.\n\n----\n\n**Q3: Leverage Large language model**\n\n**Answer:** Thanks to the reviewer for the constructive view.  Large language models exhibit powerful semantic understanding and task decomposition capabilities, so in the future, LLM could be utilized to decompose more complex constraints into several simple constraints, which can then be handed off to lower-level constraint-obey agents for execution.\n\n----\n\n*We hope these responses address your concerns. We remain open to further discussion.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their time spent reviewing our paper and are grateful for the endorsement on **novelty** ("the approach is novel in that the technique of credit assignment is applied to the constraint" - cY35, "the paper solves an interesting and intuitive problem" - z2Bi, "through a novel approach" - T5E9) and **effectiveness** ("the approach is shown to be effective" - cY35, "effective design" - z2Bi, "achieve lower violation rates" - T5E9).

---

Here is our unified response to the common questions regarding the **relationship to previous work:**  in our paper, a trajectory-level constraint involves complex logic across multiple states, entities, and timesteps. Prior works focused on simpler constraints related to single states or timesteps, which limits their ability to model complex safety requirements. Despite constraints being provided in natural language, previous methods fail to offer a unified framework for trajectory-level textual constraints due to the lack of a unified representation of trajectory dependencies and the inability to align trajectory semantics with natural language.The novelty of our work lies in the unified understanding and application of the universal constraints. We align the factual logic in the global trajectory with the semantic logic in the text without requiring manual encoding or separate models for each type of constraint. We achieve this by utilizing **the supervision inherently present in the natural language**.

We provided individual answers to every reviewer. We will adapt the paper based on all reviewers' insightful comments and questions.  And we are happy to follow up in the discussion phase.

----

**Rebuttal pdf**

We provided additional experimental results to answer some of the reviewers' questions:

- Figure 1: question about text encoder from reviewer z2Bi
- Figure 2: question about trajectory length from reviewer z2Bi
- Figure 3: question about inference time from Reviewer T5E9

Meta-review: Learning joint embeddings between text and trajectories is useful and topical.

Reviewers note that the main weakness is how the work relates to other research. Authors put this forward as a "unified framework" for reasoning about language and trajectories. With a focus on the fact that their method doesn't require manual annotations and alignment between the concepts in the trajectory and the linguistic concepts. But there is a tremendous literature that is being ignored which does exactly this.

I would really encourage the authors to engage with this literature because it points the way toward more interesting experiments for followup work. Reviewers point to work in safe RL, another area which does the same has to do with robots that follow linguist commands including those that incorporate notions of safety with LTL. This work often doesn't require annotations or alignment just like the work here.

In addition, there are so many other linguistic phenomena and logics to consider when aligning language to trajectories. Claiming that this is a unified framework for any constraints and notions of safety is far too much.

That being said, reviewers found that the topical nature of the work overrides these concerns and I agree. But it's to the author's benefit to resolve these issues with the manuscript. It will only allow them to engage with a much wider community.
Decision: Accept (poster)