From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning

Authors: pusen  dong
tianchen  zhu
yue  qiu
haoyi  zhou
jianxin  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 07:06:25

Abstract: Safe reinforcement learning (RL) requires the agent to finish a given task while obeying specific constraints. Giving constraints in natural language form has great potential for practical scenarios due to its flexible transfer capability and accessibility. Previous safe RL methods with natural language constraints typically need to design cost functions manually for each constraint, which requires domain expertise and lacks flexibility. In this paper, we harness the dual role of text in this task, using it not only to provide constraint but also as a training signal. We introduce the Trajectory-level Textual Constraints Translator (TTCT) to replace the manually designed cost function. Our empirical results demonstrate that TTCT effectively comprehends textual constraint and trajectory, and the policies trained by TTCT can achieve a lower violation rate than the standard cost function. Extra studies are conducted to demonstrate that the TTCT has zero-shot transfer capability to adapt to constraint-shift environments.

Main paper: 
 introduction: 
 In recent years, reinforcement learning (RL) has achieved remarkable success in multiple domains, such as go game [37, 38] and robotic control [21, 45] . However, deploying RL in real-world scenarios still remains challenging. Many real-world decision-making applications, such as autonomous driving [13, 9] require agents to obey certain constraints while achieving the desired goals. To learn a safe constrained policy, some safe RL works [2, 33, 44, 49, 39, 48, 6] have proposed methods to maximize the reward while minimizing the constraint violations after training or during training.However, several limitations prevent the existing safe RL methods' widespread use in real-world applications. Firstly, these methods often require mathematical or logical definitions of cost functions, which require domain expertise (Limitation 1). Secondly, their cost function definitions are frequently specific to a particular context and cannot be easily generalized to new tasks with similar constraints (Limitation 2). Lastly, most current safe RL methods focus on constraints that are logically simple, typically involving only one single entity or one single state [28] , which can't represent the real-world safety requirements and lack universality (Limitation 3).Using natural language to provide constraints [46, 29, 20] is a promising approach to overcome Limitation 1 and 2 because natural language allows for flexible, high-level expression of constraints that can easily adapt to different scenarios. Regarding Limitation 3, previous approaches primarily employ what we call the single state/entity textual constraint. The single-state/entity textual constraint focuses solely on constraints related to one specific state or entity, limiting the ability to model complex safety requirements in real-world scenarios. Many safety requirements involve interactions and dependencies among multiple states or entities over time. By only addressing a single state or Don't stay in the gas area for more than 5 minutes, your gas mask will fail. (multi states)entity, these constraints fail to capture the dynamic relationships and temporal aspects that are crucial for ensuring safety in complex environments. So we suggest using a more generalizable constraint type trajectory-level textual constraint. The trajectory-level textual constraint is a more universal constraint with complex logic semantics involving multiple states/entities. It is a nonlinear combination of multiple entities or multiple environment states and can model any constraint requirements in real-world scenarios. The trajectory-level constraints in natural language form are the highly abstract expression of the agent's behavior guidelines, serving as a more natural and straightforward way to introduce constraints. Notably, the set of trajectory-level constraints encompasses single-state constraints, as any single-state constraint can be reformulated as an equivalent trajectory-level constraint.Examples of trajectory-level textual constraint and single state/entity textual constraint are presented in Table 1 .Employing trajectory-level textual constraints across the entire trajectory poses two significant challenges. Firstly, determining whether an RL agent violates textual constraint over a trajectory is non-trivial, it needs to have the perception of the historical states and actions observed along the trajectory (Challenge 1). Secondly, the trajectory-level safety problem is susceptible to sparse cost [34] , where cost is only imposed when the agent violates textual constraints at the final time step, making it challenging for the agent to learn which actions contribute to a gradual escalation of risk (Challenge 2). For instance, the agent needs to learn constraints like, "Don't touch lava after you touch grass." Without intermediate feedback, however, it struggles to understand how early actions, such as stepping on grass, contribute to eventual violations.To address Challenge 1, we propose a general approach to align the trajectory's factual logic with the text's semantic logic, eliminating the need for manual encoding or separate models for each type of constraint. Our method employs a sequence model [41, 15] for modeling agent historical interactions with the environment, and a pre-trained language model (LM) [7] to comprehend natural language constraints. We then maximize the embedding similarities between matching pairs (trajectory, text) and minimize the embedding similarities between non-matching pairs using a contrastive learning approach [25] similar to CLIP [30] . By calculating the similarity between the textual embeddings of the constraint and the trajectory, we can predict whether a constraint is violated in this trajectory. Our method uniquely leverages text as both a source of constraints and a unified supervisory signal for trajectory encoding. In this dual role, text not only provides constraints but also guides the training process, enabling the model to naturally handle diverse semantic constraints without requiring specific model adjustments for each type. This design allows for a more flexible and generalizable system, significantly simplifying the handling of complex, multi-dimensional constraints.In addition, to address the issue of cost sparsity (Challenge 2), we introduce a method for temporal credit assignment [40] . The proposed approach involves decomposing the one episodic cost of the textual constraint into multiple parts and allocating them to each state-action pair within the trajectory. This method offers denser cost signals regarding the relationship between the textual constraint and the agent's every action. It informs the agent which behaviors are risky and which are safer, thereby enhancing safety and aiding model performance.Our experiments demonstrate that the proposed method can effectively address Challenge 1 and Challenge 2. In both 3D navigation [19] and 2D grid exploration [46] tasks, agents trained using our method achieve significantly lower violation rates (up to 4.0x) compared to agents trained with ground-truth cost functions while maintaining comparable rewards and more importantly, our method obtains the Pareto frontier [10] . In addition to this, our method has zero-shot adaptation capability to adapt to constraint-shift environments without fine-tuning. 
related work: 
 Safe RL. Safe RL aims to train policies that maximize reward while minimizing constraint violations [11, 13] . In prior work, there are usually two ways to learn safe policies: (1) consider cost as one of the optimization objectives to achieve safety [48, 6, 49, 39, 44, 33] , and (2) achieve safety by leveraging external knowledge (e.g. expert demonstration) [35, 32, 1, 47] . These typical safe RL algorithms require either human-defined cost functions or human-specified cost constraints which are unavailable in the tasks that constraints are given by natural language.RL with Natural Language. Prior works have integrated natural language into RL to improve generalization or learning efficiency in various ways. For example, Hermann et al. [14] studied how to train an agent that can follow natural language instructions to reach a specific goal. Additionally, natural language has been used to constrain agents to behave safely. For instance, Prakash et al. [29] trained a constraint checker to predict whether natural language constraints are violated. Yang et al. [46] trained a constraint interpreter to predict which entities in the environment may be relevant to the constraint and used the interpreter to predict costs. Lou et al. [28] used pre-trained language models to predict the cost of specific states, avoiding the need for artificially designed cost functions. However, previous methods cannot uniformly handle textual constraints with one framework, which limits their applicability.Credit Assignment in RL. Credit assignment studies the problem of inferring the true reward from the designed reward. Prior works have studied improving sample efficiency of RL algorithms through credit assignment, for example by using information gain [17] , as an intrinsic bonus reward to aid exploration. Goyal et al. [12] proposed the use of natural language instructions to perform reward shaping to improve the sample efficiency of RL algorithms. Liu et al. [26] learned to decompose the episodic return as the reward for policy optimization. However, to the best of our knowledge, our work is the first to apply credit assignment to safe RL.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See Section 7 Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
preliminaries: 
 Problem formulation.Trajectory-level constraint problem can be formed as the Constrained Non-Markov Decision Process (CNMDP) [3, 43] , and it can be defined by the tuple <S, A, T, R, γ, C, Y, τ * >. Here S represents the set of states, A represents the set of actions, T represents the state transition function, R represents the reward function, and γ ∈ (0, 1) represents the discount factor. In addition, Y represents the set of trajectory-level textual constraints (e.g., "You have 10 HP, you will lose 3 HP every time you touch the lava, don't die."), which describes the constraint that the agent needs to obey across the entire trajectory. C represents the cost function determined by y ∈ Y . τ * represents the set of historical trajectories.RL with constraints. The objective for the agent is to maximize reward while obeying the specified textual constraint as much as possible [27] . Thus, in our task setting, the agent needs to learn a policy π: S × Y × τ * → P (A) which maps from the state space S, textual constraints Y and historical trajectories τ * to the distributions over actions A. Given a y, we learn a policy π that maximizes the cumulative discounted reward J R while keeping the cumulative discounted cost (average violation rate) J C below a constraint violation budget B C (y):EQUATIONHere B C (y) and C(s t , a t , y, τ t ) are two functions both depending on textual constraint y. τ t represents the historical trajectory at time step t.Episodic RL. Similar to the task with episodic rewards [4] , in our task setting, a cost is only given at the end of each trajectory τ when the agent violates the textual constraint y. In other words, before violating y, the cost C(s t , a t , y, τ t ) = 0 for all t < T . For simplicity, we omit the discount factor and assume that the trajectory length is at most T so that we can denote a T as the final action that causes the agent to violate y without further confusion. Therefore, the constraint qualification of the objective in RL with constraints becomesJ C (π) = E π [C(s T , a T , y, τ T )] ≤ B C (y).Due to the sparsity of cost, a large amount of rollout trajectories are needed to help the agent distinguish the subtle effects of actions on textual constraint [24] . This situation will become more serious when trajectory-level constraints are complex and difficult to understand.  
ttct: trajectory-level textual constraints translator: 
 In this section, we introduce our proposed framework TTCT ( Trajectory-level Textual Constraints Translator) as shown in Figure 1 . TTCT has two key components: the text-trajectory alignment component and the cost assignment component. The text-trajectory alignment component is used to address the violations prediction problem. The cost assignment component is used to address the sparse cost problem.We propose a component to learn from offline data to predict whether a given trajectory violates textual constraints. The core idea of this component is to learn trajectory representations under textual supervision and connect trajectory representation to text representation. If the distance between the two representations in the embedding space is sufficiently close, we can consider that the given trajectory violates the constraint. Our approach does not require modeling entities of the environment like previous work, such as [46] , which involves labeling hazardous items artificially in every observation. Instead, we model this task as a trajectory-text multimodal learning problem. Hence, our method can learn trajectory representations and text representations from the pairs (trajectory, trajectory-level textual constraint). We believe that learning from the supervision of natural language could not only enhance the representation power but also enable flexible zero-shot transfer [30] .Formally, given a batch of N (trajectory τ , trajectory-level textual constraint y) pairs. For each pair, the trajectory corresponds to the text, indicating that the given trajectory violates the given textual constraint. The trajectory can be defined as τ = (s 1 , a 1 , s 2 , a 2 , ..., s T -1 , a T -1 , s T , a T ), where T is the step at which the textual constraint y is first violated by the trajectory. Here s t is a d sdimensional observation vector. Each state in the trajectory is processed by a state encoder to obtain a representation v s t , also action is processed by an action encoder to obtain a representation v a t . Then, we concatenate v s t and v a t to obtain a vector representation v t for each state-action pair. After that, we learn separate unimodal encoders g T and g C for the trajectory and textual constraint, respectively. The trajectory encoder g T utilizes a causal transformer to extract the trajectory representation from the input state-action representation sequence {v t } T t=1 :EQUATIONwhere H t is a d H -dimensional vector. The final embedding H T is used as the representation for the entire trajectory. Specifically, the causal Transformer processes the trajectory sequence by maintaining a left-to-right context while generating embeddings. This allows the model to capture temporal dependencies within the trajectory and obtain the embeddings for time steps before T . The textual constraint encoder g C is used to extract features that are related to the constraints and it could be one of a wide variety of language models:EQUATIONwhere L is a d L -dimensional vector representation. Then we define symmetric similarities between the two modalities with cosine distance:EQUATION)where α is a learnable parameter and T means that use last embedding of trajectory to calculate similarity. And we use softmax to calculate trajectory → text and text → trajectory similarities scores:EQUATIONLet q τ →y (τ ), q y→τ (y) indicate the ground-truth similarity scores, where the negative pair (trajectory doesn't violate textual constraint) has a probability of 0 and the positive pair (trajectory violate textual constraint) has a probability of 1. In our task setting, a trajectory can correspond to multiple textual constraints, and vice versa. For example, two textual constraints such as "Do not touch lava" and "After stepping on water, do not touch lava" might both be violated by a single given trajectory. This many-to-many relationship between trajectories and textual constraints implies that the same textual constraints (or different textual constraints with the same semantics) can apply to multiple trajectories, while a single trajectory may comply with several textual constraints. So there may be more than one positive pair in q τ →y i (τ ) and q y→τ i (y). Therefore, we use Kullback-Leibler (KL) divergence [23] as the multimodal contrastive (MC) loss to optimize our encoder similar to [42] :EQUATION)where D is the training set.In addition to the multimodal contrastive (MC) loss, we also introduce a within-trajectory (WT) loss. Specifically, suppose we have a trajectory's representation sequence (H 1 , H 2 , H 3 , ..., H (T -1) , H T ) and its corresponding textual constraint embedding L, we can calculate cosine similarity sim t (τ, y) between embedding H t and L using Equation 4. Then we can calculate similarity scores within the trajectory:EQUATIONDifferent from p y→τ i (y) in Equation 5, which measures similarity scores across N trajectories, Equation 7 is used to measure the similarity scores of different time steps within a trajectory. The reason for doing this is that the textual constraint is violated at time step T , while in the previous time steps the constraint is not violated. Therefore, the instinct is to maximize the similarity score between the final trajectory embedding H T and the textual constraint embedding L, while minimizing the similarity score between all previous time step embeddings (H 1 , H 2 , H 3 , ..., H (T -1) ) and the textual constraint embedding L. Based on this instinct, we introduce within-trajectory (WT) loss:EQUATIONwhere the first term is responsible for minimizing the similarity score for the embedding of time steps before T in the trajectory sequence, while the second term is responsible for maximizing the similarity score for the embedding of time step T .By combining these two losses L W T , L M C , we can train a text encoder to minimize the distance between embeddings of semantically similar texts, while simultaneously training a trajectory encoder to minimize the distance between embeddings of semantically similar trajectories. Crucially, this approach enables us to align the text and trajectory embeddings that correspond to the same semantic constraint, fostering a cohesive representation of their shared meaning, and further determining whether the trajectory violates the constraint by calculating embedding similarity.After solving the problem of predicting whether a given trajectory violates a textual constraint, there is still the issue of cost sparsity. Motivated by the works of temporal credit assignment [26] , we propose a component to capture the relationship between the state-action pair and the textual constraint and assign a cost value to each state-action based on its impact on satisfying the constraint.Specifically, suppose we have a (trajectory τ , textual constraint y) pair and its representation ({H t } T t=1 , L) obtained from text-trajectory alignment component. The textual constraint representation L is processed by an episodic-cost prediction layer F e to obtain a predicted episodic cost Ĉ(y) = sigmoid(F e (L)) for the entire trajectory. We expect the episodic cost can be considered as the sum of cost on all non-violation state-action pairs: Ĉ(y) = ∑ T -1 t=1 ĉ(s t , a t , y, τ t ). To evaluate the significance of each timestep's action relative to textual constraints, we employ an attention mechanism:EQUATIONHere we regard the text representation as the query, each time step's representation in the trajectory as the key, and compute the attention score e t based on the cosine similarity metric. After that, we use the sigmoid function to make sure the score falls within the range of 0 to 1. Each attention score e t quantifies the degree of influence of the state-action pair (s t , a t ) on violating the textual constraint.Then we obtain an influence-based representation H * t = e t H t . To predict the cost ĉ(s t , a t , y, τ t ), we incorporate a feed-forward layer called the cost assignment layer F c and output the predicted non-violation single step cost as:EQUATIONThe loss function for measuring inconsistency of the episodic cost Ĉ(y) and the sum of non-violation single step costs ĉ(s t , a t , y, h t ) can be formed as:EQUATIONThis mutual prediction loss function L CA is only used to update the episodic-cost prediction layer and cost assignment layer, and not to update the parameters of the trajectory encoder or text encoder during backpropagation. This helps ensure the validity of the predictions by preventing overfitting or interference from other parts of the model.The effectiveness of this component comes from two main sources. First, the text-trajectory alignment component projects semantically similar text representations to nearby points in the embedding space, allowing the episodic-cost prediction layer to assign similar values to embeddings with close distances. This aligns with the intuition that textual constraints with comparable violation difficulty should yield similar episodic costs. Second, the cost assignment layer leverages the representational power of the text-trajectory alignment component to capture complex relationships between state-action pairs and constraints, enabling accurate single-step cost predictions.In our experiment, we simultaneously train the text-trajectory alignment component and cost assignment component end-to-end with a uniform loss function L T T CT :EQUATIONBy doing this, we can avoid the need for separate pre-training or fine-tuning steps, which can be timeconsuming and require additional hyperparameter tuning. Also, this can enable the cost assignment component to gradually learn from the text-trajectory alignment component and make more accurate predictions over time.In the test phase, at time step t we encode trajectory τ t and textual constraint y with Equation 3 and Equation 2 to obtain the entire trajectory embedding H t and text embedding L. Then we calculate distance score sim(τ t , y) using Equation 4. The predicted cost function ĉ is given by:EQUATION)where β is a hyperparameter that defines the threshold of the cost prediction. ĉ = 1 indicates that the TTCT model predicts that the textual constraint is violated. Otherwise, if the textual constraint is not violated by the given trajectory, we assign a predicted cost using Equation 10. 
policy training: 
 Our Trajectory-level textual constraints Translator framework is a general method for integrating free-form natural language into safe RL algorithms. In this section, we introduce how to integrate our TTCT into safe RL algorithms so that the agents can maximize rewards while avoiding early termination of the environment due to violation of textual constraints. To enable perception of historical trajectory, the trajectory encoder and text encoder are not only used as frozen plugins g T and g C for cost prediction but also as trainable sequence models g * T and g * C for modeling historical trajectory. This allows the agent to take into account historical context when making decisions. To further improve the ability to capture relevant information from the environment, we use LoRA [18] to fine-tune both the g * T and g * C during policy training. The usage of g T , g C and g * T , g * C is illustrated in Appendix A.4 Figure 8 .Formally, let's assume we have a policy π with parameter ϕ to gather transitions from environments. We maintain a vector to record the history state-action pairs sequence, and at time step t we use g * T and g * C to encode τ t-1 and textual constraint y so that we can get historical context representation H t-1 and textual constraint representation L. The policy selects an action a t = π θ (o t , H t-1 , L) to interact with environment to get a new observation o t+1 . And we update τ t with the new state-action pair (o t , a t ) to get τ t . With τ t and L, ĉt can be predicted according to Equation 13. Then we store the transition into the buffer and keep interacting until the buffer is full. In the policy updating phase, after calculating the specific loss function for different safe RL algorithms, we update the policy π with gradient descent and update g * T , g * C with LoRA. It is worth noting that g T and g C are not updated during the whole policy training phase, as they are only used for cost prediction. The pseudo-code and more details of the policy training can be found in Appendix A.4. 
experiments: 
 Our experiments aim to answer the following questions: (1) Can our TTCT accurately recognize whether an agent violates the trajectory-level textual constraints? (2) Does the policy network, trained with predicted cost from TTCT, achieve fewer constraint violations than trained with the ground-truth cost function? (3) How much performance improvement can the cost assignment (CA) component achieve? (4) Does our TTCT have zero-shot capability to be directly applicable to constraint-shift environments without any fine-tuning? We adopt the following experiment setting to address these questions. , where orange tiles are lava, blue tiles are water and green tiles are grass. Agents need to collect reward objects in the grid while avoiding violating our designed textual constraint for the entire episode. (b) Robot navigation task SafetyGoal that is built in Safety-Gymnasium [19] , where there are multiple types of objects in the environment. Agents need to reach the goal while avoiding violating our designed textual constraint for the entire episode. (c) LavaWall [5] , a task has the same goal but different hazard objects compared to Hazard-World-Grid.Task. We evaluate TTCT on two tasks (Figure 2 (a, b )): 2D grid exploration task Hazard-World-Grid (Grid) [46] and 3D robot navigation task SafetyGoal (Goal) [19] . And we designed over 200 trajectory-level textual constraints which can be grouped into 4 categories, to constrain the agents. A detailed description of the categories of constraints will be given in Appendix A.1. Different from the default setting, in our task setting, when a trajectory-level textual constraint is violated, the environment is immediately terminated. This is a more difficult setup than the default. In this setup, the agents must collect as many rewards as possible while staying alive.Baselines. We consider the following baselines: PPO [36] , PPO_Lagrangian(PPO_Lag) [33] , CPPO_PID [39] , FOCOPS [49] . PPO does not consider constraints and simply aims to maximize the average reward. We use PPO to compare the ability of our methods to obtain rewards. As for the last three algorithms, we design two training modes for them. One is trained with standard ground-truth cost, where the cost is given by the human-designed violation checking functions, and we call it ground-truth (GC) mode. The other is trained with the predicted cost by our proposed TTCT, which we refer to as cost prediction (CP) mode. More information about the baselines and training modes can be found in Appendix A.1 A.2.Metrics. We take average episodic reward (Avg. R) and average episodic cost (Avg. C) as the main comparison metrics. Average episodic cost can also be considered as the average probability of violating the constraints. The higher Avg. R, the better performance, and the lower Avg. C, the better performance. 332B/DJ &332B3,' )2&236 $YJ5 332 &3RXU *& 332B/DJ &332B3,' )2&236 $YJ& 332 &3RXU *& (a) Hazard-World-Grid 332B/DJ &332B3,' )2&236 $YJ5 332 &3RXU *& 332B/DJ &332B3,' )2&236 $YJ& 332 &3RXU *& (b) SafetyGoalThe evaluation results are shown in Figure 3 and the learning curves are shown in Figure 4 . We can observe that in the Hazard-World-Grid task, compared with PPO, the policies trained with GC can reduce the probability of violating textual constraints to some extent, but not significantly. This is because the sparsity of the cost makes it difficult for an agent to learn the relevance of the behavior to the textual constraints, further making it difficult to find risk-avoiding paths of action. In the more difficult SafetyGoal task, it is even more challenging for GC-trained agents to learn how to avoid violations. In the CPPO_PID and FOCOPS algorithms trained with GC mode, the probability of violations even rises gradually as the training progresses. In contrast, the agents trained with predicted cost can achieve lower violation probabilities than GC-trained agents across all algorithms and tasks and get rewards close to GC-trained agents.These results show that TTCT can give an accurate predicted episodic cost at the time step when the constraint is violated, it can also give timely cost feedback to non-violation actions through the cost assignment component so that the agents can find more risk-averse action paths. And these results answer questions (1) and (2). The discussion about the violation prediction capability of the text-trajectory component can be found in Appendix B.1. The interpretability and case study of the cost assignment component can be found in Appendix B.2.To study the influence of the cost assignment component. We conduct an ablation study by removing the cost assignment component from the full TTCT. The results of the ablation study experiment are shown in Figure 4 . We can observe that even TTCT without cost assignment can achieve similar performance as GC mode. And in most of the results if we remove the cost assignment component, the performance drops. This shows that our text trajectory alignment component can accurately predict the ground truth cost, and the use of the cost assignment component can further help us learn a safer agent. These results answer questions (3). Pareto frontier. Multi-objective optimization typically involves finding the best trade-offs between multiple objectives. In this context, it is important to evaluate the performance of different methods based on their Pareto frontier [10] , which represents the set of optimal trade-offs between the reward and cost objectives. We plot the Pareto frontier of policies trained with GC and policies trained with CP on a two-dimensional graph, with the vertical axis representing the reward objective and the horizontal axis representing the cost objective as presented in Figure 6 . The solution that has the Pareto frontier closer to the origin is generally considered more effective than those that have the Pareto frontier farther from the origin. We can observe from the figure that the policies trained with predicted cost by our TTCT have a Pareto frontier closer to the origin. This proves the effectiveness of our method and further answers Questions (1) and (2). Zero-shot transfer capability. To explore whether our method has zero-shot transfer capability, we use the TTCT trained under the Hazard-World-Grid environment to apply directly to a new environment called LavaWall (Figure 2 (c)) [5] , without fine-tuning. The results are shown in Figure 7 . We can observe that the policy trained with cost prediction (CP) from TTCT trained under the Hazard-World-Grid environment can still achieve a low violation rate comparable to the GC-trained policy. This answers Question (4). Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: See Section 6 and Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion and future work: 
 In this paper, we study the problem of safe RL with trajectory-level natural language constraints and propose a method of trajectory-level textual constraints translator (TTCT) to translate constraints into a cost function. By combining the text-trajectory alignment (CA) component and the cost assignment (CA) component, our method can elegantly solve the problems of predicting constraint violations and cost sparsity. We demonstrated that our TTCT method achieves a lower violation probability compared to the standard cost function. Thanks to its powerful multimodal representation capabilities, our method also has zero-shot transfer capability to help the agent safely explore the constraint-shift environment. This work opens up new possibilities for training agents in safe RL tasks with total free-form and complex textual constraints.Our work still has room for improvement. The violation rate of our method is not absolute zero.In future work, we plan to investigate the application of TTCT in more complex environments and explore the integration of other techniques such as meta-learning [16] to further improve the performance and generalization capabilities of our method.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: See Section 6.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See the Appendix A.3.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Appendix C.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: They are open-sourced.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 DatasetIn our task setting, humans need to provide high-level textual instruction to the agent for the entire trajectory, and then TTCT can predict the cost based on the real-time state of the agent's exploration so that the agent can learn a safe policy with the predicted cost. Thus our dataset is comprised of two parts: the trajectory-level textual constraints and the environments.Trajectory-level textual constraints: To generate textual constraints, we first explore the environment using a random policy, collecting a large amount of offline trajectory data. Then we design a descriptor that automatically analyzes trajectories and gives natural language descriptions based on predefined templates. To validate whether our method can understand different difficulty levels of textual constraints, we design four types of trajectory-level textual constraints. The four types of textual constraints are:1. Quantitative textual constraint describes a quantitative relationship in which an entity in the environment cannot be touched beyond a specific number of times, which can be interpreted as the entity's tolerance threshold, and when the threshold is exceeded, the entity may experience irrecoverable damage.2. Sequential textual constraint describes a sequence-based relationship, where the occurrence of two or more distinct actions independently may not pose a risk, but when they occur in sequence, it does. For instance, it's safe to drink or drive when they happen independently, but when they occur in sequence (i.e., drinking first and then driving), it becomes dangerous.Relational textual constraint describes constraints on the relationships between an agent and entities in its environment, such as maintaining a certain distance, always being in front of that entity, or not staying too far from it.4. Mathematical textual constraints often do not provide explicit instructions to the agent regarding what actions to avoid, but rather present logical descriptions that demand the model's ability to reason mathematically. This type of constraint thereby presents a higher cognitive burden for our TTCT to comprehend.Examples of four types of trajectory-level textual constraints are included in Table 2 . Then we randomly split the (trajectory, textual constraint) pairs into 80% training and 20% test sets. And we use the training set to train our TTCT end-to-end.Environments: We use two environments Hazard-World-Grid and SafetyGoal as main benchmarks and a environment LavaWall to evaluate the zero-shot transfer capability:1. Hazard-World-Grid. The environment is a 12 × 12 grid, with the gray walls surrounding the perimeter. The agent can only explore within the grid. Inside the grid, some items provide rewards: blue keys, red balls, and yellow boxes. Collecting all of these items will be considered as completing the task. Additionally, there are hazardous materials in the grid, where orange tiles are lava, cyan tiles are water, and green tiles are grass. During exploration, the agent can only see a range of 7 × 7 pixels ahead, resulting in an observation space with size 7 × 7 × 3.SafetyGoal. Robot navigation task, the environment has a navigation target, which is completed when the robot reaches the target. The environment also contains vases (cyan squares) and Hazards (Mediumslateblue circles). The vases can be moved by the robot and hazards are fixed.LavaWall. The LavaWall environment shares the same task goal as Hazard-World-Grid but with different hazardous materials. Whenever the environment is initialized, a random lava wall with only one entrance is generated in the grid, and the agent must learn to obtain the reward on the other side while avoiding stepping on lava.For each episode, we place the agent at a randomized start location, fill the environment with objects, and randomly select a textual constraint from the constraint pool. We manually design the violation checking function for every constraint to determine whether the environment violates the textual constraint, and when the textual constraint is violated, the environment will provide a cost with a value of 1.0 to the agent and terminate the environment (the manually designed function will not be used in the cost prediction (CP) mode, all the costs will be predicted by our TTCT in CP mode).(1) PPO [36] : This algorithm does not consider constraints, and simply considers maximizing the average reward, which we use to compare the ability of our methods to obtain rewards.(2) PPO_Lagrangian(PPO_Lag) [33] : This algorithm transforms a constrained optimization problem into an unconstrained optimization problem via Lagrange multipliers (3) CPPO_PID [39] : This algorithm PID to control Lagrange multiplier, which solves the cycle fluctuation problem of PPO-Lagrange. (4) FOCOPS [49] : This algorithm finds the optimal update policy by solving a constrained optimization problem in the nonparameterized policy space, then projects the update policy back into the parametric policy space.We provide training pseudocode of TTCT in Figure 9 . The text encoder we use is Bert ((https: //huggingface.co/google-bert/bert-base-uncased/tree/main) [7] . And the hyperparameters we use are shown in Table 3 . We use the same hyperparameters across different tasks' datasets. We conduct the training on the machine with A100 GPU for about 1 day.We provide pseudocode of training policy with predicted from TTCT in Algorithm 1. To enable the agent to comply with diverse types of constraints, we launch async vectorized environments with different types of constrained environments during roll-out collection. The agent interacts with these environments and collects transitions under different constraint restrictions to update its policy.During policy updates, we fine-tune the trajectory encoder g * H and text encoder g * L using LoRA [18] at every epoch's first iteration, while not updating encoder parameters in other iterations, which saves training time. for each episode do for t = 1, T do9:Select a action a t = π θ (o t , H t-1 , L) Predict cost ĉt according to Equation 1313:Store transition (o t , a t , r t , ĉt , τ t-1 , y) in buffer 14:end for 15:end for 16:Sample batch of N transitions from buffer. 17:Encode {τ j } N i=1 and {y i } N i=1 with g T , g CCalculate specific loss function To further study the ability of our text-trajectory alignment component to predict violations, we conduct an experiment given a batch of trajectory-text pairs and we use the text-trajectory alignment component to encode the trajectory and textual constraint, and then calculate the cosine distance with Equation 5between every two embeddings across two modal. We plot a sample of heatmap of calculated cosine similarity and ground-truth as presented in 10. Further, We plot the receiver operating characteristic (ROC) [8] curve to evaluate the performance of the text-trajectory alignment component as presented in Figure 11 . AUC (Area Under the Curve) [8] values indicate the area under the ROC curve. The AUC value of our violations prediction result is 0.98. Then We set threshold β equal to the best cutoff value of the ROC curve. We determine whether the trajectory violates a given textual constraint by:EQUATIONThe results are shown in Table 4 . These results indicate that our text-trajectory alignment component can accurately predict whether a given trajectory violates a textual constraint. We visualize the assigned cost for every state-action pair to demonstrate that the cost assignment component could capture the subtle relation between the state-action pair and textual constraint.Our intuition is that we should assign larger costs to state-action pairs that lead to violations of trajectory-level textual constraints, and smaller or negative values to pairs that do not contribute to constraint violations. Using the Hazard-World-Grid environment as an example, we choose three different types of constraints to show our results in Figure 12 . The first row shows the textual constraint, the second row shows the trajectory of the agent in the environment, and to make it easier to visualize, we simplify the observation s t by representing it as a square, denoting the entity stepped on by the agent at time step t. The square emphasized by the red line indicates the final entity that makes the agent violate textual constraint at time step T . The third row shows the predicted cost of the agent at every time step t and deeper colors indicate larger cost values.The (a) constraint is mathematical textual constraint: "You only have 20 HP. Lava and grass are dangerous, they will make you lose 3 and 2 HP, respectively. However, water can regenerate 1 HP. Please don't die.". This constraint describes the two dangerous entities lava and grass, and the beneficial entity water. From the third-row heat map, we can observe that our cost assignment component assigns a high cost to the action that steps on lava or grass, with the cost increasing as the agent approaches the constraint-violating situation. Not only that, the CA component also recognizes the different levels of danger posed by lava and grass. Since stepping on lava will deduct 3 HP while stepping on grass will deduct 2 HP, the CA component assigns a larger cost value at time step 13 -14 compared to the cost value at time step 3 -5.The (b) constraint is quantitative textual constraint: "Lava is dangerous. Don't touch them more than eight times!". When stepping on the floor, the CA component considers these actions to be safe and assigns a cost of nearly 0. However, when the agent steps onto lava, it assigns a higher cost, especially when the agent steps on lava for the eighth time. At this point, our CA component concludes that the situation has become extremely dangerous, and one more step on lava will violate the constraint, thus giving the highest cost compared to the before time steps. The (c) constraint is sequential textual constraint: "After you touch lava, don't step on grass!". Our CA component captures two relevant entities: lava and grass, and understands the sequential relationship between entities. When the agent first steps onto the grass, the text-trajectory alignment component determines that this action does not violate the textual constraint. However, after the agent steps onto lava and then steps onto the grass for the second time, the text-trajectory alignment component detects the cosine similarity sim(τ, y) greater than the threshold β, thereby violating the textual constraint. And CA component captures that the key trigger condition for violating constraint is stepping onto the lava, therefore assigning a relatively larger cost to such actions at time step 7 -9. The cost assignment component also assigns relatively small costs to some safe actions that are stepping on the floor, such as steps 13 -15. This is because it detects that the agent, although hasn't stepped on grass yet, is trending towards approaching grass, which is a hazardous trend, thus providing a series of gradually increasing and small costs. This demonstrates that our component not only monitors key actions that lead to constraint violations but also monitors the hazardous trend and nudging the agent to choose relatively safer paths.To evaluate the agent's understanding of different types of trajectory-level textual constraints, we conducted an additional experiment using the CPPO_PID algorithm. During training, we separately tracked the average episodic reward and the average episodic cost for three types of textual constraints as presented in Figure 13 . From the learning curves, we can observe that for every type of constraint, our CP mode can achieve the lowest violation rate compared to CP without the CA component mode and ground-truth cost (GC) mode. We perform the trajectory length sensitivity analysis on Hazard-World-Grid. Since our framework is mainly used for reinforcement learning policy training where data is typically provided as input in batches, we counted the inference time for different trajectory lengths with 64 as the batch size, using the hardware device V100-32G. Figure 14 shows that the average inference time per trajectory is 10ms for trajectories of length 100. Figure 15 : Empirical analyses on Hazard-World-Grid with varying text encoders. We choose three different models, transformer-25M [41] , gpt2-137M [31] , and bert-base-uncased-110M [7] .We present the learning curve of our main experiment in Figure 4 . We also present the main results and ablation study in Table 5 and 6 .Table 5 : Evaluation results of our proposed method TTCT. ↑ means the higher the reward, the better the performance. ↓ means the lower the cost, the better the performance. Each value is reported as mean ± standard deviation and we shade the safest agents with the lowest averaged cost violation values for every algorithm. Our method can help train agents in reinforcement learning tasks with total free-form natural language constraints, which can be useful in various real-world applications such as autonomous driving, robotics, and game playing. There are still limitations to our work. Our method may not be able to completely eliminate constraint violations. Our method has the contextual bottleneck as the length of the trajectory increases. We performed a trajectory length sensitivity analysis on Hazard-World-Grid. As shown in Figure 16 , initially increasing the trajectory length improves performance because longer trajectories may provide more dependencies. However, beyond a certain point, further increases in trajectory length result in a slight drop in AUC. This decline is because the trajectory encoder has difficulty capturing global information. We consider this bottleneck to be related to the transformer's encoding capability. Justification: We show our approach can predict cost with trajectory-level natural language. The predicted cost by our approach can help train a safer policy than the ground-truth cost. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: All the links, data, code are provided in the anonymous form.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: Learning joint embeddings between text and trajectories is useful and topical.

Reviewers note that the main weakness is how the work relates to other research. Authors put this forward as a "unified framework" for reasoning about language and trajectories. With a focus on the fact that their method doesn't require manual annotations and alignment between the concepts in the trajectory and the linguistic concepts. But there is a tremendous literature that is being ignored which does exactly this.

I would really encourage the authors to engage with this literature because it points the way toward more interesting experiments for followup work. Reviewers point to work in safe RL, another area which does the same has to do with robots that follow linguist commands including those that incorporate notions of safety with LTL. This work often doesn't require annotations or alignment just like the work here.

In addition, there are so many other linguistic phenomena and logics to consider when aligning language to trajectories. Claiming that this is a unified framework for any constraints and notions of safety is far too much.

That being said, reviewers found that the topical nature of the work overrides these concerns and I agree. But it's to the author's benefit to resolve these issues with the manuscript. It will only allow them to engage with a much wider community.

Reviews: review: paper_summary: paper_summary: - This work proposes an approach to training RL agents with constraints. The proposed approach learns language embeddings of constraints and embeddings of trajectories. During training, a similarity score is used to align the space of constraint embeddings and the space of trajectory embeddings.
- Moreover, the language embedding of the constraint is also used to do fine-grained credit assignment across the actions in a trajectory. The authors find that this improves performance.
strength_weakness: strength_weakness: - I have a hard time understanding where this work stands in relation to prior works. The main contribution seems centered around the approach of encoding trajectory level constraints using natural language. But both trajectory level constraints and natural language constraints have been explored independently. And the combination of these two aspects seems straightforward: encode the trajectory level constraint using natural language. I may be overlooking something that makes this less straightforward. I don't quite follow the explanation for novelty on lines 101-104. The Related Works section mentions prior work which learns language representations of constraints, but says that these works do not consider trajectory-level constraints. But because the constraints are given in natural language, it doesn't seem that anything in principle prevents these prior approaches from doing so? What is the technical innovation of the proposed work that allows for trajectory level constraints?
- A few things seem missing from the setup (see questions)
- I have concerns about the soundness of the described approach. Where does the ground truth come from when training the text-trajectory alignment component? As far as I can tell, there is notion of a textual constraint but not a notion of a ground truth verification of a constraint. From what I understand of appendix 1, the textual constraint is determined to be violated by a discriminator model (see questions). In any case, the authors should explain how constraints are determined to be violated. This matters because it means that the discriminator model and the agent could both make the same systematic linguistic errors.
- I have concerns about the soundness of the approach in general. Using natural language to describe constraints seems at odds with the enterprise of making RL more "safe." Natural language can be ambiguous, and machine learned representations of language can be imperfect, and prone to making semantic errors. In contrast to other approaches which provide formal guarantees on behavior [1], this seems much less safe. However, I am conscious that others in the field think differently, given the prior work. 

## minor
- Figure 3: Labels are missing in the legend
- Line 297: "Table" --> "Figure"

## references
[1] Fulton, Nathan, and André Platzer. "Safe reinforcement learning via formal methods: Toward safe control through proof and learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.
questions: questions: - Line 174: What is a "negative pair"? Is it a constraint and a trajectory that violates that constraint?
- Line 175: Given that the mapping from trajectories to constraints is many-to-many, doesn't $q$ need to be normalized so that the probabilities sum to 1? That is, there are many positive pairs that contain the same constraint, so they should all receive an equal share of positive probability.
- Fig 5: I think I'm missing something basic: how are is the Pareto front determined? I could assume that per cost, the best reward possible can be determined by some procedure? Is this what was done?
- Section 6: Where do the textual constraints come from? Are they human annotated? Appendix 1 seems to suggest not.
  - Is it guaranteed that the discriminator can tell when a textual constraint is violated?
- Line 289: What is the difference between ground-truth mode and cost prediction mode? Is it the cost assignment?
limitations: limitations: Limitations are mentioned in the appendix.
paper_summary: paper_summary: * The paper proposes U3T, a new system for more robust safe RL under general text constraints.
* The key innovation of the paper is to generalize text constraints such that the constraints don’t refer to a specific entity/state and addressing cost sparsity where constraints are only violated at the final time step.
* The proposed system consists of a trajectory encoder and text encoder that are trained jointly using a contrastive loss defined by the KL-divergence.
* The authors solve cost sparsity by identifying actions/states that contribute to the violation of a constraint in an attention mechanism.
strength_weakness: strength_weakness: * I found some details missing in the text. I discuss some portions that could use clarification here.
    * Which text encoder was used in this experiment? I couldn’t find this in the paper and from your appendix section A.4, it seems untrained. You refer to finetuning with LORA though so I was confused. A small nit, could be interesting to connect the text encoder performance to results. I realize space is tight but if these could be incorporated into the main paper, that would clarify and make the experiments easier to interpret.
    * I’m unsure how to interpret the average reward results in comparison to the cost results. See questions. My intuition was that rewards should be larger given the pareto frontier results. 
* I’m having trouble differentiating this paper from the Lou et. al. paper. I took a quick look at Lou et. al. --  I can understand that entities needed to be modeled in the paper but from what I can tell, that paper could also be applied in the settings you considered given the use of the pretrained LLM. From Figure 1, I couldn’t see the specific entity problem you were mentioning in the paper. Why wasn’t this added as a baseline? Could you provide more discussion on why these two are different? I think the approach in this paper has considerable novelty so I won’t say that this is entirely negative but it does diminish the contribution slightly.
questions: questions: * I noticed that generally, while the cost of the method would decrease, the reward wouldn’t change significantly (Figure 2). I actually found this a bit unintuitive. I figured that lower cost would imply a longer trajectory and therefore, longer windows to collect reward over, so average reward could increase. I found this discussion missing in the text. Could this be elaborated?
* How long were the trajectories used in the paper? (What was the average size of T?) Could there be a context bottleneck?
limitations: limitations: * See question about context bottleneck. Could this be a limitation to discuss?
paper_summary: paper_summary: This paper introduces a type of universal natural language constraint to model diverse real-world safety requirements. Also, to avoid using the specific human-designed cost function, this paper introduces a Unified Trajectory-Level Textual Constraints Translator (U3T) for aligning text with trajectories and assigning costs via attention mechanisms. By conducting experiments on two environments, this paper reaches the conclusion that U3T can accurately predict whether a given trajectory violates the natural language constraint, and the policy trained with predicted cost can behave more safely.
strength_weakness: strength_weakness: 1. Complexity and computational cost: Implementing U3T involves complex multimodal learning and attention mechanisms, which may require significant computational resources and time costs, especially when dealing with large-scale datasets or real-time applications.
2. The authors do not specify exactly what model of text encoder was used in the experiments.
3. Data Dependency: Effective implementation of U3T heavily relies on high-quality and well-annotated textual and trajectory data, which may be challenging to acquire and curate in some practical applications.
questions: questions: 1. Is it feasible for the authors to experimentally analyze the inference speed of the proposed framework?
2. Could the authors specify the model of the text encoder used in the experiments and provide an in-depth exploration of its selection rationale?
3. Leveraging large language models for text understanding shows promise. Have the authors considered strategies for integrating these models to augment the comprehension and deconstruction of intricate textual constraints?
limitations: limitations: The authors have discussed the limitations in the paper.

Comments: title: Response to authors
Official Comment by Reviewer T5E9
comment: **Thank the reviewer for raising this question. Regarding the concept of a "unified framework," we would like to clarify the following:**

When we discuss a "unified framework," we are referring to the ability to consistently handle and represent textual constraints at the trajectory level. While previous methods could use natural language to describe constraints, they often required different state/trajectory encoding models or decision processes to handle different types of semantic constraints. This indicates that these methods lacked a unified framework capable of addressing complex logical associations in both spatial and temporal dimensions.

**Specifically:**

1. **Limitations in the spatial dimension:** Previous methods mainly focused on state descriptions in the spatial dimension, meaning they could only check if a constraint was violated at specific states. This "point-based" approach fails to address sequential logic in the "line-based" temporal dimension, limiting the ability to describe and constrain long-term behavior within a trajectory.
2. **Unification of the training method:** More importantly, our method no longer relies on manually labeled datasets for token classification within the text to determine which entity in the environment a constraint pertains to. This classification approach typically requires extensive manual labeling and is often restricted by specific semantic types. Our method leverages the text itself as a supervisory signal, which was not considered in previous work. The text not only describes constraints but also provides a unified supervisory training signal for trajectory encoding. This dual role allows our method to naturally handle various types of semantic constraints without needing specific model adjustments for different constraint types.

Therefore, the so-called "unified framework" refers not only to the ability to express constraints in natural language but also to a system capable of uniformly encoding trajectories and handling various complex trajectory-level constraints, which our method achieves effectively.
I thank the authors for their response. I will keep my score. I think I understand the argument for novelty that the authors are making a little better. But there are still some gaps for me. The rebuttal says that "previous methods fail to offer a unified framework for trajectory-level textual constraints," but I don't really understand what "unified framework" was lacking in the prior work. As long as a constraint can be specified in natural language, can it not be considered part of a "unified framework"?
Dear reviewer cY35,

We hope this message finds you well. We want to check if our reviews address your concerns. If you have any further questions or if there’s anything you’d like to discuss, please feel free to comment further. Thank you for your time again.

Sincerely,

Authors.
Dear Reviewers z2Bi,

We want to make sure that our reviews address your concerns. If you need further clarification, please feel free to contact us. Thank you for taking the time to review our papers.

Sincerely,

Authors.
Thanks for the authors' response. I have no further concern and will lean to keep the score.