Title: From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Safe RL

Multimodal Learning

Temporal Credit Assignment

Conditioned Reinforcement Learning
Abstract: Safe reinforcement learning (RL) requires the agent to finish a given task while obeying specific constraints. Giving constraints in natural language form has great potential for practical scenarios due to its flexible transfer capability and accessibility. Previous safe RL methods with natural language constraints typically need to design cost functions manually for each constraint, which requires domain expertise and lacks flexibility. In this paper, we harness the dual role of text in this task, using it not only to provide constraint but also as a training signal. We introduce the Trajectory-level Textual Constraints Translator (TTCT) to replace the manually designed cost function. Our empirical results demonstrate that TTCT effectively comprehends textual constraint and trajectory, and the policies trained by TTCT can achieve a lower violation rate than the standard cost function. Extra studies are conducted to demonstrate that the TTCT has zero-shot transfer capability to adapt to constraint-shift environments.

From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning

Abstract

Safe reinforcement learning (RL) requires the agent to finish a given task while obeying specific constraints. Giving constraints in natural language form has great potential for practical scenarios due to its flexible transfer capability and accessibility. Previous safe RL methods with natural language constraints typically need to design cost functions manually for each constraint, which requires domain expertise and lacks flexibility. In this paper, we harness the dual role of text in this task, using it not only to provide constraint but also as a training signal. We introduce the Trajectory-level Textual Constraints Translator (TTCT) to replace the manually designed cost function. Our empirical results demonstrate that TTCT effectively comprehends textual constraint and trajectory, and the policies trained by TTCT can achieve a lower violation rate than the standard cost function. Extra studies are conducted to demonstrate that the TTCT has zero-shot transfer capability to adapt to constraint-shift environments.

1 Introduction

In recent years, reinforcement learning (RL) has achieved remarkable success in multiple domains, such as go game [37, 38] and robotic control [21, 45]. However, deploying RL in real-world scenarios still remains challenging. Many real-world decision-making applications, such as autonomous driving [13, 9] require agents to obey certain constraints while achieving the desired goals. To learn a safe constrained policy, some safe RL works [2, 33, 44, 49, 39, 48, 6] have proposed methods to maximize the reward while minimizing the constraint violations after training or during training.

However, several limitations prevent the existing safe RL methods' widespread use in real-world applications. Firstly, these methods often require mathematical or logical definitions of cost functions, which require domain expertise ( Limitation 1 ). Secondly, their cost function definitions are frequently specific to a particular context and cannot be easily generalized to new tasks with similar constraints ( Limitation 2 ). Lastly, most current safe RL methods focus on constraints that are logically simple, typically involving only one single entity or one single state [28], which can't represent the real-world safety requirements and lack universality ( Limitation 3 ).

Using natural language to provide constraints [46, 29, 20] is a promising approach to overcome Limitation 1 and 2 because natural language allows for flexible, high-level expression of constraints that can easily adapt to different scenarios. Regarding Limitation 3 , previous approaches primarily employ what we call the single state/entity textual constraint . The single-state/entity textual constraint focuses solely on constraints related to one specific state or entity, limiting the ability to model complex safety requirements in real-world scenarios. Many safety requirements involve interactions and dependencies among multiple states or entities over time. By only addressing a single state or

entity, these constraints fail to capture the dynamic relationships and temporal aspects that are crucial for ensuring safety in complex environments. So we suggest using a more generalizable constraint type trajectory-level textual constraint . The trajectory-level textual constraint is a more universal constraint with complex logic semantics involving multiple states/entities. It is a nonlinear combination of multiple entities or multiple environment states and can model any constraint requirements in real-world scenarios. The trajectory-level constraints in natural language form are the highly abstract expression of the agent's behavior guidelines, serving as a more natural and straightforward way to introduce constraints. Notably, the set of trajectory-level constraints encompasses single-state constraints, as any single-state constraint can be reformulated as an equivalent trajectory-level constraint. Examples of trajectory-level textual constraint and single state/entity textual constraint are presented in Table 1.

Employing trajectory-level textual constraints across the entire trajectory poses two significant challenges. Firstly, determining whether an RL agent violates textual constraint over a trajectory is non-trivial, it needs to have the perception of the historical states and actions observed along the trajectory ( Challenge 1 ). Secondly, the trajectory-level safety problem is susceptible to sparse cost [34], where cost is only imposed when the agent violates textual constraints at the final time step, making it challenging for the agent to learn which actions contribute to a gradual escalation of risk ( Challenge 2 ). For instance, the agent needs to learn constraints like, "Don't touch lava after you touch grass." Without intermediate feedback, however, it struggles to understand how early actions, such as stepping on grass, contribute to eventual violations.

To address Challenge 1 , we propose a general approach to align the trajectory's factual logic with the text's semantic logic, eliminating the need for manual encoding or separate models for each type of constraint. Our method employs a sequence model [41, 15] for modeling agent historical interactions with the environment, and a pre-trained language model (LM) [7] to comprehend natural language constraints. We then maximize the embedding similarities between matching pairs (trajectory, text) and minimize the embedding similarities between non-matching pairs using a contrastive learning approach [25] similar to CLIP [30]. By calculating the similarity between the textual embeddings of the constraint and the trajectory, we can predict whether a constraint is violated in this trajectory. Our method uniquely leverages text as both a source of constraints and a unified supervisory signal for trajectory encoding. In this dual role, text not only provides constraints but also guides the training process, enabling the model to naturally handle diverse semantic constraints without requiring specific model adjustments for each type. This design allows for a more flexible and generalizable system, significantly simplifying the handling of complex, multi-dimensional constraints.

In addition, to address the issue of cost sparsity ( Challenge 2 ), we introduce a method for temporal credit assignment [40]. The proposed approach involves decomposing the one episodic cost of the textual constraint into multiple parts and allocating them to each state-action pair within the trajectory. This method offers denser cost signals regarding the relationship between the textual constraint and the agent's every action. It informs the agent which behaviors are risky and which are safer, thereby enhancing safety and aiding model performance.

Our experiments demonstrate that the proposed method can effectively address Challenge 1 and Challenge 2 . In both 3D navigation [19] and 2D grid exploration [46] tasks, agents trained using our method achieve significantly lower violation rates (up to 4.0x) compared to agents trained with ground-truth cost functions while maintaining comparable rewards and more importantly, our method obtains the Pareto frontier [10]. In addition to this, our method has zero-shot adaptation capability to adapt to constraint-shift environments without fine-tuning.

2 Related Work

Safe RL. Safe RL aims to train policies that maximize reward while minimizing constraint violations [11, 13]. In prior work, there are usually two ways to learn safe policies: (1) consider cost as one of the optimization objectives to achieve safety [48, 6, 49, 39, 44, 33], and (2) achieve safety by leveraging external knowledge (e.g. expert demonstration) [35, 32, 1, 47]. These typical safe RL algorithms require either human-defined cost functions or human-specified cost constraints which are unavailable in the tasks that constraints are given by natural language.

RL with Natural Language. Prior works have integrated natural language into RL to improve generalization or learning efficiency in various ways. For example, Hermann et al. [14] studied how to train an agent that can follow natural language instructions to reach a specific goal. Additionally, natural language has been used to constrain agents to behave safely. For instance, Prakash et al. [29] trained a constraint checker to predict whether natural language constraints are violated. Yang et al. [46] trained a constraint interpreter to predict which entities in the environment may be relevant to the constraint and used the interpreter to predict costs. Lou et al. [28] used pre-trained language models to predict the cost of specific states, avoiding the need for artificially designed cost functions. However, previous methods cannot uniformly handle textual constraints with one framework, which limits their applicability.

Credit Assignment in RL. Credit assignment studies the problem of inferring the true reward from the designed reward. Prior works have studied improving sample efficiency of RL algorithms through credit assignment, for example by using information gain [17], as an intrinsic bonus reward to aid exploration. Goyal et al. [12] proposed the use of natural language instructions to perform reward shaping to improve the sample efficiency of RL algorithms. Liu et al. [26] learned to decompose the episodic return as the reward for policy optimization. However, to the best of our knowledge, our work is the first to apply credit assignment to safe RL.

3 Preliminaries

Problem formulation. Trajectory-level constraint problem can be formed as the Constrained Non-Markov Decision Process (CNMDP) [3, 43], and it can be defined by the tuple < S, A, T, R, γ, C, Y, τ ∗ >. Here S represents the set of states, A represents the set of actions, T represents the state transition function, R represents the reward function, and γ ∈ ( 0 , 1 ) represents the discount factor. In addition, Y represents the set of trajectory-level textual constraints (e.g., 'You have 10 HP, you will lose 3 HP every time you touch the lava, don't die.'), which describes the constraint that the agent needs to obey across the entire trajectory. C represents the cost function determined by y ∈ Y . τ ∗ represents the set of historical trajectories.

RL with constraints. The objective for the agent is to maximize reward while obeying the specified textual constraint as much as possible [27]. Thus, in our task setting, the agent needs to learn a policy π : S × Y × τ ∗ → P ( A ) which maps from the state space S , textual constraints Y and historical trajectories τ ∗ to the distributions over actions A . Given a y , we learn a policy π that maximizes the cumulative discounted reward J R while keeping the cumulative discounted cost (average violation rate) J C below a constraint violation budget B C ( y ) :

Here B C ( y ) and C ( s t , a t , y, τ t ) are two functions both depending on textual constraint y . τ t represents the historical trajectory at time step t .

Episodic RL. Similar to the task with episodic rewards [4], in our task setting, a cost is only given at the end of each trajectory τ when the agent violates the textual constraint y . In other words, before violating y , the cost C ( s t , a t , y, τ t ) = 0 for all t < T . For simplicity, we omit the discount factor and assume that the trajectory length is at most T so that we can denote a T as the final action that causes the agent to violate y without further confusion. Therefore, the constraint qualification of the objective in RL with constraints becomes J C ( π ) = E π [ C ( s T , a T , y, τ T )] ≤ B C ( y ) . Due to the sparsity of cost, a large amount of rollout trajectories are needed to help the agent distinguish the subtle effects of actions on textual constraint [24]. This situation will become more serious when trajectory-level constraints are complex and difficult to understand.

H11

·L

1

H12

·L

1

·L

1

H14

·L

1

...

H21

·L

2

H22

·L

2

H23

·L

2

...

H31

·L

3

·L

3

H33

·L

3

H34

·L

3

...

HN1

·L

N

HN2

·L

N

HN3

·L

N

HN4

·L

N

...

...

...

...

...

H51

H52

H53

H54

...

...

...

...

...

H31

H33

H34

...

H21

H22

H23

...

H11

H12

H14

...

H1T

H2T

H3T

HNT

...

L1

L2

L3

...

LN

H1T

·L

1H1T

·L

2

H1T

·L

3

...

H1T

·L

N

H2T

·L

1H2T

·L

2

H2T

·L

3

...

H2T

·L

N

H3T

·L

1H3T

·L

2

H3T

·L

3

...

H3T

·L

N

HNT·L1H

NT

·L

2

HNT·L3

...

HNT·LN

...

...

...

...

...

...

s

1

a

1

a

1

s

1

a

T

s

T

...

Trajectory

Trajectory Encoder

After walking through water,

please avoid stepping on fire.

Text

Encoder

...

Ground-truth

MC Loss

Trajectory

τ

Trajectory

τ

H1

H2

H3

H4

...

HT

L

Cost Assignment

HT-1

...

Episodic Cost

Prediction

...

𝑪

(

𝒚

)

SUM

CA Loss

WT Loss

Ground-truth

Text-Trajectory Alignment

Cost Assignment

Use for predicting cost

Text

Constrain y

Text

Encoder

Trajectory

𝜏

Trajectory

Encoder

L

H

𝒔𝒊𝒎

(

𝝉

,

y

)

concat

concat

concat

concat

concat

Trajectory

τ

If

𝒔𝒊𝒎

(

𝝉

,

y

)

<

𝜷

If

𝒔𝒊𝒎

(

𝝉

,

y

)

=1

Cost

Assignment

concat

L

H

𝒄𝑻-𝟏

4 TTCT: Trajectory-level Textual Constraints Translator

In this section, we introduce our proposed framework TTCT ( T rajectory-level T extual C onstraints T ranslator) as shown in Figure 1. TTCT has two key components: the text-trajectory alignment component and the cost assignment component . The text-trajectory alignment component is used to address the violations prediction problem. The cost assignment component is used to address the sparse cost problem.

4.1 Text-Trajectory Alignment Component

We propose a component to learn from offline data to predict whether a given trajectory violates textual constraints. The core idea of this component is to learn trajectory representations under textual supervision and connect trajectory representation to text representation. If the distance between the two representations in the embedding space is sufficiently close, we can consider that the given trajectory violates the constraint. Our approach does not require modeling entities of the environment like previous work, such as [46], which involves labeling hazardous items artificially in every observation. Instead, we model this task as a trajectory-text multimodal learning problem. Hence, our method can learn trajectory representations and text representations from the pairs (trajectory, trajectory-level textual constraint) . We believe that learning from the supervision of natural language could not only enhance the representation power but also enable flexible zero-shot transfer [30].

Formally, given a batch of N (trajectory τ , trajectory-level textual constraint y ) pairs. For each pair, the trajectory corresponds to the text, indicating that the given trajectory violates the given textual constraint. The trajectory can be defined as τ = ( s 1 , a 1 , s 2 , a 2 , ..., s T -1 , a T -1 , s T , a T ) , where T is the step at which the textual constraint y is first violated by the trajectory. Here s t is a d s -dimensional observation vector. Each state in the trajectory is processed by a state encoder to obtain a representation v s t , also action is processed by an action encoder to obtain a representation v a t . Then, we concatenate v s t and v a t to obtain a vector representation v t for each state-action pair. After that, we learn separate unimodal encoders g T and g C for the trajectory and textual constraint, respectively. The trajectory encoder g T utilizes a causal transformer to extract the trajectory representation from the input state-action representation sequence { v t } T t = 1 :

where H t is a d H -dimensional vector. The final embedding H T is used as the representation for the entire trajectory. Specifically, the causal Transformer processes the trajectory sequence by maintaining a left-to-right context while generating embeddings. This allows the model to capture temporal dependencies within the trajectory and obtain the embeddings for time steps before T . The textual constraint encoder g C is used to extract features that are related to the constraints and it could be one of a wide variety of language models:

where L is a d L -dimensional vector representation. Then we define symmetric similarities between the two modalities with cosine distance:

where α is a learnable parameter and T means that use last embedding of trajectory to calculate similarity. And we use softmax to calculate trajectory → text and text → trajectory similarities scores:

Let q τ → y ( τ ) , q y → τ ( y ) indicate the ground-truth similarity scores, where the negative pair (trajectory doesn't violate textual constraint) has a probability of 0 and the positive pair (trajectory violate textual constraint) has a probability of 1. In our task setting, a trajectory can correspond to multiple textual constraints, and vice versa. For example, two textual constraints such as 'Do not touch lava' and 'After stepping on water, do not touch lava' might both be violated by a single given trajectory. This many-to-many relationship between trajectories and textual constraints implies that the same textual constraints (or different textual constraints with the same semantics) can apply to multiple trajectories, while a single trajectory may comply with several textual constraints. So there may be more than one positive pair in q τ → y i ( τ ) and q y → τ i ( y ) . Therefore, we use Kullback-Leibler (KL) divergence [23] as the multimodal contrastive (MC) loss to optimize our encoder similar to [42]:

where D is the training set.

In addition to the multimodal contrastive (MC) loss, we also introduce a within-trajectory (WT) loss. Specifically, suppose we have a trajectory's representation sequence ( H 1 , H 2 , H 3 , ..., H ( T -1 ) , H T ) and its corresponding textual constraint embedding L , we can calculate cosine similarity sim t ( τ, y ) between embedding H t and L using Equation 4. Then we can calculate similarity scores within the trajectory:

Different from p y → τ i ( y ) in Equation 5, which measures similarity scores across N trajectories, Equation 7 is used to measure the similarity scores of different time steps within a trajectory. The reason for doing this is that the textual constraint is violated at time step T , while in the previous time steps the constraint is not violated. Therefore, the instinct is to maximize the similarity score between the final trajectory embedding H T and the textual constraint embedding L , while minimizing the similarity score between all previous time step embeddings ( H 1 , H 2 , H 3 , ..., H ( T -1 ) ) and the textual constraint embedding L . Based on this instinct, we introduce within-trajectory (WT) loss:

where the first term is responsible for minimizing the similarity score for the embedding of time steps before T in the trajectory sequence, while the second term is responsible for maximizing the similarity score for the embedding of time step T .

By combining these two losses L WT , L MC , we can train a text encoder to minimize the distance between embeddings of semantically similar texts, while simultaneously training a trajectory encoder to minimize the distance between embeddings of semantically similar trajectories. Crucially, this approach enables us to align the text and trajectory embeddings that correspond to the same semantic constraint, fostering a cohesive representation of their shared meaning, and further determining whether the trajectory violates the constraint by calculating embedding similarity.

4.2 Cost Assignment Component

After solving the problem of predicting whether a given trajectory violates a textual constraint, there is still the issue of cost sparsity. Motivated by the works of temporal credit assignment [26], we propose

a component to capture the relationship between the state-action pair and the textual constraint and assign a cost value to each state-action based on its impact on satisfying the constraint.

Specifically, suppose we have a (trajectory τ , textual constraint y ) pair and its representation ( { H t } T t = 1 , L ) obtained from text-trajectory alignment component. The textual constraint representation L is processed by an episodic-cost prediction layer F e to obtain a predicted episodic cost ˆ C ( y ) = sigmoid ( F e ( L )) for the entire trajectory. We expect the episodic cost can be considered as the sum of cost on all non-violation state-action pairs: ˆ C ( y ) = ∑ T -1 t = 1 ˆ c ( s t , a t , y, τ t ) . To evaluate the significance of each timestep's action relative to textual constraints, we employ an attention mechanism:

Here we regard the text representation as the query , each time step's representation in the trajectory as the key , and compute the attention score e t based on the cosine similarity metric. After that, we use the sigmoid function to make sure the score falls within the range of 0 to 1. Each attention score e t quantifies the degree of influence of the state-action pair ( s t , a t ) on violating the textual constraint. Then we obtain an influence-based representation H ∗ t = e t H t . To predict the cost ˆ c ( s t , a t , y, τ t ) , we incorporate a feed-forward layer called the cost assignment layer F c and output the predicted non-violation single step cost as:

The loss function for measuring inconsistency of the episodic cost ˆ C ( y ) and the sum of non-violation single step costs ˆ c ( s t , a t , y, h t ) can be formed as:

This mutual prediction loss function L CA is only used to update the episodic-cost prediction layer and cost assignment layer, and not to update the parameters of the trajectory encoder or text encoder during backpropagation. This helps ensure the validity of the predictions by preventing overfitting or interference from other parts of the model.

The effectiveness of this component comes from two main sources. First, the text-trajectory alignment component projects semantically similar text representations to nearby points in the embedding space, allowing the episodic-cost prediction layer to assign similar values to embeddings with close distances. This aligns with the intuition that textual constraints with comparable violation difficulty should yield similar episodic costs. Second, the cost assignment layer leverages the representational power of the text-trajectory alignment component to capture complex relationships between state-action pairs and constraints, enabling accurate single-step cost predictions.

In our experiment, we simultaneously train the text-trajectory alignment component and cost assignment component end-to-end with a uniform loss function L TTCT :

By doing this, we can avoid the need for separate pre-training or fine-tuning steps, which can be timeconsuming and require additional hyperparameter tuning. Also, this can enable the cost assignment component to gradually learn from the text-trajectory alignment component and make more accurate predictions over time.

In the test phase, at time step t we encode trajectory τ t and textual constraint y with Equation 3 and Equation 2 to obtain the entire trajectory embedding H t and text embedding L . Then we calculate distance score sim ( τ t , y ) using Equation 4. The predicted cost function ˆ c is given by:

where β is a hyperparameter that defines the threshold of the cost prediction. ˆ c = 1 indicates that the TTCT model predicts that the textual constraint is violated. Otherwise, if the textual constraint is not violated by the given trajectory, we assign a predicted cost using Equation 10.

5 Policy Training

Our Trajectory-level textual constraints Translator framework is a general method for integrating free-form natural language into safe RL algorithms. In this section, we introduce how to integrate our TTCT into safe RL algorithms so that the agents can maximize rewards while avoiding early termination of the environment due to violation of textual constraints. To enable perception of historical trajectory, the trajectory encoder and text encoder are not only used as frozen plugins g T and g C for cost prediction but also as trainable sequence models g ∗ T and g ∗ C for modeling historical trajectory. This allows the agent to take into account historical context when making decisions. To further improve the ability to capture relevant information from the environment, we use LoRA [18] to fine-tune both the g ∗ T and g ∗ C during policy training. The usage of g T , g C and g ∗ T , g ∗ C is illustrated in Appendix A.4 Figure 8.

Formally, let's assume we have a policy π with parameter ϕ to gather transitions from environments. We maintain a vector to record the history state-action pairs sequence, and at time step t we use g ∗ T and g ∗ C to encode τ t -1 and textual constraint y so that we can get historical context representation H t -1 and textual constraint representation L . The policy selects an action a t = π θ ( o t , H t -1 , L ) to interact with environment to get a new observation o t + 1 . And we update τ t with the new state-action pair ( o t , a t ) to get τ t . With τ t and L , ˆ c t can be predicted according to Equation 13. Then we store the transition into the buffer and keep interacting until the buffer is full. In the policy updating phase, after calculating the specific loss function for different safe RL algorithms, we update the policy π with gradient descent and update g ∗ T , g ∗ C with LoRA. It is worth noting that g T and g C are not updated during the whole policy training phase, as they are only used for cost prediction. The pseudo-code and more details of the policy training can be found in Appendix A.4.

6 Experiments

Our experiments aim to answer the following questions: (1) Can our TTCT accurately recognize whether an agent violates the trajectory-level textual constraints? (2) Does the policy network, trained with predicted cost from TTCT, achieve fewer constraint violations than trained with the ground-truth cost function? (3) How much performance improvement can the cost assignment (CA) component achieve? (4) Does our TTCT have zero-shot capability to be directly applicable to constraint-shift environments without any fine-tuning? We adopt the following experiment setting to address these questions.

(b) SafetyGoal

6.1 Setup

Task. We evaluate TTCT on two tasks (Figure 2 (a,b)): 2D grid exploration task Hazard-WorldGrid (Grid) [46] and 3D robot navigation task SafetyGoal (Goal) [19]. And we designed over 200 trajectory-level textual constraints which can be grouped into 4 categories, to constrain the agents. A detailed description of the categories of constraints will be given in Appendix A.1. Different from the default setting, in our task setting, when a trajectory-level textual constraint is violated, the

environment is immediately terminated. This is a more difficult setup than the default. In this setup, the agents must collect as many rewards as possible while staying alive.

Baselines. We consider the following baselines: PPO [36], PPO _ Lagrangian ( PPO _ Lag ) [33], CPPO _ PID [39], FOCOPS [49]. PPO does not consider constraints and simply aims to maximize the average reward. We use PPO to compare the ability of our methods to obtain rewards. As for the last three algorithms, we design two training modes for them. One is trained with standard ground-truth cost, where the cost is given by the human-designed violation checking functions, and we call it ground-truth (GC) mode. The other is trained with the predicted cost by our proposed TTCT, which we refer to as cost prediction (CP) mode. More information about the baselines and training modes can be found in Appendix A.1 A.2.

Metrics. We take average episodic reward (Avg. R) and average episodic cost (Avg. C) as the main comparison metrics. Average episodic cost can also be considered as the average probability of violating the constraints. The higher Avg. R, the better performance, and the lower Avg. C, the better performance.















































































(a) Hazard-World-Grid

  









































































(b) SafetyGoal

6.2 Main Results and Analysis

The evaluation results are shown in Figure 3 and the learning curves are shown in Figure 4. We can observe that in the Hazard-World-Grid task, compared with PPO , the policies trained with GC can reduce the probability of violating textual constraints to some extent, but not significantly. This is because the sparsity of the cost makes it difficult for an agent to learn the relevance of the behavior to the textual constraints, further making it difficult to find risk-avoiding paths of action. In the more difficult SafetyGoal task, it is even more challenging for GC-trained agents to learn how to avoid violations. In the CPPO _ PID and FOCOPS algorithms trained with GC mode, the probability of violations even rises gradually as the training progresses. In contrast, the agents trained with predicted cost can achieve lower violation probabilities than GC-trained agents across all algorithms and tasks and get rewards close to GC-trained agents.

These results show that TTCT can give an accurate predicted episodic cost at the time step when the constraint is violated, it can also give timely cost feedback to non-violation actions through the cost assignment component so that the agents can find more risk-averse action paths. And these results answer questions (1) and (2) . The discussion about the violation prediction capability of the text-trajectory component can be found in Appendix B.1. The interpretability and case study of the cost assignment component can be found in Appendix B.2.

6.3 Ablation Study

To study the influence of the cost assignment component. We conduct an ablation study by removing the cost assignment component from the full TTCT. The results of the ablation study experiment are shown in Figure 4. We can observe that even TTCT without cost assignment can achieve similar performance as GC mode. And in most of the results if we remove the cost assignment component, the performance drops. This shows that our text trajectory alignment component can accurately predict the ground truth cost, and the use of the cost assignment component can further help us learn a safer agent . These results answer questions (3) .

PPO

{algo} with GC

{algo} with CP (our)

{algo}with CP

GLYPH(cmap:d835)/GLYPH(cmap:d835)

CA

PPO_Lag

CPPO_PID

FOCOPS

CPPO_PID

FOCOPS

Avg. C

(a) Hazard-World-Grid

(b) SafetyGoal

PPO_Lag

3.0

3.0

3.0

Z

2.5

2.0

2.5

2.0

0

D

0

1.5

1.5

-1

[-

-1

0.5

0

20

40

60

80

100

1.0

0.5

0

20

40

60

80

100

1.0

0.5

0

M

20

40

60

80

100

1

0.6

-3

0

50

100

150

250

00E

0.6

3

-2

0

50

100

150

200

OOE

).6

-2

-3

0

50

100

150

200

300

0.8

0.6

0.8

0.6

0.3

0.5

0.4

0.5

0.4

0.3

0.3

0.5

0.4

0.4

0.2

0

20

40

60

80

100

0.4

0.2

0

20

40

60

08

100

0.4

0.2

0

20

40

upoda

60

80

100

0.2

0.1

0.0

0

0S

100

150

200

250

300

0.2

0.1

0.0

0

50

100

150

200

250

300

0.2

0.1

0°0

0

50

100

150

200

250

300















































































(a) Hazard-World-Grid

  













































































(b) SafetyGoal

6.4 Further Results

Pareto frontier. Multi-objective optimization typically involves finding the best trade-offs between multiple objectives. In this context, it is important to evaluate the performance of different methods based on their Pareto frontier [10], which represents the set of optimal trade-offs between the reward and cost objectives. We plot the Pareto frontier of policies trained with GC and policies trained with CP on a two-dimensional graph, with the vertical axis representing the reward objective and the horizontal axis representing the cost objective as presented in Figure 6. The solution that has the Pareto frontier closer to the origin is generally considered more effective than those that have the Pareto frontier farther from the origin. We can observe from the figure that the policies trained with predicted cost by our TTCT have a Pareto frontier closer to the origin. This proves the effectiveness of our method and further answers Questions (1) and (2) .

(a)

PPO

_

Lag

(b)

CPPO

_

PID

(c)

FOCOPS

0.5

0.5

0.5

1.0

algo

Rewar

1.5

CP(our)

GC

Reward

1.0

algo

1.0

algo

2.0

CP(our)

GC

1.5

CP(our)

GC

1.5

GC

CP(our)

2.5

★

CP(our)

★

CP(our)

★

2.0

GC

2.0

GC

3.0

0.2

0.4

0.6

0.8

1.0

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

1.0

Cost

Cost

Cost

Zero-shot transfer capability. To explore whether our method has zero-shot transfer capability, we use the TTCT trained under the Hazard-World-Grid environment to apply directly to a new environment called LavaWall (Figure 2 (c)) [5], without fine-tuning. The results are shown in Figure 7. We can observe that the policy trained with cost prediction (CP) from TTCT trained under the Hazard-World-Grid environment can still achieve a low violation rate comparable to the GC-trained policy. This answers Question (4) .

0

20

40

60

80

100

1

2

3

4

5

Reward

algo

CP(our)

GC

(a) Zero-shot reward

0

20

40

60

80

100

0.0

0.2

0.4

0.6

Cost

algo

CP(our)

GC

(b) Zero-shot cost

7 Conclusion and Future Work

In this paper, we study the problem of safe RL with trajectory-level natural language constraints and propose a method of trajectory-level textual constraints translator (TTCT) to translate constraints into a cost function. By combining the text-trajectory alignment (CA) component and the cost assignment (CA) component, our method can elegantly solve the problems of predicting constraint violations and cost sparsity. We demonstrated that our TTCT method achieves a lower violation probability compared to the standard cost function. Thanks to its powerful multimodal representation capabilities, our method also has zero-shot transfer capability to help the agent safely explore the constraint-shift environment. This work opens up new possibilities for training agents in safe RL tasks with total free-form and complex textual constraints.

Our work still has room for improvement. The violation rate of our method is not absolute zero. In future work, we plan to investigate the application of TTCT in more complex environments and explore the integration of other techniques such as meta-learning [16] to further improve the performance and generalization capabilities of our method.

Acknowledgments

This work was supported by the grants from the Natural Science Foundation of China (62225202, 62202029), and Young Elite Scientists Sponsorship Program by CAST (No. 2023QNRC001). Thanks for the computing infrastructure provided by Beijing Advanced Innovation Center for Big Data and Brain Computing. This work was also sponsored by CAAI-Huawei MindSpore Open Fund. Jianxin Li is the corresponding author.

References

Pieter Abbeel, Adam Coates, and Andrew Y Ng. Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research , 29(13):1608-1639, 2010.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

Eitan Altman. Constrained Markov decision processes . Routledge, 2021.

Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint arXiv:1606.04460 , 2016.

Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. Advances in Neural Information Processing Systems , 36, 2024.

Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv:1901.10031 , 2019.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Tom Fawcett. An introduction to roc analysis. Pattern recognition letters , 27(8):861-874, 2006.

Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X Liu. Dense reinforcement learning for safety validation of autonomous vehicles. Nature , 615(7953):620-627, 2023.

Drew Fudenberg and Jean Tirole. Game theory . MIT press, 1991.

Javier Garcıa and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480, 2015.

Prasoon Goyal, Scott Niekum, and Raymond J Mooney. Using natural language for reward shaping in reinforcement learning. arXiv preprint arXiv:1903.02020 , 2019.

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330 , 2022.

Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning in a simulated 3d world. arXiv preprint arXiv:1706.06551 , 2017.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence , 44(9):5149-5169, 2021.

Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems , 29, 2016.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.

Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems , 36, 2023.

Russell Kaplan, Christopher Sauer, and Alexander Sosa. Beating atari with natural language guided reinforcement learning. arXiv preprint arXiv:1704.05539 , 2017.

Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature , 620(7976):982-987, 2023.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics , 22(1):79-86, 1951.

Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion , 85:1-22, 2022.

Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. Ieee Access , 8:193907-193934, 2020.

Yang Liu, Yunan Luo, Yuanyi Zhong, Xi Chen, Qiang Liu, and Jian Peng. Sequence modeling of temporal credit assignment for episodic reinforcement learning. arXiv preprint arXiv:1905.13420 , 2019.

Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints in model-free reinforcement learning: A survey. In The 30th international joint conference on artificial intelligence (ijcai) , 2021.

Xingzhou Lou, Junge Zhang, Ziyan Wang, Kaiqi Huang, and Yali Du. Safe reinforcement learning with free-form natural language constraints and pre-trained language models. arXiv preprint arXiv:2401.07553 , 2024.

Bharat Prakash, Nicholas R Waytowich, Ashwinkumar Ganesan, Tim Oates, and Tinoosh Mohsenin. Guiding safe reinforcement learning policies using structured language constraints. In SafeAI@ AAAI , pages 153-161, 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087 , 2017.

Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7(1):2, 2019.

Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International conference on machine learning , pages 4344-4353. PMLR, 2018.

Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning , pages 9133-9143. PMLR, 2020.

Richard Stuart Sutton. Temporal credit assignment in reinforcement learning . University of Massachusetts Amherst, 1984.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472 , 2021.

Steven D Whitehead and Long-Ji Lin. Reinforcement learning of non-markov decision processes. Artificial intelligence , 73(1-2):271-306, 1995.

Long Yang, Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. Advances in Neural Information Processing Systems , 35:9111-9124, 2022.

Tao Yang, Youhua Xiao, Zhen Zhang, Yiming Liang, Guorui Li, Mingqi Zhang, Shijian Li, Tuck-Whye Wong, Yong Wang, Tiefeng Li, et al. A soft artificial muscle driven robot with reinforcement learning. Scientific reports , 8(1):14518, 2018.

Tsung-Yen Yang, Michael Y Hu, Yinlam Chow, Peter J Ramadge, and Karthik Narasimhan. Safe reinforcement learning with natural language constraints. Advances in Neural Information Processing Systems , 34:13794-13808, 2021.

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Accelerating safe reinforcement learning with constraint-mismatched policies. arXiv preprint arXiv:2006.11645 , 2020.

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152 , 2020.

Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems , 33:15338-15349, 2020.

A Dataset and Training Details

A.1 Dataset

In our task setting, humans need to provide high-level textual instruction to the agent for the entire trajectory, and then TTCT can predict the cost based on the real-time state of the agent's exploration so that the agent can learn a safe policy with the predicted cost. Thus our dataset is comprised of two parts: the trajectory-level textual constraints and the environments.

Trajectory-level textual constraints: To generate textual constraints, we first explore the environment using a random policy, collecting a large amount of offline trajectory data. Then we design a descriptor that automatically analyzes trajectories and gives natural language descriptions based on predefined templates. To validate whether our method can understand different difficulty levels of textual constraints, we design four types of trajectory-level textual constraints. The four types of textual constraints are:

Quantitative textual constraint describes a quantitative relationship in which an entity in the environment cannot be touched beyond a specific number of times, which can be interpreted as the entity's tolerance threshold, and when the threshold is exceeded, the entity may experience irrecoverable damage.

Sequential textual constraint describes a sequence-based relationship, where the occurrence of two or more distinct actions independently may not pose a risk, but when they occur in sequence, it does. For instance, it's safe to drink or drive when they happen independently, but when they occur in sequence (i.e., drinking first and then driving), it becomes dangerous.

Relational textual constraint describes constraints on the relationships between an agent and entities in its environment, such as maintaining a certain distance, always being in front of that entity, or not staying too far from it.

Mathematical textual constraints often do not provide explicit instructions to the agent regarding what actions to avoid, but rather present logical descriptions that demand the model's ability to reason mathematically. This type of constraint thereby presents a higher cognitive burden for our TTCT to comprehend.

Examples of four types of trajectory-level textual constraints are included in Table 2. Then we randomly split the (trajectory, textual constraint) pairs into 80% training and 20% test sets. And we use the training set to train our TTCT end-to-end.

Environments: Weuse two environments Hazard-World-Grid and SafetyGoal as main benchmarks and a environment LavaWall to evaluate the zero-shot transfer capability:

Hazard-World-Grid. The environment is a 12 × 12 grid, with the gray walls surrounding the perimeter. The agent can only explore within the grid. Inside the grid, some items provide rewards: blue keys, red balls, and yellow boxes. Collecting all of these items will be considered as completing the task. Additionally, there are hazardous materials in the grid, where orange tiles are lava, cyan tiles are water, and green tiles are grass. During exploration, the agent can only see a range of 7 × 7 pixels ahead, resulting in an observation space with size 7 × 7 × 3 .

SafetyGoal. Robot navigation task, the environment has a navigation target, which is completed when the robot reaches the target. The environment also contains vases (cyan squares) and Hazards (Mediumslateblue circles). The vases can be moved by the robot and hazards are fixed.

LavaWall. The LavaWall environment shares the same task goal as Hazard-World-Grid but with different hazardous materials. Whenever the environment is initialized, a random lava wall with only one entrance is generated in the grid, and the agent must learn to obtain the reward on the other side while avoiding stepping on lava.

For each episode, we place the agent at a randomized start location, fill the environment with objects, and randomly select a textual constraint from the constraint pool. We manually design the violation checking function for every constraint to determine whether the environment violates the textual constraint, and when the textual constraint is violated, the environment will provide a cost with a

value of 1.0 to the agent and terminate the environment (the manually designed function will not be used in the cost prediction (CP) mode, all the costs will be predicted by our TTCT in CP mode).

A.2 Baselines

(1) PPO [36]: This algorithm does not consider constraints, and simply considers maximizing the average reward, which we use to compare the ability of our methods to obtain rewards.

PPO _ Lagrangian ( PPO _ Lag ) [33]: This algorithm transforms a constrained optimization problem into an unconstrained optimization problem via Lagrange multipliers

(3) CPPO _ PID [39]: This algorithm PID to control Lagrange multiplier, which solves the cycle fluctuation problem of PPO-Lagrange.

(4) FOCOPS [49]: This algorithm finds the optimal update policy by solving a constrained optimization problem in the nonparameterized policy space, then projects the update policy back into the parametric policy space.

A.3 Training Detail of TTCT

We provide training pseudocode of TTCT in Figure 9. The text encoder we use is Bert ( (https: //huggingface.co/google-bert/bert-base-uncased/tree/main ) [7]. And the hyperparameters we use are shown in Table 3. We use the same hyperparameters across different tasks' datasets. We conduct the training on the machine with A100 GPU for about 1 day.

A.4 Policy Training Detail

We provide pseudocode of training policy with predicted from TTCT in Algorithm 1. To enable the agent to comply with diverse types of constraints, we launch async vectorized environments with different types of constrained environments during roll-out collection. The agent interacts with these environments and collects transitions under different constraint restrictions to update its policy. During policy updates, we fine-tune the trajectory encoder g ∗ H and text encoder g ∗ L using LoRA [18] at every epoch's first iteration, while not updating encoder parameters in other iterations, which saves training time.

Algorithm 1 Safe RL algorithm with TTCT

1: Initialize value function network ϕ , cost value function network ϕ c , policy network θ , trajectory encoder g ∗ T , textual constraint encoder g ∗ C , frozen trajectory encoder g T for cost prediction, frozen textual constraint encoder g C for cost prediction, cost alignment network F c

2: for each epoch do

3: for each episode do

4: Sample a textual constraint y for this episode

5: Get text representation L of y with g ∗ C

6: Reset environment to get observation o 1

8: for t = 1 , T do 9: Select a action a t = π θ ( o t , H t 1 , L )

7: Initialize sequence τ 0 = { o pad , a pad } and encoded sequence representation H 0 = g ∗ T ( τ 1 )

-10: Execute action a t in emulator and observe reward r t and next observation o t 1

12: Predict cost ˆ c t according to Equation 13

11: Append ( o t , a t ) into τ t -1 to get τ t and update sequence representation H t = g ∗ T ( τ t )

13: Store transition ( o t , a t , r t , ˆ c t , τ t -1 , y ) in buffer

14: end for

15: end for

16: Sample batch of N transitions from buffer.

17: Encode { τ j } N i = 1 and { y i } N i = 1 with g T , g C 18: Calculate specific loss function

19: Update value function ϕ , cost value function ϕ c , policy network θ

21: end for

20: Update trajectory encoder g ∗ T and textual constraint encoder g ∗ C with LoRA

TTCT training

dataset

pretrained

trajectory encoder

pretrained

text encoder

policy training

g T *

g C *

g T

g C

Actor

Environment

observation

cost predictor

action

cost

Frozen parameters

Trainable parameters

B Additional Experiments

B.1 Violations Prediction Capability of Text-Trajectory Alignment Component

9.2

0.022

0.026

0.27

0.65

0.16

0.27

-0.021

-0.41

0.026

0.096

7.6

0.065

0.27

-0.22

-0.013

0.27

0.042

0.23

0.065

0.24

-0.22

8.8

0.14

-0.19

0.065

0.14

0.26

-0.22

8.8

-0.053

-0.43

0.1

8.2

0.087

-0.065

8.2

-0.52

0.21

0.1

0.73

0.051

0.21

-0.17

7.6

0.018

-0.17

0.63

-0.17

0.21

-0.34

-0.7

-0.054

0.034

-0.071

8.6

0.034

-0.043

0.037

-0.054

0.21

-0.02

0.3

7.8

0.67

-0.27

7.8

-0.42

0.2

0.3

-0.19

-0.21

0.18

-0.53

-0.25

-0.32

-0.53

8

0.58

0.18

-0.33

0.035

-0.34

-0.31

-0.024

-0.45

-0.31

0.071

9.5

-0.34

0.18

-0.068

7.7

0.57

0.67

-0.25

0.57

0.32

0.067

7.7

0

2

4

6

8

1

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

1

0

0

0

1

0

0

1

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

1

0

0

1

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

1

0

0

0

1

0

0

0

0

0

0

1

0.0

0.2

0.4

0.6

0.8

1.0

(a) Cosine similarity (after scale)

(b) Ground-truth similarity

To further study the ability of our text-trajectory alignment component to predict violations, we conduct an experiment given a batch of trajectory-text pairs and we use the text-trajectory alignment component to encode the trajectory and textual constraint, and then calculate the cosine distance with Equation 5 between every two embeddings across two modal. We plot a sample of heatmap of calculated cosine similarity and ground-truth as presented in 10. Further, We plot the receiver operating characteristic (ROC) [8] curve to evaluate the performance of the text-trajectory alignment component as presented in Figure 11. AUC (Area Under the Curve) [8] values indicate the area under the ROC curve. The AUC value of our violations prediction result is 0.98. Then We set threshold β equal to the best cutoff value of the ROC curve. We determine whether the trajectory violates a given textual constraint by:

The results are shown in Table 4. These results indicate that our text-trajectory alignment component can accurately predict whether a given trajectory violates a textual constraint.

B.2 Case Study of Cost Assignment (CA) Component

We visualize the assigned cost for every state-action pair to demonstrate that the cost assignment component could capture the subtle relation between the state-action pair and textual constraint. Our intuition is that we should assign larger costs to state-action pairs that lead to violations of trajectory-level textual constraints, and smaller or negative values to pairs that do not contribute to constraint violations. Using the Hazard-World-Grid environment as an example, we choose three different types of constraints to show our results in Figure 12. The first row shows the textual constraint, the second row shows the trajectory of the agent in the environment, and to make it easier to visualize, we simplify the observation s t by representing it as a square, denoting the entity stepped on by the agent at time step t . The square emphasized by the red line indicates the final entity that makes the agent violate textual constraint at time step T . The third row shows the predicted cost of the agent at every time step t and deeper colors indicate larger cost values.

The (a) constraint is mathematical textual constraint: 'You only have 20 HP. Lava and grass are dangerous, they will make you lose 3 and 2 HP, respectively. However, water can regenerate 1 HP. Please don't die. '. This constraint describes the two dangerous entities lava and grass, and the beneficial entity water. From the third-row heat map, we can observe that our cost assignment

0.0

0.2

0.4

0.6

0.8

1.0

False Positive Rate

0.0

0.2

0.4

0.6

0.8

1.0

True Positive Rate

Threshold:7.09

Receiver Operating Characteristic

ROC curve (area = 0.98)

Mathematical Text Constraint:

'You only have 20 HP. Lava and grass are dangerous, they will make you

lose 3 and 2 HP, respectively. However, water can regenerate 1 HP. Please don't die.'

Quantitative Text Constraint:

'Lava is dangerous. Don't touch them more than eight times!'

Sequential Text Constraint:

'After you touch lava, don't step on grass!'

(a)

(b)

(c)

= grass

= water

= floor

0.01

0

11

12

1314

516171819202122

0.00

0.03

0.02

0.01

6

10

11

丝M系

0.03

0.02

0.01

10

11

1.2

18

19

0.00

component assigns a high cost to the action that steps on lava or grass, with the cost increasing as the agent approaches the constraint-violating situation. Not only that, the CA component also recognizes the different levels of danger posed by lava and grass. Since stepping on lava will deduct 3 HP while stepping on grass will deduct 2 HP, the CA component assigns a larger cost value at time step 13 -14 compared to the cost value at time step 3 -5 .

The (b) constraint is quantitative textual constraint : 'Lava is dangerous. Don't touch them more than eight times!'. When stepping on the floor, the CA component considers these actions to be safe and assigns a cost of nearly 0. However, when the agent steps onto lava, it assigns a higher cost, especially when the agent steps on lava for the eighth time. At this point, our CA component concludes that the situation has become extremely dangerous, and one more step on lava will violate the constraint, thus giving the highest cost compared to the before time steps.

The (c) constraint is sequential textual constraint : 'After you touch lava, don't step on grass!'. Our CA component captures two relevant entities: lava and grass, and understands the sequential relationship between entities. When the agent first steps onto the grass, the text-trajectory alignment component determines that this action does not violate the textual constraint. However, after the

agent steps onto lava and then steps onto the grass for the second time, the text-trajectory alignment component detects the cosine similarity sim ( τ, y ) greater than the threshold β , thereby violating the textual constraint. And CA component captures that the key trigger condition for violating constraint is stepping onto the lava, therefore assigning a relatively larger cost to such actions at time step 7 -9 . The cost assignment component also assigns relatively small costs to some safe actions that are stepping on the floor, such as steps 13 -15 . This is because it detects that the agent, although hasn't stepped on grass yet, is trending towards approaching grass, which is a hazardous trend, thus providing a series of gradually increasing and small costs. This demonstrates that our component not only monitors key actions that lead to constraint violations but also monitors the hazardous trend and nudging the agent to choose relatively safer paths.

B.3 Results for Different Types of Constraints

To evaluate the agent's understanding of different types of trajectory-level textual constraints, we conducted an additional experiment using the CPPO _ PID algorithm. During training, we separately tracked the average episodic reward and the average episodic cost for three types of textual constraints as presented in Figure 13. From the learning curves, we can observe that for every type of constraint, our CP mode can achieve the lowest violation rate compared to CP without the CA component mode and ground-truth cost (GC) mode.

0

20

40

60

80

100

0.5

1.0

1.5

2.0

2.5

Reward

algo

CP(our)

CP w/o CA

GC

0

20

40

60

80

100

0.2

0.4

0.6

0.8

Cost

algo

CP(our)

CP w/o CA

GC

(a) Quantitative constraints

0

20

40

60

80

100

0.5

1.0

1.5

2.0

Reward

algo

CP(our)

CP w/o CA

GC

0

20

40

60

80

100

0.2

0.4

0.6

0.8

Cost

algo

CP(our)

CP w/o CA

GC

(b) Sequential constraints

0

20

40

60

80

100

1

2

3

Reward

algo

CP(our)

CP w/o CA

GC

0

20

40

60

80

100

0.0

0.2

0.4

0.6

0.8

Cost

algo

CP(our)

CP w/o CA

GC

(c) Mathematical constraints

B.4 Inference time

We perform the trajectory length sensitivity analysis on Hazard-World-Grid. Since our framework is mainly used for reinforcement learning policy training where data is typically provided as input in batches, we counted the inference time for different trajectory lengths with 64 as the batch size, using the hardware device V100-32G. Figure 14 shows that the average inference time per trajectory is 10ms for trajectories of length 100.

100

200

300

400

Trajectory Length

0.7

0.8

0.9

1.0

1.1

1.2

1.3

1.4

Inference Time (second)

B.5 Different Text Encoder

0

20

40

60

80

100

0.5

1.0

1.5

2.0

2.5

3.0

Reward

TextEncoder

Transformer

gpt2

(a) Average episodic reward

0

20

40

60

80

100

0.2

0.4

0.6

0.8

Cost

TextEncoder

Transformer

gpt2

(b) Average episodic cost

B.6 Additional Results

We present the learning curve of our main experiment in Figure 4. We also present the main results and ablation study in Table 5 and 6.

↓

±

±

±

±

±

±

±

↓

±

±

±

C Broader Impacts and Limitation

Our method can help train agents in reinforcement learning tasks with total free-form natural language constraints, which can be useful in various real-world applications such as autonomous driving, robotics, and game playing. There are still limitations to our work. Our method may not be able to completely eliminate constraint violations. Our method has the contextual bottleneck as the length of the trajectory increases. We performed a trajectory length sensitivity analysis on Hazard-World-Grid. As shown in Figure 16, initially increasing the trajectory length improves performance because longer trajectories may provide more dependencies. However, beyond a certain point, further increases in trajectory length result in a slight drop in AUC. This decline is because the trajectory encoder has difficulty capturing global information. We consider this bottleneck to be related to the transformer's encoding capability.

50

100

150

200

250

300

350

400

Trajectory Length

0.800

0.825

0.850

0.875

0.900

0.925

0.950

0.975

1.000

AUC

±

±

±

±

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We show our approach can predict cost with trajectory-level natural language. The predicted cost by our approach can help train a safer policy than the ground-truth cost. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 7

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We show the used parameters and the pseudocode of TTCT in Appendix A.3. Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code to reproduce our results is available in the anonymous repository https: //github.com/formember/TTCT .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 6 and Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See the Appendix A.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All the links, data, code are provided in the anonymous form.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: They are open-sourced.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a type of universal natural language constraint to model diverse real-world safety requirements. Also, to avoid using the specific human-designed cost function, this paper introduces a Unified Trajectory-Level Textual Constraints Translator (U3T) for aligning text with trajectories and assigning costs via attention mechanisms. By conducting experiments on two environments, this paper reaches the conclusion that U3T can accurately predict whether a given trajectory violates the natural language constraint, and the policy trained with predicted cost can behave more safely.\n\nSTRENGTHS:\n1. Broad applicability: U3T introduces trajectory-level textual constraints, capable of modeling diverse constraint requirements in real-world scenarios, making it more widely applicable across different types of constraints and complex environments compared to traditional methods.\n2. Novelty: Addressing the natural language constrained reinforcement learning challenge through a novel approach involving multimodal alignment and novelly integrating a credit assignment component within the framework.\n3. Automated constraint handling: Through its text-trajectory alignment and attention-based cost assignment components, U3T automates constraint handling and cost allocation, reducing the need for manual design of cost functions and enhancing system flexibility and generality.\n4. Empirical results demonstrate that policies trained with U3T achieve lower violation rates compared to standard cost functions.\n\nWEAKNESSES:\n1. Complexity and computational cost: Implementing U3T involves complex multimodal learning and attention mechanisms, which may require significant computational resources and time costs, especially when dealing with large-scale datasets or real-time applications.\n2. The authors do not specify exactly what model of text encoder was used in the experiments.\n3. Data Dependency: Effective implementation of U3T heavily relies on high-quality and well-annotated textual and trajectory data, which may be challenging to acquire and curate in some practical applications.\n\nQUESTIONS:\n1. Is it feasible for the authors to experimentally analyze the inference speed of the proposed framework?\n2. Could the authors specify the model of the text encoder used in the experiments and provide an in-depth exploration of its selection rationale?\n3. Leveraging large language models for text understanding shows promise. Have the authors considered strategies for integrating these models to augment the comprehension and deconstruction of intricate textual constraints?\n\nLIMITATIONS:\nThe authors have discussed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* The paper proposes U3T, a new system for more robust safe RL under general text constraints.\n* The key innovation of the paper is to generalize text constraints such that the constraints don’t refer to a specific entity/state and addressing cost sparsity where constraints are only violated at the final time step.\n* The proposed system consists of a trajectory encoder and text encoder that are trained jointly using a contrastive loss defined by the KL-divergence.\n* The authors solve cost sparsity by identifying actions/states that contribute to the violation of a constraint in an attention mechanism.\n\nSTRENGTHS:\n* The paper solves an interesting and intuitive problem. There is a clear application of generalizing text constraints from single entity to general entities. \n* The formulation is intuitive and well-explained. The components introduced build on a simple and effective design\n* Ablation studies seem to demonstrate the benefit of the cost assignment portion of their system design, a portion that could be extremely expensive cost-wise.\n* The general cost (Figure 2/Figure 3) of their method seems to reduce in comparison to their baselines that use ground truth.\n\nWEAKNESSES:\n* I found some details missing in the text. I discuss some portions that could use clarification here.\n    * Which text encoder was used in this experiment? I couldn’t find this in the paper and from your appendix section A.4, it seems untrained. You refer to finetuning with LORA though so I was confused. A small nit, could be interesting to connect the text encoder performance to results. I realize space is tight but if these could be incorporated into the main paper, that would clarify and make the experiments easier to interpret.\n    * I’m unsure how to interpret the average reward results in comparison to the cost results. See questions. My intuition was that rewards should be larger given the pareto frontier results. \n* I’m having trouble differentiating this paper from the Lou et. al. paper. I took a quick look at Lou et. al. --  I can understand that entities needed to be modeled in the paper but from what I can tell, that paper could also be applied in the settings you considered given the use of the pretrained LLM. From Figure 1, I couldn’t see the specific entity problem you were mentioning in the paper. Why wasn’t this added as a baseline? Could you provide more discussion on why these two are different? I think the approach in this paper has considerable novelty so I won’t say that this is entirely negative but it does diminish the contribution slightly.\n\nQUESTIONS:\n* I noticed that generally, while the cost of the method would decrease, the reward wouldn’t change significantly (Figure 2). I actually found this a bit unintuitive. I figured that lower cost would imply a longer trajectory and therefore, longer windows to collect reward over, so average reward could increase. I found this discussion missing in the text. Could this be elaborated?\n* How long were the trajectories used in the paper? (What was the average size of T?) Could there be a context bottleneck?\n\nLIMITATIONS:\n* See question about context bottleneck. Could this be a limitation to discuss?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This work proposes an approach to training RL agents with constraints. The proposed approach learns language embeddings of constraints and embeddings of trajectories. During training, a similarity score is used to align the space of constraint embeddings and the space of trajectory embeddings.\n- Moreover, the language embedding of the constraint is also used to do fine-grained credit assignment across the actions in a trajectory. The authors find that this improves performance.\n\nSTRENGTHS:\n- The approach is novel in that the technique of credit assignment is applied to the constraint rather than the reward. Or at least this is the claim, and I was not able to find otherwise. \n- The approach is shown to be effective. With an ablation study, the authors show that their credit assignment approach gives better performance\n- I have questions about the setup and novelty (see weaknesses), but overall this work seems to represent a positive contribution: the authors show that it is possible to encode trajectory-level constraints with text and to train an RL agent to respect those constraints.\n\nWEAKNESSES:\n- I have a hard time understanding where this work stands in relation to prior works. The main contribution seems centered around the approach of encoding trajectory level constraints using natural language. But both trajectory level constraints and natural language constraints have been explored independently. And the combination of these two aspects seems straightforward: encode the trajectory level constraint using natural language. I may be overlooking something that makes this less straightforward. I don't quite follow the explanation for novelty on lines 101-104. The Related Works section mentions prior work which learns language representations of constraints, but says that these works do not consider trajectory-level constraints. But because the constraints are given in natural language, it doesn't seem that anything in principle prevents these prior approaches from doing so? What is the technical innovation of the proposed work that allows for trajectory level constraints?\n- A few things seem missing from the setup (see questions)\n- I have concerns about the soundness of the described approach. Where does the ground truth come from when training the text-trajectory alignment component? As far as I can tell, there is notion of a textual constraint but not a notion of a ground truth verification of a constraint. From what I understand of appendix 1, the textual constraint is determined to be violated by a discriminator model (see questions). In any case, the authors should explain how constraints are determined to be violated. This matters because it means that the discriminator model and the agent could both make the same systematic linguistic errors.\n- I have concerns about the soundness of the approach in general. Using natural language to describe constraints seems at odds with the enterprise of making RL more \"safe.\" Natural language can be ambiguous, and machine learned representations of language can be imperfect, and prone to making semantic errors. In contrast to other approaches which provide formal guarantees on behavior [1], this seems much less safe. However, I am conscious that others in the field think differently, given the prior work. \n\n## minor\n- Figure 3: Labels are missing in the legend\n- Line 297: \"Table\" --> \"Figure\"\n\n## references\n[1] Fulton, Nathan, and André Platzer. \"Safe reinforcement learning via formal methods: Toward safe control through proof and learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.\n\nQUESTIONS:\n- Line 174: What is a \"negative pair\"? Is it a constraint and a trajectory that violates that constraint?\n- Line 175: Given that the mapping from trajectories to constraints is many-to-many, doesn't $q$ need to be normalized so that the probabilities sum to 1? That is, there are many positive pairs that contain the same constraint, so they should all receive an equal share of positive probability.\n- Fig 5: I think I'm missing something basic: how are is the Pareto front determined? I could assume that per cost, the best reward possible can be determined by some procedure? Is this what was done?\n- Section 6: Where do the textual constraints come from? Are they human annotated? Appendix 1 seems to suggest not.\n  - Is it guaranteed that the discriminator can tell when a textual constraint is violated?\n- Line 289: What is the difference between ground-truth mode and cost prediction mode? Is it the cost assignment?\n\nLIMITATIONS:\nLimitations are mentioned in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-20"}