VVC-Gym: A Fixed-Wing UAV Reinforcement Learning Environment for Multi-Goal Long-Horizon Problems
Authors: Xudong Gong
Feng Dawei
Kele Xu
weijia wang
Zhangjun Sun
Xing Zhou
Si Zheng
Bo Ding
Huaimin Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning Environment
Demonstrations
Goal-Conditioned Reinforcement Learning
Fixed-wing UAV Velocity Vector Control
Primary area: datasets and benchmarks
TL;DR: We provide a novel fixed-wing UAV RL environment, demonstrations, and baselines for multi-goal long-horizon problem research.

Abstract: Multi-goal long-horizon problems are prevalent in real-world applications. The additional goal space introduced by multi-goal problems intensifies the spatial complexity of exploration; meanwhile, the long interaction sequences in long-horizon problems exacerbate the temporal complexity of exploration. Addressing the great exploration challenge posed by multi-goal long-horizon problems depends not only on the design of algorithms but also on the design of environments and the availability of demonstrations to assist in training. To facilitate the above research, we propose a multi-goal long-horizon Reinforcement Learning (RL) environment based on realistic fixed-wing UAV's velocity vector control, named VVC-Gym, and generate multiple demonstration sets of various quality. Through experimentation, we analyze the impact of different environment designs on training, assess the quantity and quality of demonstrations and their influence on training, and assess the effectiveness of various RL algorithms, providing baselines on VVC-Gym and its corresponding demonstrations. The results suggest that VVC-Gym is suitable for studying: (1) the influence of environment designs on addressing multi-goal long-horizon problems with RL. (2) the assistance that demonstrations can provide in overcoming the exploration challenges of multi-goal long-horizon problems. (3) the RL algorithm designs with the least possible impact from environment designs on the efficiency and effectiveness of training.

Published as a conference paper at ICLR 2025
VVC-GYM: A FIXED-WING UAV REINFORCEMENT LEARNING ENVIRONMENT FOR MULTI-GOAL LONG- HORIZON PROBLEMS
Xudong Gongl.2 Dawei Feng Kele Xu Weijia Wang
Zhangjun Sun
Xing Zhou' Si Zheng
Bo Dingl.2 Huaimin Wang
1 College of Computer Science and Technology, National University of Defense Technology,
Changsha, Hunan, China
2 State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China
3 Flight Automatic Control Research Institute, AVIC, Xian, Shaanxi, China
4 College of Intelligence Science and Technology, National University of Defense Technology,
Changsha, Hunan, China
5 Qiyuan Lab, Beijing, China
ABSTRACT
Multi-goal long-horizon problems are prevalent in real-world applications. The additional goal space introduced by multi-goal problems intensifies the spatial complexity of exploration; meanwhile, the long interaction sequences in long- horizon problems exacerbate the temporal complexity of exploration. Address- ing the great exploration challenge posed by multi-goal long-horizon problems depends not only on the design algorithms but also on the design of environ- ments and the availability of demonstrations to assist in training. To facilitate the above research, we propose multi-goal long-horizon Reinforcement Learn- ing (RI.) environment based on realistic fixed-wing UAV's velocity vector con- trol, named VVC-Gym, and generate multiple demonstration sets of various qual- ity. Through experimentation designs on training, assess the quantity and quality of demonstrations and their influence training, and assess the effectiveness of various RL algorithms, pro- viding baselines on VVC-Gym and its corresponding demonstrations. The results suggest that VVC-Gym suitable for studying: (1) the influence of environment designs on addressing multi-goal long-horizon problems with RI. (2) the assis- tance that demonstrations can provide in overcoming the exploration challenges of multi-goal long-horizon problems. (3) the RL algorithm designs with the least possible impact from environment designs on the efficiency and effectiveness of training. Our code is available at https://gtthub.com wwa.ll.ooooooog.o.... "roogguuong/fi craff and https//ggtttbo.coooooooooououogy able at httpp://www..oppm...r////4000000
analyze the impact of different environment
and demonstrations are avail-
INTRODUCTION
Many real-world applications fall into the category of multi-goal long-horizon problems. For in- stance, a UAV must be capable of achieving not only the left-side goal but also the right-side goal (multi-goal); when completing an ascending turn, it is necessary perform horizontal turn first, then accelerate in a straight line, and finally climb in altitude (long-horizon). Addressing multi-goal long-horizon problems with Reinforcement Learning (RL) (Sutton & Barto 2018) encounters a sig- nificant exploration challenge: (1) the multi-goal nature requires the policy to explore not only the state and action spaces but also the additional goal space during training, which intensifies the spatial complexity of exploration; (2) due to the learning signal decreasing exponentially with the horizon (Osband et al. 2016), the long- morizon nature exacerbates the temporal complexity of exploration.
*Correspondence Dawei Feng davyfeng wwq....coo..

---

Published as a conference paper at ICLR 2025
VVC-Gym
Reinforcement Learning
Baselines
Space Definition
Transition
Interaction
conditioned
Task
Observation Space
Dynamics
Reward Functions
Demonstration-baeed RL
Non Markovian reward
Action Space
Termination Conditions
Imitation Learning
Action
State
Equations of Motion
Simulator
AeroDynamic Calculation
Aircraft Model
Demonstrations
Figure 1: The framework of VVC-Gym.
Existing work predominantly focuses on the design of algorithms, neglecting the importance of en- vironment design and the potential benefits that demonstrations can provide during training. Specif- ically, (1) insufficient environment designs hinder training efficiency and effectiveness, such as the absence of effective termination conditions, which can lead to the collection of numerous ineffec- tive samples after severe states occur; (2) demonstrations can assist in overcoming the exploration challenge from multiple perspectives, for instance, by constraining the policy's 's exploration space and providing behavioral prior for the policy. Moreover, within the Goal-Condition
RL (GCRL) community (Liu et al. 2022), existing multi-goal environments (Plappert et al. 2018 Ghosh et al., 2020 Liu et al. [202]] Hu et al. 2022) often feature tasks that are relatively simple with short control sequences, while also commonly encountering the aforementioned two issues. Consequently, the GCRL community urgently requires a multi-goal long-horizon environment that allows researchers to (I) evaluate GCRL more realistic and complex tasks, (2) investigate the impact of environ- ment designs on GCRL training, (3) investigate how demonstrations can be utilized to overcome the exploration challenge, and (4) when engaged in algorithm design, suffer the least possible impact from environment design on the efficiency and effectiveness of training.
To facilitate the community's 's research multi-goal long-horizon problems, we: (1) provide the GCRL community with the first RL. environment on realistic fixed-wing UAV's velocity vector con- trol (VVC) task, VyC-Gym. VVC is typical multi-goal long-horizon problem; one hand, VVC requires the UAV to achieve arbitrary desired velocity vectors, which a typical multi-goal problem. On the other hand, wind UAVs are high-speed vehicles, and precise control high velocities necessitates frequent control outputs (Wang& & Wang 2020), resulting in the long-horizon problem. (2) conduct ablation studies on the environment design of VVC-Gym. showcasing how VVC-Gym can facilitate researchers in studying the influence of environment design on GCRL training. (3) Equip VVC-Gym with multi-quality demonstration sets, evaluate them from various perspectives, and show their effectiveness in aiding GCRL in overcoming the exploration challenge. (4) Provide basclines on VVC-Gym and corresponding demonstrations to illustrate their broad applicability in studying RL, including GCRL, deemonstration-basee RL (Ramírez et al. 2022).
RELATED WORK
Multi-goal environments. Existing GCRL research often conducts experiments on open-source multi-goal environments, such as: (1) the robotic arm control task based on Mujoco (Nair et al., Plappert al.) 22188) Gupta et 2020 Foundation, 2022). DMC (Tunyasuvunakool
2018
al. Panda-Gym (Gallouèdec et al. (021). (2) mike tasks such as PointMaze (Trott et al.
2020
and AntMaze (Nachum al. 2018). (3) Atari games (Warde-Farley al., 2018), etc. Most of these environments either utilize non-realistic tasks with short horizons or employ simplistic envi- ronment designs, such as sparse rewards, without considering mechanisms for terminating episodes in the event of severe states, resulting in low exploration efficiency at the environment level. Addi- tionally, some lack accompanying demonstrations, which precludes the support for demonstration- based RL. In contrast, introduce VVC-Gym address these issues. VVC-Gym features various environment designs and includes accompanying demonstration sets. It facilitates studying the influ- ence of environment designs GCRL training and demonstrations' aids in overcoming exploration
2019
2

---

Published as a conference paper at ICLR 2025
challenges. Additionally, the default environment designs of VVC-Gym enable effective GCRL algorithm evaluation unaffected by environmental designs.
RL for multi-goal problems. Hindsight experience replay (HER) (Andrychowicz et al. 2017 is
a core method employed by existing GCRL algorithms (GCRL algorithm background is provided in Appendix E address the multi-goal challenge. HER enhances sample efficiency by replacing the desired goals of failed trajectories with the achieved states to yield positive rewards. (Pitis al. 2020) summarizes a general goal-conditioned off-policy RL framework for solving multi-goal problems. In this framework, most research focuses on how to sample behavioral goals to further improve sampling efficiency. with the core idea being to avoid sampling training data on goals that the current policy has no capability of achieving (also known as self-curriclum methods). For instance, RIG (Nair et al.] 2018) samples goals directly from the distribution of achieved goals, DISCERN (Warde-Parley et al. 2018) samples uniformly on the support set of the distribution of achieved goals, and MEGA (Pitis et al.) 2020) uses inverse probability weighting sampling (Jaeq) al. 2023) on the distribution of achieved goals to samples goals that the current policy can achieve but not well. Additionally, (Gong et al. 2024a) introduces a general framework for goal-conditioned on-policy RL algorithms. These studies focus on the algorithmic perspective, without considering the potential impact of environment design on training efficiency.
RL for long-horizon problems. Existing work addresses the long-horizon challenges through three approaches. The first approach is Imitation Learning (II.) (Zheng et al. 2022), where poli- cies learned from demonstrations serve as a behavioral prior to bias the exploration of RL (Baker iet al. 20222 Gong et al., 2024b). This method implicitly constrains the exploration space, thereby significantly enhaneing exploration efficiency (Ramírez et al. 2022). The second approach involves designing efficient exploration strategies, with a focus on the behavioral goal selection strategy (re- fer the above RL for multi-goal problems). The third approach is to design subgoal generation strategies, which are centered around the idea of decomposing long- horizon tasks into smaller, more manageable subtasks (Nasiriany et al. 2019). While the aforementioned research concentrates on al- gorithms designs to tackle the long-horizon challenge, our work is dedicated to providing the GCRI. community with a novel and realistic multi-goal long-horizon RL environment. We offer baselines for the three aforementioned methods on VVC-Gym, demonstrating its suitability testbed for researching multi- goal long-horizon problems.
VVC-GYM
The framework of VVC-Gym is depicted in Fig. VVC-Gym comprises two main modules: the Simulator and the Task. The Simulator responsible for UAV-related simulations, while the Task encapsulates the velocity vector control task based on the Gymnasium. Additionally, we equip VVC-Gym with multi multi-quality demonstrations and provide baselines on VVC-Gym. The detailed simulation process is described in Appendix [ and the baselines are presented in Section 4.3 In this section, we provide detailed introduction to the Task and demonstrations.
3.1 PROBLEM FORMULATION
During the process, roll angle Φ, pitch angle 0. yaw angle vh, flight path elevator angle μ., flight path azimuth angle x, true airspeed v, altitude h, roll rate (observation space) are observable. The agent tasked with manipulating the UAV's velocity vector (v. μ,, x) match a desired velocity vector (Ug, Hg. Xg) (goal space), by controlling the deflections of the aileron actuator 5a, elevator actuator de. rudder actuator 8, and power level actuator 6pta (action space).
Two control modes are provided: The first is the guidance law mode, where the controller outputs intermediate control commands: roll rate command Pe, overload command nzee and the position of the power level actuator &pta- These three intermediate commands are then transformed into final control commands 8,, de, ô,to through a control law model that smooths the output, which is detailed in Appendix B The smoothing action of the control law model serves purpose similar to the temporal abstraction provided by frame-skipping temporal complexity of exploration. The second mode is the end-to-en end mode, where the controller directly outputs 8,, d., 5,, 8pta- This mode exhibits significant oscillation in actions during training, and combined with the long-horizon nature of the VVC task, the training suffers greatly from the
(Kalyanakrishnan et al.) 2021). alleviating the
3

---

Published as a conference paper at ICLR 2025
temporal complexity of exploration. In Appendix we provide experimental analysis of the distinct impacts of these two training modes on RL training.
3.2 TRANSITION
The VVC-Gym's Transition employs modular design principle, comprising the Dynamics sub- module and the Termination Condition sub-module. The Dynamics sub-module is responsible for communicating with the Simulator, sending actions, and receiving the new state of the UAV. The Simulator performs the core computations of the aerodynamic equations detailed in Appendix [A] The Termination Condition sub-module tasked with terminating an episode based on certain con- ditions, which are user-definee and can be used to study how to avoid collecting ineffective samples, thereby enhancing exploration efficiency. To facilitate research, we provide categories, totaling default termination conditions:
The first category consists of termination conditions for determining whether the UAV has achieved its desired goals:
(1) Reach Target Termination (RT): if error of ø keeps in the error band for least Ta steps.
The second category consists of termination conditions for truncating excessively long trajectories from both temporal and spatial dimensions:
2) Timeout termination (T): the goal is not achieved within the step limit Taaz- Crash Termination (C): if the altitude of UAV is less than the safe altitude ha-
The third category consists of termination conditions for truncating unreasonable or invalid trajec- tories:
(4) Continuously Move Away Termination (CMA): the velocity vector moves away from the desired velocity vector continuously for TM steps.
(5) Continuously Roll Termination (CR): the UAV rolls Φmaz continuously.
(6) Extreme State Termination (ES): if the true airspeed reaches the limit Umaz or the roll rate p reaches the limit Pmoa-
(7) Negative Overload and Big Roll Angle Termination (NOBR): if the overload component along the OZ axis of the Body Coordinate System Viz is negative and the roll angle é is bigger than a limit dv for more than Tv steps.
We believe that for multi-g multi-goal long- -orizon problems, termination conditions can be designed by ref- erencing the above three categories, which respectively represent (1) the conditions for goal achieve- ment, (2) the maximum limit for exploration, and (3) the truncation unreasonable or invalid tra- jectories based the characteristics of the task itself. The 7 termination conditions proposed for our fixed-wing UAV control scenario are all relatively simple. Although they are straightforward, experiments in Section show that they can significantly enhance exploration efficiency.
3.3 REWARD FUNCTION
Addressing multi-goal long-horizon problems' exploration challenge requires informative rewards, which remarkably enhance training efficiency(Sutiie & Barto 2018 Fekstein & Schiffmann,
2020). Existing multi-goal long-horizon environments typically employ either sparse rewards or simplistic distance- based rewards, such as (s:) -s))(s)) ). ggll. where ( maps a state its achieved goal and ||- calculates the difference between two goals (Liu et al. 2022). Such simplistic
reward functions are not conducive investigation into the impact of reward design on training. To this end, we propose general distance-basee coal-condiiioned reward structure:
0,
if triggers RT
Tggt
Ppenaltys -
if triggers any of CMA, CR, C, ES, or NOBR else
(1((+)))911))
where Tpenalty is a negative number, ||-|| calculates the difference in two velocity vectors, or is a normalization factor for velocity vector difference such that [1(()=))=ll E [0, 1]. b is scaling factor
4

---

Published as a conference paper at ICLR 2025
that controls change rate of the reward function over different error intervals. This reward function gives the agent a negative reward between [-1,0] 0] at each step if the agent does not trigger any termination condition. The closer the current velocity vector aligns with the desired velocity vector, the closer the reward gets to O. This negative reward spurs the agent to achieve the desired velocity vector promptly. The differences from previous rewards (Böhn al. 2019: Koch et al. 2019h a)
are highlighted in two aspects:
(1) Penalty lpeealtty is incorporated into the reward work in synergy with the termination con- ditions, thereby enhancing exploration efficiency. It penalizes the agent for actívating non-RT ter- minations. Crucially, ''penatty must be much less than the minimum single-step reward prevent premature episode terminations. This ensures the agent doesn't 't intentionally trigger termination to avoid low cumulative reward due to negative rewards in each step (Fekstein & Schiffmann, 2020).
We propose two approaches to set "penalty: The large negative constant method, which sets I "penalty much lower than the average reward from triggering termination to discourage premature episode terminations via other conditions. The pessimistic estimation method, which sets penaltty based on the remaining steps until the maximum Tmaz. the agent triggers a lermination at step Tr
Zman Tr
and receives the worst reward of -1 per step, Ppenalty is given by Tpenolty analyze the effects of these methods on learning in Section 4.4.2
We
(2) Scaling factor is incorporated into the reward to adjust the change rate of rewards in different learning stages. As shown in Fig.2 with > 1, the reward reacts quickly to large errors but slowly to small ones, speeding up goal approach from afar but hindering fine adjustments near the goal. Conversely, b < favors accuracy over speed. The choice of b balances the speed of goal achievement with its accuracy. The effect of b on learning is analyzed in Section 4.4.2
Large
crange sokeV
nomallaed entr
Figure 2: reward b
3.4 DEMONSTRATIONS
Demonstrations facilitate the utilization of demonstration--asee methods enhance the learning efficiency of RL, enabling RL to tackle ppploration-challengee tasks (Ramirez et al. 2022). When collecting demonstrations, is significantly less sabor-intensive for human experts to design a sim- ple controller than to collect directly. However, this approach presents the challenge of low demon- stration quality (Brown et al. 2020; Sasaki & Yamashina 2020; Xu et al. 2022) from the simple controller designs. Nevertheless, demonstrations generated by human-designed simple controllers still hold significant value in enhancing the efficiency of RL training (Gong et al. 2024b). There- fore, based on simple human-due numan-designed PID controller (Visioli 2006), detailed in Appendix we
generate multi-quality demonstrations through the following three steps:
Generating demonstrations with the PID controller. We first discretize the desired goal space, detailed in Appendix [D] to obtain a desired goal set. We then use the PID controller to sample within this goal set, recording trajectories that successfully trigger the RT condition as De-
Augmenting demonstrations based on symmetry. Demonstration DE is augmented by leveraging the symmetry in the flight path azimuth angle, as detailed in Appendix D The augmented demon- stration set is denoted as Dg.
Generating more and high-quality demonstrations. Iterative Regularized Policy Optimization (IRPO) (Gong et al., 2024b) is employed to iteratively optimize the policy and generate demon- strations DE, i E [1,2, 1,,,...,Npp ND), with increasingly improved quantity and quality. These demon- strations are then further augmented with the method detailed in the second step, resulting in D', i [1, Np).
In Section we provide a detailed analysis of the differences between these demonstration sets.
EXPERIMENTS
In this section, we: (1) compare the impact of using versus not using the environment designs introduced in Section 3 on GCRL training. (2) present analysis on demonstration quantity and quality. (3) provide baselines for GCRL and demonstration- RI., showcasing the extensive
5

---

Published as a conference paper at ICLR 2025
0.8
0.6
env designs designs
0.5
Success Rate
Success Rate
Success Rate
0.4
0.6
0.4
0.3
0.4
0.2
w/ erv designs
0.2
0.1
wio pre-train
w/o env designs
0.0
0.00
0.25
0.50
0.75
1.00
0.0
0.5
0.0
0.5
2.0
Environment Steps le6
Environment Steps
Environment Steps 1e8
(a) SAC+HER
(b) PPO
Figure 3: Comparison between policies trained by SAC+HER and PPO on using versus not using the environment designs introduced in Section "w/ env designs" refers using all 7 termination conditions and reward Eq. | while "w/o env designs" refers to only using RT and T and reward rg (s{))) -1((())) Results come from experiments over 5 random seeds.
Figure 4: GCRL with and with- out pre-training from demonstra- tions. Results come from PPO and D trained on the hard ver- sion of VVC-Gym over 5 ran- dom seeds.
applicability of VVC-Gym and demonstrations in GCRL research. (4) conduct ablation studies to assess the impact of various environment designs on GCRL training.
In the experiments, we employ two versions of the velocity vector control task: the normal ver- sion and the hard version, which are detailed in Appendix [F] In the experiments, unless otherwise specified, the normal version are used, with the hard version only used when specifically indicated.
4.1 MAIN RESULTS
To demonstrate the benefits of the termination conditions and informative reward function in ad- dressing multi-goal long-horizon problems, we evaluate the impact of the environment designs proposed Section 3 on GCRL training by training policies with SAC (Haarnoja et al.] 2018 HER (Andrychowicz et al. 2017) and PPO (Schulman et al. 2017) (with goal-condittoned policy (Schaul et al. 2015)) on VVC-Gym. The experimental results are displayed in Fig. 3 The results indicate that, irrespective of the specific algorithm employed, policies trained with environment designs surpass those trained without these designs, when an equivalent training budget is allocated. This demonstrates that the termination conditions and reward function tailored for VVC-Gym effec- tively facilitate more efficient training for GCRL algorithms. We provide a detailed analysis of the reasons why the termination conditions and reward function contribute the training in Section 4.4
To illustrate the benefits of demonstrations for GCRL training, we examine the effect of pre-training with demonstrations evident that when an algorithm attempts to train a policy directly without pre-training, it struggles learn any skills due the task's intrinsic exploration complexity. In contrast, after pre-training the policy with Goal-Conditioned Behavioral Cloning (GCBC) (Ding et al.) 2019). the algorithm capable of further improving policy performance through subsequent GCRL training. This high- lights the importance of demonstrations in facilitating more efficient GCRL training. An extensive analysis of demonstration quantity and quality is provided in Section 4.2 and the performance of various algorithms different demonstration sets is detailed in Section
the subsequent GCRL. training. The results are presented in Fig. It is
4.2 ANALYSIS OF DEMONSTRATION QUANTITY AND QUALITY
Demonstrations are collected with methods detailed in Section 3.4 IRPO is used to collect No demonstration sets. We analyze the statistical information of the demonstration sets, and the results presented in Table [ In terms of quantity, the index increases. Di covers increasing number of goals, resulting in an increase total transitions. In terms of quality, there is no signif- icant difference in the goal achieving accuracy. In addition, as discussed in Appendix M also consider the state and action smoothness, which also do not exhibit significant differences across dif- ferent Di. However, the average length of the trajectories decreases. indicating that increases, the trajectories contained within Dj able to achieve goals goal achieving accuracy and state and action smoothness. In summary, as increases, the trajectory quantity and quality of Di improves. The variations among these 8 demonstration sets enable
quickly without compromising

---

Published as a conference paper at ICLR 2025
Table 1: Trajectory Number, Average Length of Trajectory, Total Transition Number, and Goal Achieving Accuracy of Demonstrations
Number of Irajectories
Goal space coverage (%)
Average length of trajectories
Number of transitions
Accuracy
Demonstration
eFTOTy
enror
Dg
10184
20.08
282.01=149.90
2872051
6.56#3.25
0.36+0.35
0,53=0.45
D
10264
20.24
281 11.833119945
2892731
6.56+3.25
0,36+0.36
0.53=0.45
Dg
24924
49.15
124.64a53.07
3106516
4.12+3.45
0.59+0.32
0.57=0.41
D
27021
53.28
119644247.55
3232896
4.47=3.49
0.58+0.32
0.60=0.44
D
33114
65.29
117.65+46.24 24
3895791
4.83#3.45
0,57+0.33
0.66#0.54
D
34952
68.92
115.76a45.65
4045887
5.1643.47
0,56+0.33
0.68+0.60
D3 38654
76.22
116.59+46.81
4506827
5.24=3.41
0.60=0.34
0.71=0.69
D
39835
78.55
116.56247.62
4643048
5.29+3.38
0.60#0.35
0.7420.75
Table 2: Performance of GCRL algorithms on VVC-Gym. Table 3: Performance pre-train &
Results (% success rates 5 random seeds) come from fine-tune algorithms on the hard ver- the hard version of VVC-Gym.
sion of VVC-Gym (% success rates over 5 random seeds).
(a) GCRL algorithms
(b) Curriculum methods
RI. type
Algorithm
Success
Curriculum
Success rute
Demos
GCBC
GCBC PPO
Off-policy
SAC
1,0820.88
None
38.31+1.62
D2 D
17.08=0.57
38.31=1.62
HER
8.32+1.86
RIG
49.03 1.54
97
53,83+0.80
On-policy
PPO
0,04=0.03
DISCERN
49.36*1.91
Di
41.79=0044
68.47=1.200
GCBC + PPO
38.31&1662
MEGA
48.6242.35
Dj.
42,77=1.35
71.68=2.86
searchers investigate the influence of demonstration quantity and quality on demonstration-aasse
RL.
4.3 GCRL AND DEMONSTRATTIO-BAASR RL BASELINES
In this section, we show the potential of VVC-Gym in studying RL by establishing baselines for: (1) GCRL. (2) demonstration-based RL, including IL and RL from demonstrations (RLID) (Ramírez ct al. 2022). Additionally, the reward function in VVC-Gym can be configured as Non-Markovian Reward (NMR) (Abel al. 2021). We discuss NMR Appendix H and provide relevant baselines.
Investigating GCRL. To demonstrate the suitability of VVC-Gym for studying GCRL, we conduct experiments on VVC-Gym as follows:
(I) We train the off-policy GCRL algorithm SAC and HER, the on-policy GCRL algorithm PPO and GCBC+PPO, with the corresponding results presented in Table 2a It noted that, firstly, while employing a goal-conditioned policy standard RL algorithms (SAC and PPO) to address multi-goal problems, However, the integration of techniques specifically designed to tackle exploration challenges, such as HER and GCBC- GCBC-pretraining, leads to a significant improvement in policy performance. Secondly, both HER and GCBC+PPO show potential for increased success rates, suggesting that VVC-Gym poses a challenging task that is well-suited for studying multi-goal long-horizon problems.
coal-conditioned value function directly can adapt
performance is poor.
(2) We train self-curriulum methods including RIG (air al. 2018), DISCERN (Warde-Farle) et al. 2018), and MEGA (Pitis et al,, 2020). with the results presented in Table 2b is evident that different self-curriculum methods among these three methods are not pronounced, suggesting that VVC-Gym is suitable for studying seff-curriculum in GCRL. It has been observed that while the implementation of curriculum methods can significantly enhance the policy's 's success rate, the rate remains relatively low for the VVC task. This is primarily attributed to the inherent challenges associated with the VVC task, which we have thoroughly discussed in Appendix[] !
enhance learning effectiveness. although the differences
(3) We utilize the policy trained by GCBC+PPO+MEGA (Gong et al. 2025 complete a long-horizon ascending turn task, as shown in Fig. 5 involves initially performing horizontal turn, followed by straight-line acceleration, and
The optimal solution to this task

---

Published as a conference paper at ICLR 2025
200
(a) Cumulative reward dur- (b) Success rate during (c) Triggering proportions (d) Episode length when ing training
training
of each termination condi- termination condition trig- tion during training
gers
Figure 6: The left two figures: Comparison between training process with and without termination conditions. The right two figures: Analysis of training process when employing the 7 termination conditions. Results come from experiments over random seeds.
finally ascending in altitude. We sequentially assign these three sub- goals to the aforementioned trained policy, and it can be observed that the policy is able to complete the task effectively. This demonstrates that VVC-Gym supports the modification of the goal for the current episode necessary (Bohn al., 2023). From the inverse perspec- tive, if presented with a complex long-horizon task, VVC-Gym can facilitate research on sub-goal generation methods.
Investigating demonstration-basel RL. Demoonstaation-basee RL help overcome the exploration challenge (Ramírez al. 2022
Gong et al.) 2024b). To showcase the suitability of VVC-Gym for studying demonstration-basee
Figure 5:
A long-horizon ascending turn task finished by GCRL policy with pre- defined sub-goals.
RL. conduct experiments on VVC- Gym with the GCBC and RLfD algorithm GCBC+PPO (Vinyals al. 2019 Baker et al. 2022), with the results presented in Table It is evident that, firstly. the performance of GCBC trained on the
RI. relatively low-quality and low-quantity DE far exceeds that of the non-demonsttaaiion-bann algorithms such as SAC, SAC+HER, and PPO as shown in Table 2a Secondly, both GCBC and GCBC+PPO exhibit improved policy performance the quantity and quality of the demonstrations increase. This suggests that VVC-Gym and the accompanying demonstrations are well-suited for studying demonstration-basee RL, including the investigation of the impact of demonstration quan- tity and quality on training, and the development of high-pe performing policies even with imperfect demonstrations.
4.4 ABLATION STUDIES
In this section, we conduct ablation studies on the termination conditions and the reward function, examining how their settings affect training. In the experiments, all configurations of the environ- ment and algorithm, except for those being studied in the ablation study, are listed in Appendix F in
G
4.4.1 ABLATION ON TERMINATION CONDITIONS
To demonstrate the role of each termination condition during RL training, conduct ablation studies on termination conditions by training policies using PPO, with experimental details listed in Appendix[K] Fig.//shows the corresponding results. Fa (ndolllallll the trends in cumulative reward and success rate with and without the use of termination conditions. Without them, the policy experiences rapid rise cumulative reward, which then plateaus at a certain level, while the success rate remains near zero throughout the training process. The rapid increase in cumulative reward is primarily due to the fact that the reward for triggering is higher than that for other non- RT termination conditions. Conversely, with the use of termination conditions, the policy's cumulative reward can be consistently enhanced, incrementally raising the success rate.
Fig. 6c and 6d illustrate why the use of termination conditions results in better policies. Fig. bc shows how the frequency termination conditions changes over training, while Fig. shows the average length of episodes when termination conditions are triggered. It is evident that employing
8

---

Published as a conference paper at ICLR 2025
20
(a) Triggering proportions (b) Weighted sum of errors of each termination condi-in magnitude and direction tion during training of velocity vector
(a) Success rate during(b) Cumulative reward dur- training
ing training
Figure 7:
Results of ablation on Tpenalty- Results come from experiments over 5 random seeds.
Figure 8: Results of ablation study on b. Results come from experiments over 5 random seeds.
termination conditions can lead the training process to exhibit several distinct stages, each with unique characteristics, outlined in Appendix K The training stages indicate two key advantages of using termination conditions: First, they enable early termination of unreasonable episodes at various stages, avoiding the collection of meaningless samples after the occurrence of severe states (as shown by Fig. [6d). This results more episodes sampled within the same computational budget, thus enhancing training efficiency. Second, the training process features several smooth transitions, which aid in the gradual mastery of goal achievement (as shown by Fig. [6c). This incremental learning process eases policy learning, allowing capabilities evolve progressively.
4.4.2 ABLATION ON REWARD FUNCTIONS
In this section, we conduct ablation studies on the settings of penaatyy b to analyze the impact of different settings training. The specific experimental settings are detailed in Appendix
Ablation study on the setting of Peenatty- In our experiments, we set the penalty reward T Ppenalty with large negative constant and pessimistic estimation, with the results shown in Fig. 7 Fig. 7a illustrates the trend in the proportion of termination conditions triggered during training when us- ing the large negative constant. It is obvious that the agent prefers timeout terminations over other types, indicating a distinct intermediate learning phase where it shifts from frequently triggering non non-timeout terminations to timeout termination. Fig. 76 shows the trend in the error between the end-of-episode state and the goal during training. It is observed that the policy intentionally increases the error, or moves away from the goal, to trigger timeout termination and thus avoid the large penalty associated with triggering C, CR, NOBR, ES, and CMA. Furthermore, in Ap- pendix L.2. we provide an analysis of cumulative rewards and success rates, which further corrob- orates the aforementioned analysis. In summary, employing the large negative constant method in- troduces an intermediate learning stage where the policy adopts an opportunistic and easier-to-leas manner trigger timeout terminations by moving away from the goal. This intermediate phase is counterproductive to the agent's ' goal achievement learning process.
Ablation study on the setting of b. To analyze the impact of the scaling factor policy training. we conduct training with different b and show the corresponding results in Fig. X Fig. Sa and 8b show the success rate and cumulative reward during training, respectively. It can be observed that, except for b = 0.5, policies trained with other b barely possess the capability to achieve goals. However, the larger the value of b, the higher the cumulative reward, indicating that with larger b, the policy tends to have a higher average reward per step. This suggests that with a larger b, although meet the precision requirements for triggering RT, it can quickly move closer to the goal from a greater distance. However, the low success rate of larger b indicates that it fails in helping the policy to continuously approach the goal. We provide detailed analysis of the impact of 6 on velocity direction in the appendix L.3] In summary, the selection of 6 should be carefully considered based on the task requirements, trading off between the speed and accuracy of achieving goals.
the policy may
In summary, through ablation studies, we demonstrate that both termination conditions and reward structure have significant impact on GCRL training, affecting both the training process and the final training outcomes. The default termination conditions and general reward structure of VVC-Gym facilitate researchers in exploring the effects of various environment designs GCRL training.

---

Published as a conference paper at ICLR 2025
5 CONCLUSION, LIMITATIONS, AND FUTURE WORK
In this paper, we introduce VVC-Gym, fixed-wing UAV environment tailored for researching multi-goal long-horizon problems, accompanied by multi-quality demonstration sets. We offer comprehensive description of the VVC-Gym design and the methodology for generating demon- strations. Additionally, we provide several baselines for Coal-Conditioned Reinforcement Learning (GCRL) and demonstration--aase Reinforcement Learning (RL) VVC-Gym and the accompa- nying demonstrations. Finally, we conduct ablation studies on environment designs, illustrating that VVC-Gym is well-suited for investigating the impact of environment designs GCRL training.
Despite the contributions of our work, there are several limitations and directions for extension that should be noted in future research: (1) construct tasks with longer control sequences, including Basic Flight Maneuvers (BFMs) such as Slow Roll and Knife Edge, to facilitate GCRL researchers in utilizing VVC-Gym
challenging long-horizon task environment for their studies. (2) Establish
baselines for automatic sub-goal generation methods and offline RL algorithms (Liu et al. 2024
2.25). (3) Explore methods for collecting low-cost demonstrations for velocity vector control tasks
from human play data.
ACKNOWLEDGMENTS
This work supported by the Science and Technology Innovation Program of Hunan Province (No. 2023RC1005) and partly supported by the National Natural Science Foundation of China (No. 62306325).
REFERENCES
David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. Advances in Neural Information Process- ing Systems, 34:7799-7812, 2021.
David Abel, André Barreto, Michael Bowling, Will Dabney, Steven Hansen, Anna Harutyunyan, Mark K Ho, Ramana Kumar, Michael L Littman, Doina Precup, et al. Expressing non-markov reward to markov agent. In Multidisciplinary Conference on Reinforcement Learning and De- cision Making, 2022.
David Allerton. Principles of flight simulation, volume 27. John Wiley & Sons, 2009.
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.
Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24654, 2022.
Suneel Belkhale, Yuchen Cui, and Dorsa Sadigh. Data quality in imitation learning. Advances in neural information processing systems, 2023.
Eivind Bøhn, Erlend M Coates, Signe Moe, and Tor Ame Johansen. Deep reinforcement learning using proximal policy optimization. In 20/9 international
attitude control of fixed-wing conference on unmanned aircraft systems (ICUAS), pp. 523-533. IEEE, 2019.
Eivind Bohn, Erlend M Coates, Dirk Reinhardt, and Tor Arne Johansen. Data-efficient deep rein- forcement learning for attitude control of fixed-wing uavs: Field experiments. IEEE Transactions on Neurul Networks and Learning Systems, 2023.
Daniel S Brown, Wonjoon Goo, and Scott Nickum. demonsssratr imitation learning via automatically-rantee demonstrations. Conference on robot learning, pp. 330-359, PMLR, 2020.
10

---

Published as a conference paper at ICLR 2025
Shanelle G Clarke and Inseok Hwang. Deep reinforcement learning control for aerobatic maneuver- ing of agile fixed axed-wing aircraft. AIAA Scitech 2020 Forum, pp. 0136, 2020.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances neural information processing systems. 32, 2019.
Felix Eckstein and W Schiffmann. Learning to bl-bbuidiigg autopilot system based neural networks and reinforcement learning. Master's 's thesis, Fern Universitt Hagen, 2020.
Farama Foundation. An api standard for single-agent reinforcement learning environments, with
popular reference environments and related utilities (formerly gym), 2022. /tttps://github com Fa rama- Foundat inn ooniion////naaa Gymnas
Accessed: (2024-04-02)).
Quentin Gallouédec, Nicolas Cazin, Emmanuel Dellandréa, and Liming Chen. panda-gym: Open- source goal-conditioned environments for robotic learning. 4th Robot Learning Workshop: Self- Supervised and Lifelong Learning at NewrlPS, 2021.
Maor Gaon and Ronen Brafman. Reinforcement learning with non-markovian rewards. In Proceed- ings of the AAAI conference on artificial intelligence, volume 34, pp. 3980-3987, 2020.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysen- bach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations, 2020.
Adam Gleave, Mohammad Taufeeque, Juan Rocamonde, Erik Jenner, Steven H. Wang. Sam Toyer, Maximilian Ernestus, Nora Belrose, Scott Emmons. and Stuart Russell imitation: Clean imitation learning implementations. urXiv:2211.1199 [cs.LG], 2022. URL
fabs/221l.199) 11972
Xudong Gong, Dawei Feng, Kele Xu, Bo Ding, and Huaimin Wang. Goal-conditioned on-policy re- inforcement learning. The Tiirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Xudong Gong, Dawei Feng, Kele Xu, Yuanzhao Zhai, ChengKang Yao, Weijia Wang. Bo Ding, and Huaimin Wang. Iterative regularized policy optimization with imperfect demonstrations. In International Conference on Machine Learning. PMLR, 2024b.
Xudong Gong, Dawei Feng, Kele Xu, Xing Zhou, Si Zheng, Bo Ding, and Huaimin Wang. V-pilot: A velocity vector control agent for fixed-wing wing uavs from imperfect demonstrations. In 2025 IEEE International Conference Robotics and Automation (ICRA). IEEE, 2025.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning, In Conference Robot Learming, pp. 1025-1037. PMLR, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- machine learning. pp. 1861-1870. PMLR, 2018.
Edward S Hu, Richard Chang, Oleh Rybkin, and Dinesh Jayaraman. Planning goals for exploration. In The Eleventh International Conference Learning Representations, 2022.
Alexis D Jacq, Manu Orsini, Gabriel Dulac-Arnold, Olivier Pietquin, Matthieu Geist, and Olivier Bachem. On the importance of data collection for training general goal-reaching policies. In Sixteenth European Workshop on Reinforcement Learning. 2023.
Shivaram Kalyanakrishnan, Siddharth Aravindan, Vishwajeet Bagdawat, Varun Bhatt, Harshith Goka, Archit Gupta, Kalpesh Krishna, and Vihari Piratla. An analysis of frame-skipping in rein- forcement learning. arXiv preprint arXiv:2102.037/6 2021.
William Koch, Renato Mancuso, and Azer Bestavros. Neuroflight: next generation flight control firmware. Computing Research Repository, 2019a.

---

Published as a conference paper at ICLR 2025
William Koch, Renato Mancuso, Richard West, and Azer Bestavros. Reinforcement learning for uav attitude control. ACM Transactions on Cyber-Physical Systems, 3(2):1-21, 2019b.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-cooditioned reinforcement learning: Prob- lems and solutions. International Joint Conference on Artificial Intelligence, pp. 5502-5511, 2022.
Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, and Xin Xu. Adaptive advantage-guided policy regularization for offline reinforcement learning. In Forty-first International Conference on Machine Learning. 2024.
Tenglong Liu, Jianxiong Li, Yinan Zheng, Haoyi Niu, Yixing Lan, Xin Xu, and Xianyuan Zhan. Skill expansion and composition in parameter space. In International Conference on Learning Representations, 2025.
Siddharth Mysore, Bassel Mabsout, Renato Mancuso, and Kate Saenko. Regularizing action poli- cies for smooth control with reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1810-1816. IEEE, 2021.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. Advances in neural information processing systems, 32, 2019.
lan Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions. In International Conference on Machine Learning, pp. 2377-2386. PMLR, 2016.
Silviu Pitis. Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning. pp. 7750-7761. PMLR, 2020.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint arXiv: 09464, 2018.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral computation, 3(1):88 077 1991.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL
Dormann.
20 1364. html
Jorge Ramírez, Wen Yu, and Adolfo Perrusquía. Model-free reinforcement learning from expert demonstrations: a survey. Artificial Intelligence Review, 322-3321,,1,,
Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In Interna- tional Conference on Learning Representations, 2020.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06337, 2017.
Flightgear Flight Simulator. Open source aircraft btgg/Fllightgerr/ttpppp http:///rrrrffffoe
http://sttsss/mmrrrrs..tt ibw.roo....iiii...... rera ft (Acessedd: (2024-05-30).
12

---

Published as a conference paper at ICLR 2025
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards, Advances in Neural Information Processing Systems, 32, 2019.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czameeki, Michaël Mathicu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3550 2019.
Antonio Visioli. Practical PID control. Springer Science & Business Media, 2006.
Hao Wang and Weijia Wang. Autonomous control of fixed-wing unmanned aerial system by re- inforcement learning. 2020 3rd International Conference on Unmanned Systems (ICUS), pp- 911-916. IEEE, 2020.
David Warde- Farley Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through parammmmit discriminative rewards. In In- termational Conference on Learning Representations, 2018.
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator- offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, pp. 24725 24742. PMLR, 2022.
Boyuan Zheng, Sunny Verma, Jianlong Zhou, Ivor Tsang, and Fang Chen. Imitation learning: Progress, taxonomies and challenges. IEEE Transactions Neural Networks and Learning Sys- tems, 199):1-16, 2022.
13

---

Published as a conference paper at ICLR 2025
Appendix
A EQUATIONS OF MOTION
The Equations of Motion (FoM) of Fixed- Wing air- craft comprised of translational and rotational nonlin- ear equations of O-Degrees-of-Frreddom (6-DoF) system about the Body Coordinate System (BCS) axes. This generates 12-dimensional fully observable state space, positions and their rates Clarke & Hwang] (2220: the ve- locity component along the OX axis of the BCS u, the velocity component along the OY axis of the BCS the velocity component along the OZ. axis of the BCS w, the roll angle o, the pitch angle 0, the yaw angle v. the roll rate p. the pitch rate q, the yaw rate r. the coordinate of the x-axis in the Geodetic Coordinate System (GCS) zE. the coordinate of the y-axis in the GCS ye, and the coor- dinate of the z-axis in the GCS ZF.
Mn
Z,r
Figure 9; Body coordinate system of fixed-wing aireraft (from Fig.3.9 of Allerton 2009)).
du
T. - D cos a cos 3 Y cos B + Lsin mg sin 0
+ qw
(2)
dt
du
Ty Dsin 3 + Y cos B + mg sin ø cos 0 + pw
(3)
dl
m
dw
T. Dsin a cos Y sin 3 - L cos + mg cos & cos
p
(4)
dt
de
= p + tan 0 (q ø cos
(5)
dt
dt cos Φ - rsin e
(6)
dv
di cos 0 (qsin & + ros の)
dp
(cir + c2p) + c3 + M,M2
(8)
csp ca (p² 2²) + c7 dr
(c&p c2r) q + M₂ cgMs
(10)
where is the thrust force, is the drag force, is the side force, L is the lift force, M is the moment (T. D. Y, L, M calculated by aerodynamic model with open-source aircraft model Simulator (2022). Please refer to the Fig. of Wang & Wang (020)) for the detailed calculation process.). m is the mass the aircraft, g the gravitational acceleration, C1 (I, (1,-1_)1.-13
I.l.
(1, (G-I,+1.)1. C3 T.,--1, C1
1an T_ cs - cs 4.- 1
(1. 1 +12
1=l.-12 12. 1. and C9 1,1,-1 where Iz, Igp I Lee moment of inertia.
Ial. 12
14

---

Published as a conference paper at ICLR 2025
Table Comparison of efficiency between VyC-Gym and popular RL environments. The content within the parentheses indicates the physical engine used in the environment.
Reach (Panda)
HalfCheetah (Mujoco)
Ant (Mujoco)
Hammer (Adroit)
VVC-Gym
FPS
2189
4042
2504
2308 4306
3623
Wall-clock time (s)
4506
2435
3947
2854
For the linear velocities along the three axes in the Global Coordinate System (GCS), denoted as Vz yy Vs, the relationships are follows
dræ cos cos vi + v(snn 0 sin e cos cos cos vb) + w(sin 0 cos @ cos v sin @ sin v))
U2 dt
(11)
dyæ dt
Vy
cos cos sin v v(sin O sin ssin dp cos di cos vb) + w(sin 0 cos e sin dh sin øi cos v)
(12)
dzg
V2 di
sin + v sin @ cos 0 + w cos sinn 0
(13)
Vu2 + 2Z + +u² v + 22 + w2, flight path elevator angle μ arcsin( }), the flight path azimuth angle X arctan( Ke), angle of attack a arctan (.), and sideslip angle 3 = arcsin ()).
Besides, the airspeed V
Although VVC-Gym is implemented in Python, we implement the aforementioned computations with C++, which renders the simulations in VVC-Gym highly efficient. On single node (equipped with Intel Core 19-10980XE CPU, NVIDIA GeForce RTX 3090, and 128GB of memory), train for 10" steps using the PPO algorithm (with 64 rollout workers) within the StableBaselines3 framework and compare VVC-Gym with commonly used RL environments. The results are pre- sented in Table can be observed that VVC-Gym achieves the sampling speed of environments commonly used in academic research and even surpasses several of them.
B THE CONTROL LAW
The control law is responsible for converting the pilot's commands the signals from the automatic flight system into precise movements of the aircraft's 's control surfaces, such as the ailerons, elevators, and rudder, achieve stable flight and precise maneuvering of the aircraft. Our control law model receives overload command nse and roll rate command Pa as input, and outputs the deflections of the elevator and aileron, denoted 8e and da. respectively,
8 kp,e (ng nzee) + kie I (nz neod) dt + ka,e
(14)
8, = kp.a (p - po) + ki,a (p-po) dt
(15)
where kp.e. ki,e, kd,e the proportional gain, integral gain, and derivative gain of the PID troller that controls the elevator, and kp.as ki,o are the proportional gain and integral gain the PID controller that controls the aileron.
15

---

Published as a conference paper at ICLR 2025
C THE PID CONTROLLER OF GUIDANCE LAW FOR VELOCITY VECTOR CONTROL
The expert model of the guidance law takes desired velocity vector Ve: Mes X as input and produces overload command Rasc. roll rate command Po. and the deflection of the power level actuator ópta:
Ax
Ryc.e
Vg.hor/99
(16)
Tx
Au
Azzee
Vg/g + co8
(17)
тμ
Rze
|gycee + nzzc,
(18)
de tan -1 Muya.
(19)
-0s 0
kppp (de - ()) + kd
(20)
Opto kpplo (Ue- - v) + Alla
(21)
where Ryc.e is the lateral overload command in GCS, nzz,, is the normal overload command in GCS, Ay = fe 1 is the error in flight path elevator angle, Ax= = Xx azimuth angle, Tx is the estimation of the time required for the aircraft reach the desired flight path azimuth angle, T, is the estimation of the time required for the aircraft to reach the desired flight path elevator angle, V, is the aircraft ground speed, hor is the horizontal component of Vg, Pe is an intermediate variable representing the roll angle command, kp.p,kd.p are the proportional gain and integral gain of the PID controller that controls the roll rate command, kp.pla is the proportional gain of the PID controller that controls the deflection of the power level actuator, and A pta is constant that ensures that the aircraft maintains its velocity without decrease during level flight.
is the error in flight path
Table 5: Parameters used to discretize the goal space.
D GENERATING DEMONSTRATIONS
Pmmnn Umax) [pminn [maz] Xminn Xmaz] is discretized to G {mmnn + E 1}}, where No, Ny, Ny are the number that equally divide the v, 14, X spaces. In the experiments, we utilize the parameters specified in Ta- to generate the discrete goal set.
Discretizing the goal space. The goal space
l E [1,2,.. .,, No 1]} {umin + Himay [1. N₂ 1]} x [Xmin + i Xmmog - Xmii li [1.2..... Nx N.-1
V
X
N.
min
100
-85
-170
max N
300
85
170
21
35
69
#discretized goals: 50715
If DE contain a trajectory To ((s1,a)),- a.....,..... (sT,aT) that achieves goal g (v, then Dr is aug- {( aa ")T)),..... "..,..., .. "), where g (u, -0),,-)). (0, v, - -x, p4, hp, v), a (-p, n2 Spla). If Dg has a trajectory that can achieve
Augmenting demonstrations based on symmetry.
mented with Tg
g then the shorter one kept.
E ALGORITHM BACKGROUND
E GOAL-CONDITIONEED REINFORCEMENT LEARNING
Let A(X') denote the probability distribution over a set X. Goal-conditineed RL is described by augmeneedet MDP Liu al. ( Pitis et al. (020) M = (S, A, r, 7, 9, Py: Py:C), where S. A. are the state set, action set, and discount factor, T S x A + A(S) is the transition probabilities, r = {r,(rg : S + Rgg E G} is the goal-conditioeed reward functions, G is the space of goals, Pg is the desired goal distribution, and Ç S G is tractable mapping function that maps the state to a specific goal. For a fixed goal g. solving it means finding optimal policy from standard MDP M2
ASA.AT,ra 7). There are three kinds of goals in goal-conditioned RL: desired goal is the requirements of the task, and the desired goal distribution is denoted as Pag: achieved goal is the corresponding goal achieved by the current timestamp and state, and the
16

---

Published as a conference paper at ICLR 2025
Table 6: Parameters used in Environment
(a) Normal Version
(b) Hard Version
Parameter
Value
Parameter
Value
Unniny Umaz
150, 250
Uminn Umas
100, 300
Panins Mynoa
-10, 10
Mmnin MmnnsyPmar Hynor
-85, 85
Xminn Xmna
-30,
Xonin Xman
1170 170
simulation frequency
10
simulation frequency
T Tmaz We: Wd
400
Twax
400
0.5, 0.5
We: We
0.5, 0.5
Ou dd b
100, 180
Ge da
100, 180
0.5
0.5
Tpenalty TR, TM
pessimistic estimation
Fpenalty
pessimistic estimation
10, 20
TR, TM
10, 20
hg
0
hg
0
Omar
720
Pmax
720
Umazs Pmar in ES N TN
400, 300
Umaz Pmnar in ES Φ TN
400, 300
60, 20
60, 20
achieved goal distribution is denoted as Payi behavioral goal is the goal for sampling trajectory in the current episode Liu et al. (2022). The policy parameterized by 0 is modeled based on the idea of universal value function approximators (UVFA) Schaul et al. (015)). The
Liu
S x G - ((A)
objective of goal-conditioned RI. is to maximize the expectation E. gyppaa
1= r (c)]-
E.2 IMITATION LEARNING
Imitation learning is demonstration driven, sample-efficient method for learning policies by imi- tating demonstrators Belkhale al. (2023). IL assumes access dataset DE (Ti...., TN) of N demonstrations. sampled by the demonstrator T (s.) through environment dynamics T T-|ss,, ar). The objective of IL is to learn policy Te S + (AA) parameterized by 0 from Dg. Behavioral cloning (Pomerlea 1991 is a widely applied IL method, which learns the imitation policy by optimizing a supervised
{($1, ) ...., (8T,, ar;}}
is sequence of length T, of state-action pairs
loss maximize the likelihood of demonstrator actions Sasaki & Yamashina (2020):
(())= = De (logm((sss))
(22)
F TWo VERSIONS OF VELOCITY VECTOR CONTROL TASKS
We implement two versions of the velocity vector control task in VVC-Gym: the normal version and the hard version, which are detailed in Table The distinction between the two versions lies in the definition of the goal space. The hard version features larger goal space, which encompasses nearly the entire three-dimennional space, making it more challenging.
G ALGORITHM DETAILS
The Imitation Gleave et al. 2022) framework is utilized to implement behavioral cloning (BC) Pomerleau (1991) algorithm with parameters listed in Table 77a and the Stable Basclines3 Raffin et al. (2021) framework for SAC and PPO with parameters listed in Table 7b and 7c 128*128 fully connected network and the Tanh activation function are used.
H SUPPORT FOR RESEARCH ON NON-MARKOVIAN REWARD PROBLEMS
The values of TR, Tas : Ty in termination conditions R. CMA, and NOBR respectively shape the reward function's 's characteristics, impacting the policy's form. When Tk TM Tn = 1. rewards are Markovian that only depend on the current state, allowing policies to be based solely on the current state. If any of TR, T T, Ty exceed 1, rewards become dependent history of consecutive
17

---

Published as a conference paper at ICLR 2025
Table 7: Algorithm configurations.
(a) BC
(b) SAC
(c) PPO
Parameter
Value
Parameter
Value
Parameler
Value
12.weight
0
ent_coel"
'auto'
ent_coef
10
ent weight
10-2
batch
4096
gamma
0.995
gamma
0.995
epochs
300
Ir batch size
3 10
gae_lambda
0.95
1024
x 10-4
buffer_size
2 10a
batch_size
4096
learning.star starts
10240
train_steps
x 10a
gradient_stepp
1 108
proccessp num
64
train steps
n_steps
2048
tollout_proces__um
I
n.epochs
use sde
False 4
usc_sdc
True
HER:
normalize addaitage
True
Table 8: Success rate MR and various NMR problems. Results come from experiments over random seeds.
TR
Reward type
Success rate
MR
84,43+4.74 12.93
10
NMR
20
NMR
45.03=+00.5
30
NMR
21.9946.63
states, making them non- Non-Markovian, and policies must account for state history Abel et al. 202 Gaon Brafman (2020); Abel et al. (2022).
Investigating non-Markovian reward problems. Non-Markovian reward (NMR) refers to the ward that depends on multiple steps of states and action [Abel et al. (202)) such as determining whether an UAV has stably achieved desired velocity vector based cally, the longer the state sequence that a NMR depends on, the more challenging it becomes to train policies with RL. To illustrate the impact of NMR on RL training, we set different values of TR in VVC-Gym, which represents the dependency of the NMR on the length of the state sequence. The results are presented Table 8 is observed that as TR increases, the policy's 's performance deteri- orates. Additionally, adjusting Tw and Ty can also alter the Markovian nature of reward function. This indicates that VVC-Gym is well-suited for studying NMR problems.
sequence of states. Typi-
I THE TWo CONTROL MODES OF VVC-GYM
We train both guidance law model and an end-to-end model using SAC on the easy version of VVC-Gym, and present the corresponding results in Table 9 It can be observed that the trained guidance law model demonstrates superior performance. The reason for this is that the control law model provided in Appendix is capable of stabilizing the control outputs, which serves purpose analogous to the temporal abstraction achieved through frame-skipping, thereby reducing the temporal complexity of exploration.
The above results can be interpreted from two perspectives: On one hand, the use of simple control law model to stabilize RL outputs helps address the temporal complexity of exploration. On the other hand, although the end-to-end mode is more difficult to train, it also serves more challenging testbed for evaluating RL exploration methods. Additionally, the end-to-end mode can also be employed to investigate the oscillation action problem Mysore al. (021 in RL.
18

---

Published as a conference paper at ICLR 2025
Table 9:
Success rate of SAC between training guidance law model and training end-to-end model. Results come from experiments over 5 random seeds.
Guidance law model End-to-end model
Success Rate (%)
90.40=0:49
7.60+5.08
Table 10: Success rate (%) of GCRL algorithms Reach, PointMaze, and VVC. The demon- strations used in the Reach experiments are from the official script provided by Panda-Gym, the demonstrations used in the PointMaze experiments from Mínari, and the demonstrations used in the VVC experiments are Dg. Results come from experiments over random seeds.
MEGA
GCBC
GCBC+PPO
Reach
100,0=0.0
70.63+2.99
100.0#0.0
PointMaze
100.0#0.0
75.96+5.34
93.3343.06
VVC
8.32+1.86
17.08+0.57
38.31#1.62
J CHALLENGE OF THE VVC TASK
Fixed-wing UAV's VVC is a challenging task. The difficulties lie in: (1) The large exploration space of the policy, which is a continuous state, continuous action problem, and the policy requires additional exploration of the goal space during training.
(2) The long interaction sequences, with the average length of demonstrations exceeding 280. Even well-trained policies require an average of over 100 steps to achieve goal, and more challenging goals can demand upwards of 300 steps (see Table T for corresponding results).
We provide evidence that VVC is challenging task through the following three sets of experiments:
Firstly, standard RL algorithms struggle to solve the VVC task. Table shlshows the success rates of SAC (1.08%) and PPO (0.04%). It is evident that SAC and PPO struggle solve the VVC task.
Secondly, existing GCRL algorithms can effectively solve common multi-goal tasks in aca- demic research, but they can only solve the VVC task to certain extent. We compare the performance of different GCRL algorithms on VVC and common multi-goal tasks in academic re- search, PointMaze (PointMaze Large DIVERSE G-v3) and Reach (PandaReach- are shown in Table TO It can be seen that these GCRL algorithms can almost completely solve Reach and PointMaze, but the best algorithm achieves only 38.31% success rate VVC. These results indirectly reflect that VVC is challenging task.
The results
Thirdly, the human-designed classical PID controller (detailed in Appendix C has only 20.08 % success rate, which also indirectly reflects that VVC is a challenging task.
We believe that for academic research, the difficulty of the task should progress in tandem with the research on algorithms. The task should have appropriate levels of difficulty to properly evaluate different algorithms. If the task too easy, too hard, or unsolvable, it will fail to provide a useful signal for benchmarking. Therefore, we believe that the current success rates of GCRL algorithms on VVC being less than 50% is helpful for researchers to discover more insights when designing algorithms.
K ABLATION ON TERMINATIONS
K.1 EXPERIMENT DETAILS
For the experiment with termination conditions, all the 7 termination conditions are used in the environment, with parameters listed in 6a For the experiment without terminations conditions, only RT, T, and ES are used in the environment, with the corresponding parameters the same as 6aa In the comparative experiments, the primary purpose of using ES is prevent excessive values of linear velocity and angular velocity p, which could lead to floating point overflow errors. However, we utilize the largest possible values for Umar 600, Pmaz 600 ensure that ES operates with the
19

---

Published as a conference paper at ICLR 2025
(a) Error in magnitude of (b) Error direction of (c) Weighted sum of er- (d) Triggering proportions velocity vector
velocity vector rors in magnitude and di- of each termination condi-
rection of velocity vector tion during training
Figure 10: Statistical information on training process of termination conditions. Results come from experiments over 5 random seeds.
minimal necessary intervention. For the PPO algorithm, both experiments use parameters listed in
7c
K.2
ADDITIONAL. ANALYSIS ON THE ABLATION OF TERMINATION CONDITIONS
From Fig. 6c and 6d it can be observed that when termination conditions are employed, the entire training process can be divided into four stages:
In the first stage, when the number of training episodes is less than 10*, the policy lacks any mean- ingful capability and tends to explore randomly. The policy frequently triggers CR, NOBR, ES, and CMA, with short average episode lengths when these conditions are triggered.
The second stage occurs when the number training episodes between 10 and 5 * 10 During this stage, the frequency of CR being triggered significantly decreases, while the frequency of C being triggered increases substantially. Additionally, the average episode lengths for CR, NOBR, ES, and CMA are notably longer. This indicates that in this stage, the policy has learned how to quickly avoid triggering CR, NOBR, ES, and CMA. And the higher frequency of triggering C is due the policy's s lack of capability continuously approach the goal, leading to frequent triggering of C.
The third stage when the number of training episodes is between 5 10a and 3 10a In this stage, the frequency of C being triggered significantly decreases, while the frequency of T being triggered rapidly increases. This suggests that as training progresses in this stage, the policy begins to acquire the ability to approach the goal, but it is not yet able to do so quickly or continuously, resulting in frequent triggering T.
The fourth stage occurs when the number of training episodes exceeds 3 * 10 In this stage, the pol- icy's ability to approach the desired goal is gradually improved, leading to an increasing frequency of triggering RT. The policy starts to possess the capability to complete parts of the goals.
Furthermore, the trend in the error between the state at the end of episode and the goal, which is present in Fig. T0 also corroborates the aforementioned four stages exhibited by the learning process when termination conditions are used.
L ABLATION ON REWARD FUNCTIONS
L.I EXPERIMENT DETAILS.
Aside from the parameters investigated in the ablation study, all other environment and algorithm configuration parameters are consistent with those detailed in Table 6 and 7 In the ablation study on Fpenalty Tpenatty = - -1000 is employed for the large negative constant method, and Tpenolty Tmax T. is employed for the pessimistic estimation method, where T, is the length of current sampled episode r. In the ablation study on the scaling factor b, 0.25, 0.5, 1.0, 2.0, and 4.0 are employed for training. From the previous experimental results, it is evident that controlling the
20

---

Published as a conference paper at ICLR 2025
-10
(a) Cumulative reward dur- (b) Success rate during (c) Error in magnitude of (d) Error in direction of ing training
training
velocity vector
velocity vector
Figure 11: Statistical information on training process of different ppenatty- experiments over 5 random seeds.
Results come from
105
46
<pioad
36
(a) Error in magnitude of ve- locity vector
(b) Error direction of ve-
(c) Weighted of magmitude direction velocity vector
locity vector
Figure 12: Errors of velocity vector of policies trained with different b during training. Results come from experiments over 5 random seeds.
(a)
0.25
(b) b 0.5
(c) 1.0
(d) b 2.0
4.0
Figure 13: Triggering proportions of each termination condition during training. Results come from experiments over 5 random seeds.
magnitude velocity is simpler than controlling the direction of velocity. Therefore, we used a larger wa compared to w. specificallly setting wa 0.5, 0.625, 0.75, and 0.875 for training.
L.2 ADDITIONAL ANALYSIS OF THE ABLATION ON Tpenalty-
Fig. 1Ic and Fig. 1ld respectively depict the error in magnitude of velocity vector and the error evident that when large negative constant is
in direction of velocity vector during training. employed, the policy opportunistically moves away from the goal trigger the timeout termina- tion, thereby avoiding the greater punishment associated with non-timeout termination. The above analysis aligns with our conclusion in Sec. 4.4.2
Additionally, Fig. Ha and Ilb illustrate the changes in cumulative reward and success rate during training for the two methods of setting Tpenalty. When using the large negative constant method, the cumulative reward plateaus at around -300 after rising from -1000, and the policy's success rate remains nearly
throughout, indicating that the policy has learned to avoid triggering C, CR, NOBR, ES, and CMA by frequently triggering timeout terminations, and difficult to learn to achieve goals based this behavior.
21

---

Published as a conference paper at ICLR 2025
Table 11: Error in the direction of the velocity vector and the immediate reward at the last step of the trajectory finished by policies trained with different b. The 'Error interval' indicates the range to which the error the direction of the velocity vector must decrease before the reward change rate exceeds (The curve of reward varying with error can be found in Fig. 2). The 'Error between the state at the end of episode and the goal' represents the error in the direction of the velocity vector between the last state of the trajectory finished by policies and the goal. The 'Immediate reward at the last step of episode' is the immediate reward of the last state of the trajectory finished by policies.
Error between the state
Error interval
Immediate reward at the last step of episode
b
at the end of episode and the goal
(reward change rate ? 1)
0.25
(0, 63.64)
70
-0.79
0.5
(0, 45)
40
-0.47
1.0
60
-0.33
2.0
(90, 180) (113.39, 180)
40
-0.05
4.0
60
-0.01
L.3 ADDITIONAL ANALYSIS OF THE ABLATION ON b.
Fig. 12a and 12b present the trends in the error of velocity magnitude and velocity direction between the state at the end of an episode and the goal during training. Controlling the velocity magnitude of the UAV is relatively simple, and a smaller b is more beneficial for the UAV to improve the accuracy of reaching the desired velocity magnitude. Controlling the velocity direction of the UAV is more challenging, and the experiments with = 0.5 and = 2.0 achieved the best results. This is because when 6 > I, the reward function changes rapidly in the region of high error but slowly in the region of low error, which is beneficial for the policy to quickly approach the goal when the error is large but not beneficial for the policy to continuously approach the goal when the error is small; when b < 1, the reward function's 's characteristics are opposite. Therefore, a b slightly greater or less than can effectively balance the policy's s speed in different error intervals, somewhat improving the precision of approaching the goal, as shown in Fig. 12c However, when b is too large or too small, significantly increases the policy's ss speed one interval but severely reduces another, which, as evidenced by the experimental results, is not beneficial for the policy to achieve goals.
Fig. 13 presents the proportion of termination conditions triggered by policies trained with different b during training. From overall perspective, learning progresses, the policy effectively learns to avoid triggering the four termination conditions CMA, CR, ES, and NOBR, with C and T dominating the training process for most of the time: (2) as b increases, the proportion of policy triggering T also increases, indicating that a larger b is beneficial for the policy to approach the goal. This is consistent with the analysis of Fig. 12
Additionally, Table TI presents the immediate reward at the last step of the completed trajectory and the error between the state and the goal. It is observed that only when reduce the error to a region where the reward change rate exceeds 1. When b is too small, the reward remains in the error range where dr < 1, which is not conducive to the policy quickly approaching de
0.5 can the policy
the goal. When 6 = 0.25, the policy does not even reduce the error to the dr b is too large, the reward remains in the error range where dr < 1, which is not conducive to the policy continuously approaching the goal. When b and b = 4, although the policy can reduce the error to the dr de < 1 error range, the slow change in reward with error makes it difficult for the policy to learn, thus preventing the policy from continuously reducing the error to trigger RT.
de > 1 error range. When
In summary, When b < 1, it is beneficial for the policy to learn the ability to continuously approach the goal in the error-small interval, but it is not conducive to the policy acquiring the ability to quickly approach the goal in the error-large interval. Conversely, when b > 1, it is beneficial for the policy to acquire the ability to quickly approach the goal in the error-large interval, but it is not conducive to the policy acquiring the ability continuously approach the goal in the emrll-small interval. Therefore, the selection of b should be carefully considered based on the task requirements, balancing the tfade-off between the speed and accuracy of achieving goals.
22

---

Published as a conference paper at ICLR 2025
(a) Errors in magnitude of (b) Errors in direction of (c) Cumulative reward dur- (d) Success rate during velocity vector
velocity vector
ing training
training
Figure 14: Results of ablation study Wa, wd. Results come from experiments over 5 random seeds.
Table 12: The Smoothness Measure of State and Action of Each Demonstration. As there is almost no difference between D'g and Di E {0, 1, 2, 3). we show the smoothness measure of Di in this table. Optimal values are highlighted in bold, and sub-optimal values are underlined.
Demondiaaion
action
6,44
0.25+0.05
0.0640.83
0.23e0.12
0.23w8.12
1.47a1331
0U6400447
0,1500,09
0,46H0.23
0.00668.005
0.001500.0016
0.1466009
1.962.14
0.3600116
03110016
0084006
2.241164
3065545.14
0.2000112
2.1162.21
0.030007
0.012840
0.31+0.49
0.56-0.2
13540277
0:310.:17
1.440.64
17/
0.8150:46
3.1862.17
1,160.87
08150.51
04860221
63443.90
1600223.16
0.4790.19
7.5466.11
0.4750.54
0.01610
0.2561118
0.3120029
0.0000.37
0.56:0.31
02440115
4.8343118
1435083
0.3120.1
1.520992
0.0568003
19g
0.82e11.38
3.34+2.02
1,30=1.07
0.8300.4
0.520223
6.9523.94
18.55x26.42
0.48a0.20
10.98:1165
0.4360.47
0,01680015
0.28401:0
0.3920.65
0.0080:39
0.58
0:275018
4.8365.11
1312093
0.3120.18
1.5550999
0.06:0003
0.004200
0.8261133
3.34x261
1.20g11000
0.8120.39
03569923
7.1563.81
20.15a27.62
0.4960.19
11.9413364
0.4820.45
0,01800
L.4 ABLATION STUDY ON THE SETTING OF w, AND wa-
In the experimental process, we discovered that Eq. ! can be further refined by breaking down the term -( -(16()))= y into two separate components: -ww,( [1 It w.(( [ [ y*), where II-lle du
d2
calculates the difference in magnitude of two velocity vectors, and ||-Ila calculates the difference in the direction of two velocity vectors,0,: : da are normalization factors for velocity and direction such [ |EE-
that 11,l= [0, 1], for velocity and direction. After the decomposition,
de E [0, 1). w. E [0, I], wa E [0, W. wa 1.0 weight factors
de
balance the importance of the velocity direction and the magnitude by adjusting the weights wy and wd.
To analyze the impact of w, and wa on policy training, conduct training with different values of w,, and Wa and present the corresponding results in Fig. [4] From Fig. [14a] it is evident that the smaller the value of We, the larger the error in velocity magnitude at the end of episode, indicating a positive correlation between the increase in reward weight w, and the reduction in error for the relatively simple control of velocity magnitude.
Fig. 14b shows that when wa is set to 0.5, 0.625, and 0.75, the resulting policies have similar errors in velocity direction, but the larger the value of wa, the faster the error decreases during training, suggesting that a higher Wa is more beneficial for the policy to reduce the error in velocity direction. However, when wa is set to the highest value of 0.875, the error velocity direction is the largest. This is because the policy is unable to reduce the error in velocity magnitude to within 10, which prevents the policy from triggering RT and limits its ability to decrease the error in velocity direction.
Fig. 14c and 14d illustrate the trends in cumulative reward and success rate. It can be observed that appropriately increasing wa is beneficial for enhancing the policy's capability to obtain cumulative reward and to speed up the increase in success rate. However, excessively large wa affects the error in velocity magnitude, which is detrimental to the improvement of cumulative reward and further restricts the increase in success rate.
M STATE AND ACTION SMOOTHNESS OF DEMONSTRATIONS
Two metrics introduced to measure the smoothness of states and actions. Taking the pitch angle 0 as an example, the formulas for these two metrics are given in the following. And the other variables of state and action are the same. The first metric is inspired by Discrete Fourier Transform (DFT),
23

---

Published as a conference paper at ICLR 2025
and is calculated as
(T)
M,S.-
(23)
n.f t=1
where is the sampling frequency, M, and f; are the amplitude and frequency of the 1 frequency component Mysore et al. (202I). Larger values on this metric indicate the presence of larger high- frequency signal components, thus less smoothness the signal The second metric calculated by accumulating the change of variable over the entire trajectory, and is calculated as
|r|-1
<T(T) = Σ 10+1 0.l.
(24)
The larger this value, the greater the oscillation of variable between consecutive timestamps.
Table 12 shows that Di has nearly the smallest 81 and 82 values among all demonstration sets. This because the derivative term of the PID controller explicitly suppresses oscillations the control variables. When IRPO starts updating Dg with the policy trained by RL, both smoothness metrics and 82 begin increase. This because we do not include terms explicitly in the reward function to penalize oscillations, and the inadequately trained RL policy is unable to achieve goals at the fastest speed, resulting in some degree of oscillation. However, there is little difference between & and 82 across 1e Di. and Dj, This indicates that the policies trained with VVC-Gym's 's reward function do not exhibit significant differences the smoothness states and actions.
N SUGGESTIONS FOR TRANSFERRING MODELS TRAINED BY VVC-GYM TO REAL WORLD UAVS
We suggest the followings when transferring models trained by VVC-Gym to real world UAVs:
VVC- employs open-source UAV model Simulator (2022)
need to consider the differences between the UAV model the environment and real-world UAVs.
We have not yet incorporated environmental dynamic factors into VVC-Gym, such as changes in wind, air temperature, and humidity. Before deploying the model to real-world UAVs, it is necessary to thoroughly evaluate the robustness of the model.
is necessary to consider the computational capabilities of the onboard computer to ensure that it can complete the model's inference within the physical time of an environment step.
O SOCIETAL IMPACTS
We open-source VVC-Gym and its accompanying demonstrations. They can be utilized to train control policies for fixed-wing UAVs. However, is important note that VVC-Gym's design does not account for specific hardware-related requirements, such as the need to avoid (1) bang-bang control to minimize hardware wear and tear, and (2) prolonged overload beyond a certain threshold to ensure the structural integrity of the aireraft within acceptable limits.
24


Reviews:
[{"review_id": "RRRxesGYcK", "reviewer": "Reviewer_HNY5", "summary": "The Paper presents a Multi-Goal Long Horizon RL Environment based on Fixed Wing UAV Velocity Vector control. The paper further provides a set of various demonstrations, analyzing the quantity and quality of the demonstration and their effect of training. The paper claims that the environment presented is suitable for studing curriculum learning that combined imitation learning and RL, influence of the environment designs on multi-goal long-horizon problems.", "questions": "- It is unclear how the environment would work with the presence of Adversarial Fixed Wing Agents [1, 2].\n\n- Is the environment applicable to Inverse Reinforcement Learning applications?\n\n- How much reward engineering is required to train an agent to learn a different policy than what is provided?\n\n- What are the challenges of transfering models trainined in this environment to real world UAVs?\n\n[1] Strickland L. G., “Coordinating Team Tactics for Swarm-vs-Swarm Adversarial Games,” Ph.D. Thesis, Georgia Inst. of Technology, Atlanta, GA, July 2022, [https://smartech.gatech.edu/handle/1853/67090](https://smartech.gatech.edu/handle/1853/67090) \n[2] B. Vlahov, E. Squires, L. Strickland and C. Pippin, \"On Developing a UAV Pursuit-Evasion Policy Using Reinforcement Learning,\" 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, FL, USA, 2018, pp. 859-864, doi: 10.1109/ICMLA.2018.00138.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a detailed analysis of the presented environment, including evaluation of Demonstrations and training of various RL Algorithms with and without Curriculum Methods.\n\nClear problem formulation and transition function.\n\nDetailed Appendix and Supplementary Materials containing demonstrations.", "weaknesses": "- It is unclear what the novelty of the environment is compared to other simulation based fixed wing control environments.\n\n- It is unclear if the machine generated demonstrations can be substituted by human demonstrations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QdCgaOsy4I", "reviewer": "Reviewer_exNf", "summary": "This paper proposes a multi-goal long-horizon Reinforcement Learning (RL) environment based on realistic fixed-wing UAV velocity vector control, named VVC-Gym. The proposed environment is studied through various ablation studies using different Goal-Conditioned RL (GCRL) algorithms and is equipped with multi-quality demonstration sets. Baselines are also provided on the environment.", "questions": "1. In Fig. 6c, we can conclude that there are several distinct stages in training where different termination conditions are triggered. However, what can we conclude from Fig. 6d?\n\n2. Seems that the success rates of all the baseline methods are often low (less than 50%). What might be the possible reasons that the GCRL algorithms fail to achieve a higher success rate?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed benchmark is useful for the GCRL community. \n\n2. The benchmark is equipped with demonstrations with different quality and several baseline algorithms. \n\n3. The influence of the environment-related parameters is studied via ablations.\n\n4. Baseline methods are provided with the benchmark.\n\n5. The paper is well-written and easy to read.", "weaknesses": "1. Some writing issues:\n\n    - The citation format is not very suitable. The authors may consider using `\\citep` for most of the citations.\n    \n    - Line 349 and Line 357: I guess Fig. 2a and Fig. 2b should be Table. 2a and Table. 2b.\n\n    - Figure 5 is provided but not mentioned in the text.\n\n2. The only task provided is the velocity vector control. If various types of different tasks are provided, the paper will have a larger influence and contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6bGD3o917t", "reviewer": "Reviewer_ecAp", "summary": "This paper propose a multi-goal long-horizon Reinforcement Learning (RL) environment based on realistic fixed-wing UAV’s velocity vector control, named VVC-Gym, and generate multiple demonstration sets of various quality. I suggest that the authors improve their academic writing skills, especially in organizing their ideas properly and expressing their points of view.", "questions": "Generally, I strongly recommend several papers, as shown below, in which authors can learn how to improve academic writing skills and organize corresponding ideas from them.\n\n1) Yang, Q., & Parasuraman, R. Bayesian strategy networks based soft actor-critic learning. ACM Transactions on Intelligent Systems and Technology (TIST).\n\n2) Mannion P, Devlin S, Duggan J, Howley E. Reward shaping for knowledge-based multi-objective multi-agent reinforcement learning. The Knowledge Engineering Review. 2018", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper propose a multi-goal long-horizon Reinforcement Learning (RL) environment based on realistic fixed-wing UAV’s velocity vector control, named VVC-Gym, and generate multiple demonstration sets of various quality.", "weaknesses": "I suggest that the authors improve their academic writing skills, especially in organizing their ideas properly and expressing their points of view. For example, the paper first presents the concept of multi-goal long-horizon problems. It does not clearly define the problem or list related works to support the issues, but it spends many words discussing the GCRL, which confuses me about the paper's writing logic. If the authors think the multi-goal long-horizon problems belong to the GCRL, they need to discuss the background of the GCRL and the current state-of-the-art. Then, they need to analyze their relationships and provide recent research on the challenge. Moreover, I believe this problem might belong to the multi-objective multi-agent area. Please check the paper: \nRădulescu, R., Mannion, P., Roijers, D.M. et al. Multi-objective multi-agent decision making: a utility-based analysis and survey. Auton Agent Multi-Agent Syst 34, 10 (2020).\n\nI feel it is more like a technical paper introducing a platform in reinforcement learning than a research paper. Additionally, I suggest the authors add more baseline algorithms to their experiments, not just PPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2xu8SgGU4H", "reviewer": "Reviewer_soQJ", "summary": "This paper presents VVC-Gym, a fix-wing UAV RL environment for multi-goal long-horizon tasks. The environment is built upon real dynamics and rooted in a real-world UAV control problem. Various simulation shows VVC-Gym is suitable for studying the influence of environment designs on multi-goal long-horizon problems, the necessity of expert demonstrations on the task and suitable RL algorithm factors for the environment.", "questions": "Please see \"Weakness\" section. I will read the rebuttal and adjust scores in reviewer-author discussion.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Fix-wing UAV control inherently requires multi-goal and long-horizon RL. The biggest strength to me is that the problem and the environment is based on real need, not simulations. I also appreciate the explorations of the authors take on how to design the environment and how to make RL work in this context.", "weaknesses": "1. I am not a researcher on fixed-wing UAV control and multi-goal long-horizon problem. The biggest problem for me is that while the authors present an environment for future researchers, they failed to give a clear guidance on how to use this environment for benchmark. For example, as a user of the environment, I want to know:\n\n* What are the number of tasks in this environment, and what are their difficulties, respectively? Seems to me there is only one task. In this way, users might not use this environment in their study since it is likely to be rejected by other reviewers by the reason like \"small amount of experiments\".\n\n* What are the recommended configuration, hyperparameters and algorithms for me to start with, for researchers in the field of (1) GCRL (2) demonstration-based RL and (3) fixed-wing UAV control?\n\n* As for demonstrations, if I want to generate trajectories to support my own task, do I have tools for that in the environment? Are there standardized demonstrations recommended by the authors?\n\n2. If conventional algorithms such as PID/MPC can already solve the task, why should we use RL for this problem? Maybe we can significantly reduce the complexity of the problem by planning?\n\n3. After searching on google, I find some repo doing similar tasks like this: https://github.com/liuqh16/LAG, which is also based on fixed wings. Can you also discuss existing simulators of fixed-wing UAV control?\n\n4. Some minors: in page 7, ... with the results presented in Fig. 2b, which should be Table. 2b. Please also check others.\n\nTo sum up, as RL researchers, we always need real-world environments for simulations and I appreciate the large amount of work by authors to develop this environment and make it work. However, I'm not sure how to use the environment without pain. Maybe the authors could revise the paper by giving more recommendations to users?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)