OrbitZoo: Real Orbital Systems Challenges for Reinforcement Learning
Authors: Alexandre Oliveira
Katarina Dyreby
Francisco Miguel Caldas
Claudia Soares
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Agent Reinforcement Learning
Orbital Dynamics
Data Generation
Primary area: reinforcement_learning
TL;DR: We introduce OrbitZoo, a multi-agent RL environment designed for orbital dynamics, enabling scalable and flexible simulations for autonomous satellite operations with robust and accurate dynamics.

Abstract: The increasing number of satellites and orbital debris has made space congestion a critical issue, threatening satellite safety and sustainability. Challenges such as collision avoidance, station-keeping, and orbital maneuvering require advanced techniques to handle dynamic uncertainties and multi-agent interactions. Reinforcement learning (RL) has shown promise in this domain, enabling adaptive, autonomous policies for space operations; however, many existing RL frameworks rely on custom-built environments developed from scratch, which often use simplified models and require significant time to implement and validate the orbital dynamics, limiting their ability to fully capture real-world complexities.  To address this, we introduce OrbitZoo, a versatile multi-agent RL environment built on a high-fidelity industry standard library, that enables realistic data generation, supports scenarios like collision avoidance and cooperative maneuvers, and ensures robust and accurate orbital dynamics. The environment is validated against various real satellite constellations, including Starlink, achieving a Mean Absolute Percentage Error (MAPE) of 0.16% compared to real-world data. This validation ensures reliability for generating high-fidelity simulations and enabling autonomous and independent satellite operations. This project is open source and has a dedicated project page.

OrbitZoo: Real Orbital Systems Challenges for
Reinforcement Learning
Alexandre Oliveira
Katarina Dyreby
NOVA University of Lisbon Caparica, Almada, Portugal
NOVA University of Lisbon Caparica, Almada, Portugal
aan. oli oliveira@campus. fct. unl. pt
k. dyyebyycampusu...... fct. fct.unl...
Francisco Caldas
Cláudia Soares
NOVA University of Lisbon Caparica, Almada, Portugal
NOVA University of Lisbon Caparica, Almada, Portugal cam soares0fct. unl.
f. caldas@campus fct. act.unl.pp unl.pt
Abstract
The increasing number of satellites and orbital debris has made space congestion critical issue, threatening satellite safety and sustainability. Challenges such collision avoidance, station-keeping, and orbital mancuvering require advanced techniques handle dynamic uncertainties and multi-agent interactions. Rein- forcement learning (RL) has shown promise in this domain, enabling adaptive, autonomous policies for space operations; however, many existing RL frameworks rely on custom-buit built environments developed from scratch, which often use sim- plified models and require significant time implement and validate the orbital dynamics, limiting their ability to fully capture real-world complexities. To address this, we introduce OrbitZoo, versatile multi-agent RI environment built high- fidelity industry standard library, that enables realistic data generation, supports scenarios like collision avoidance and cooperative maneuvers, and ensures robust and accurate orbital dynamics. The environment is validated against various real satellite constellations, including Starlink, achieving Mean Absolute Percentage Error (MAPE) of 0.16% compared to real-world data. This validation ensures reliability for generating high-fidelit simulations and enabling autonomous and independent satellite operations. This project is open source and has a dedicated project ?
Introduction
Since the dawn of the space age in 1957, humanity has successfully launched approximately 20,000 satellites into Earth's 's orbit (23], of which only about 50% remain operational. These satellites are crucial for our daily life, providing critical services such as global communication, navigation, weather forecasting, Earth observation, and scientific research. However, these advances come with many problems.
Earth's orbital environment hosts an estimated 140 million debris objects, with approximately million of these being larger than cm - large and fast enough to cause catastrophic damage upon impact (22). Collisions with space debris generate more debris, leading to further collisions. This chain-reaction known as Kessler syndrome, results in an exponential increase in debris, threatening
http:///iihbb.ca//oobttto//oabar httpe://orbizoo.gittbb.b.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the long-term sustainability of Earth's ' orbits [39]] If no further measures are taken to address this issue, Earth's 's orbits could become unusable.
More recently, the congestion Lower Earth Orbit (LFF) is likely to undergo a major change of scale for megaaconstellltionn of telecommunication satellites, leveraging tens of thousands of satellites [2]. The increasing density of satellites and debris, particularly in LEO (IT) presents formidable challenges for Space Traffic Management (STM), related to the accurate monitoring and tracking of space objects and development of adequate decision-support frameworks [47) compounded by issues of data scarcity and uncertainty. Despite the growing availability of orbital data and tools, current solutions for satellite maneuvers ranging from orbital transfers to collision avoidance remain heavily reliant on manual processes. Human experts must navigate an ever-increasing complexity of scenarios, making critical decisions under time constraints with incomplete or inaccurate information 26 As the volume of satellites and orbital debris continues rising, these traditional methods are quickly becoming unsustainable, which demands the development of new strategies to minimize our impact, such as debris removal 7 and faster, more capable, and autonomous intelligent systems for decision-making. Adaptive control systems [4] provide robustness to uncertainty but struggle with complex, dynamic, and nonlinear space systems. RL, on the other hand, excels in real-time adaptation to such environments (77].
We examine how RL applications orbital dynamics are shaped by two key factors and limited by a lack of standardization. First, the choice of duta-generation tools affects the trade-off between simplicity and realism. Second, RL mission frameworks are often built from scratch, requiring extensive validation of orbital dynamics. To address these challenges, we introduce OrbitZoo, an environment that combines fast, high-fidelity orbital data generation with RL. development. The contributions of this paper fall into three categories:
Data Generation: Built on Python and with a robust space dynamics library on its back- ground, OrbitZoo generates high fighlfidliity orbital data by incorporating realistic forces and perturbations, providing accurate datasets essential for machine learning and strategy valida- tion, while making use of parallel computations for fast propagation;
Reinforcement Learning: OrbitZoo is standardized for RL research, leveraging the Petting- Zoo library 76 to support multi-agent RI. (MARL) with a Partially Observable Markov Decision Process (POMDP) structure, while also making use of the parallelism provided by the space dynamics library to properly scale to systems with thousands of bodies. This integration enables the development, training, and benchmarking of intelligent satellite maneuvering strategies for single and multi-agent missions in cooperative, competitive, or mixed scenarios;
Customizable Framework with Visualization: OrbitZoo's ''s modular design allows users to define scenarios with an arbitrary number of bodies with or without thrusting capabilities, incorporate custom models, and adapt the environment to specific needs, with clear sepa- ration of abstraction levels. It also features interactive 3D visual component, making it versatile for a wide range of applications while providing an accessible way to understand orbital behaviors and decision-making processes.
2
Related Work
Existing efforts in this domain can be broadly categorized into three areas: high-fidelity tools for generating and simulating orbital dynamics data, RL for orbital maneuvering, and MARL environments for orbital dynamics.
one of the most comprehensive open-source libraries for astrodynamics and orbital mechanics, offering advanced capabilities for precise orbit determination [46] (58 I6), propagation and associated uncertainties [6]. attitude determination, and trajectory analysis 19 Its modular, Java- based architecture allows users to model systems ranging from simple Newtonian attraction to highly complex scenarios incorporating detailed gravity fields and perturbations, such as atmospheric drag, solar radiation pressure (SRP), and third-body effects. While versatile, Orekit's 's steep learning curve and reliance on Java can pose challenges for users unfamiliar with its technicalities.
Data Generation and Simulation Tools. Orekit 45

---

Python-natinn native library, offers a more accessible alternative, featuring tools for orbit propagation and transfer planning. Although it lacks the high- high-fidelity modeling of Orekit, Poliastro integrates with the Cesium library [17 for accurate 3D geospatial visualization, making it suitable for simpler scenarios or users with limited expertise in orbital mechanics. Systems Tool Kit (STK) [3). the leading commercial orbital simulation software, offers extensive capabilities for mission planning and operational analysis. Despite its advanced visualization features and high precision, STK's substantial cost restricts its accessibility to well-funded organizations.
Poliastro 62],
Reinforcement I Learning for Orbital Dynamics. RL has been widely applied to satellite ma- neuvering, often relying custom-buill environments and simplified dynamical models. Many studies use the Circular Restricted Strce-Body Problem (CR3BP) [49] 24 73 [1 for tractable simulations, though it falls short in capturing real-world perturbations, making challenging for non-experts to extend RL to realistic space missions. For LEO station-keeping, [31 implemented Proximal Policy Optimization (PPO) [69] with 4th-order Yoshida integrator, modeling gravitational forces and atmospheric drag, while 75 used Soft Actor-Critic (SAC) (28 with dynamics developed from scratch that also included body forces and SRP.
Orekit has also been employed RL frameworks with more accurate dynamies. 40) for instance, used Orekit and Deep Deterministic Policy Gradient (DDPG) 43) for low-thrust transfers. In geostationary orbit (GEO), [3] used Orekit and a synchronous variant of A3C (50] for a perigee- raising maneuver. Despite these contributions, all relied simplified or mission-specific physics without multi-agent or uncertainty support, limiting their generalization to more complex scenarios.
Collision Avoidance Maneuvers (CAMs) have received growing attention. 38 introduced ColAv- Gym, a single-agent CAM environment in LEO using Orekit and PPO. It leverages real Conjunction Data Messages (CDMs) to retroactively reconstruct satellite and debris orbits for training. In con- trast, 72 and [3] employed synthetic debris scenarios Gym-based environments incorporating Orekit and an adaptation of SpaceNav [27]. These studies investigated different observation and reward strategies, modeling the problem as a POMDP with temporal features captured by Long Short- Term Memory (LSTM) [22] layers in the Deep Q-Network (DQN) [51] architecture, showcasing how RL algorithms with discrete action spaces and recurrent networks can be useful in such scenarios.
Multi-Agent Reinforcement Learning Environments. MARL has recently gained traction in orbital dynamics, with applications in satellite coordination and task assignment, although with limited focus maneuvers. (34) introduced the RL-Enabled Distributed Assignment (REDA) algo- rithm, using Poliastro and DQN for task allocation across satellite constellations. However, REDA lacked integration with realistic orbital data. Similarly, [84] applied Multi-Agent PPO (MAPPO) 83 for multi-satellite observation planning, combining STK and MATLAB for a custom simula- tion framework but without leveraging industry-standde astrodynamics libraries like Orekit. [60] developed a MARL framework for cooperative formation flying of multiple spacecraft considering only Newtonian attraction, while (20) investigated multi-agent CAM using framework (54) that aggregates local neighborhood information through a graph neural network (GNN) (64), considering two dynamic models [6] 80 that did not account for SRP and third-body forces.
3 Background: Challenges in Multi-Agent RL for Orbital Dynamies
A major challenge in applying RL to orbital dynamics lies in bridging the Reality Gap between simulation and real-world operations. This requires incorporating high-fidelity dynamics, accounting for uncertainties, and validating simulations against real-world data. According to what was discussed in Sec. [ the essential capabilities an orbital RL environment include: support for multi-agent in- teractions (both cooperative and competitive scenarios), integration with industry- staanard simulators such as Orekit or Poliastro (rather than custom- built models), high-fidelity dynamics with gravity fields and perturbations, support for algorithms with continuous control, realistic body and thrust modeling (capturing distinct spacecraft characteristics and continuous thrust integration), interactive real-time visualization, and public code availability.
As shown in Tab. | OrbitZoo distinguishes itself by containing all the aforementioned capabilities. Unlike previous environments based on simplified models, OrbitZoo incorporates multiple pertur- bative forces, supports Cartesian, Keplerian, and equinoctial representations, enables realistie body and thrust modeling through diverse integration methods, propagates uncertainty, and offers several
3

---

Table 1: Comparison of RL environments for orbital dynamics.
Multi Multi-Agent
Industry Standard
High- Fidelity Dynamics
Continoous Control
Realistic Bodies and Thrust
Interactive Visualization
Publicly Available
RL
Sianulator
Kolosa (2019) 4D
Miller (2019)
Herren (2020)
3
V (drag only)
Federici (2021)
Sullivan (2021)
Casas (2022) [6
Bonasera (2022)
||
Dolan (2023) 20
<02 only)
Bourriez (2023)
LaFarge (2023)
41
V (third bodies only)
Zhang (2023)
Qiagyu (2023) 60
Holder (2024) X1
Solomon (2024)
(2024)
OrbitZoo (ours)
4
methods to assist in the development of RL missions or multi-agent systems analysis. Its publicly available nature further establishes it as comprehensive platform for advancing learning-basd -aasen space operations.
3.1 Multi-Agent Reinforcement Learning
RL provides framework for decision-making under uncertainty, where agents learn through trial and error to maximize cumulative rewards [74] [70]. In the context of MARL, multiple agents operate in shared environment, each pursuing individual collective goals. MARL presents unique challenges in orbital dynamics, where environments are partially observable, highly dynamic, and governed by complex physical interactions.
MARL builds upon Markov Decision Processes (MDPs) 74 10 and Partially Observable Markov
Decision Processes (POMDPs), commonly used model scenarios where agents cannot access the full environment state 337 A POMDP is defined by the tuple (S, A, P, R, O, Q). At each timestep f, agent observes o, O based the current state 8, S, selects action a, A, and transitions to new state S141 with probability P(s [8, az), receiving reward R(s,, az). The goal is to learn policy (a,loo that maximizes expected returns over time.
The introduction of Deep Q-Networks (DQN) 51 revolutionized RL by combining deep function approximation with Q Q-learning making possible to tackle discrete decision becision-making tasks from raw, high-dimensional observations. This success established the foundation for subsequent methods that aimed generalize deep RL to continuous domains. Deep RL algorithms such DDPG 43 and PPO 669 have been instrumental this extension, albeit the topic had been explored carlier 77 In MARL, the interaction between agents is often modeled as multi-agent POMDP (MA-POMDP), where each agent observes of, performs an action at, and receives a reward r;. PPO and variants, such MAPPO { well-suited for these scenarios, allowing stable decentralized centralized training of agents in shared environments. This is particularly important for missions involving constellations of satellites, where each satellite may need to coordinate, compete, or cooperate with others to achieve shared objectives in complex - and potentially noisy - orbital dynamics.
3.2 Challenges in Bridging the Reality Gap
Simulating orbital dynamics presents a substantial reality gap, driven by the need for high-fidelity physical modeling and the incorporation of real-world uncertainties. This section highlights the main challenges of applying RI. to orbital dynamics, whose understanding relies on the fundamental concepts presented in Appendix B

---

Complex Orbital Dynamies. Orbital dynamics follow Newton's laws of motion and gravitation, requiring accurate modeling of perturbative forces such as atmospheric drag, SRP. and third-body effects. Simplified propagators like the Simplified General Perturbations (SGP) model are com- putationally efficient but lack the precision required for RI tasks. In contrast, Orekit's numerical propagators [45] provide high-fidelity simulations by incorporating complex perturbations, enabling realistic long-term trajectory prediction and maneuver planning. The choice of integration method in numerical propagation also affects realism: classical approaches such Gauss' variational equations (GVEs) and fixed-step integrators like the 4th-order Yoshida or RK4 [4] are commonly used for orbital propagation due to their simplicity and efficiency, but they lack the adaptive precision of higher-order variable-step schemes such as Normand-Prined
Coordinate Systems and State Representations. Orbital states commonly represented using Keplerian elements, but these suffer from singularities in cases like circular or equatorial orbits. Equinoctial elements avoid such issues and are often preferred in RL applications for their robustness. However, Cartesian coordinates remain essential not only for numerical integration but also for tasks like computing inter-body distances collision probabilities. An RL framework should therefore support all major representations. multi-agent settings, maintaining consistent reference frames and relative state representations particularly challenging, as agents may operate in different orbital regimes.
Thrust Modeling and Control. Thrust actions often defined in local reference frames like RSW (Radial, Along-track, Cross-track). A polar parametrization using thrust magnitude, deviation angle, and azimuthal angle provides realistic and constrained action space, offering flexibility in modeling satellite maneuvers. This representation can be converted to the RSW frame for compatibility with numerical propagators. In multi-agent settings, additional challenges arise each spacecraft may have distinct thrust capabilities, fuel constraints, and control dynamics, making coordinated maneuver planning and learning more complex.
Exploration in High-Dimensione Action Spaces. Exploration in continuous action spaces, such those encountered in orbital dynamics, presents significant challenges, particularly in multi-agent settings where agents may operate in joint state and action spaces with cooperative or adversarial interactions. To address this complexity and the wide range of possible learning objectives, it is essential to design a modular framework that supports both on- and off-policy RL methods, as well as discrete and continuous action spaces. Such flexibility enables consistent experimentation and comparison across diverse mission scenarios. Moreover, having the capability to handle multi- objective RL (MORL) (30 is important, as spacecraft missions often involve trade-off's (c.g.. fuel vs. time vs. risk) that must be balanced within the policy learning process while coordinating multiple agents.
Ephemeris Data and Validation. Ephemerides offer time-stamped predictions of orbital posi- tions and velocities, enabling validation of simulated trajectories. For instance, Orekit's numerical propagator demonstrates high accuracy when compared with Starlink satellite ephemeris data. By tuning physical parameters such as drag and reflection coefficients the simulation closely matches real-world behavior, helping bridge the reality gap in RL. applications.
3.3 Multi-Agent Coordination and Scalability
In MARL for orbital dynamics, agents operate in partially observable environments with decentralized policies. While MARI. algorithms commonly assume fully cooperative scenarios (Dec-POMDP), in reality agents often assume partial or fully competitive scenarios (MA-POMDP), In such cases, task
coordination, interference avoidance, and scaling to large constellations, that is, groups of satellites working together to accomplish shared objectives, become key challenges. MPYL [57] extends
PyMARL for flexible agent collaboration in MARL environments, while MARLlib [36 compatible with PettingZoo environments like OrbitZoo, supports a broad set of algorithms and architectures. Federated Learning (FL) and Federated RL (FRL) [59] further enable decentralized learning and coordination in both cooperative and competitive settings while maintaining information privacy. Nonetheless, challenges inherent to the space environment such competition, communication constraints, uncertainty, and heterogeneity between agents - make challenging to apply standard MARL algorithms, highlighting the need for new, tailored approaches.
5

---

Table 2: Summary of reinforcement learning challenges and how OrbitZoo addresses them.
Research Gap
Linitations of Standard
Enabled by OrbitZoo
Environments
Sim-to-Real Transfer
Neglects real-world orbital perturbations and environ- mental uncertainties.
Realistic dynamics with harmonic grav- ity fields, drag, SRP, third-body forces, uncertainties and variable-step integra- tion methods.
Continuous Control
Impulsive unconstrained control models.
T'mee-continouss thrust with fuel and tuation constraints.
Multi-Agent Coordination
Mostly single-agent, fully observable setups.
Decentralized agents with partial observ- ability via PettingZoo.
Reward Design
Dense, artificial reward sig- nals.
Flexible, physically grounded reward functions.
Safety and Learning
Adversarial Rarely models safety or ad- versarial dynamics.
Supports pursuit-evasion and safe neuvering tasks.
Visualization
Limited (2D) or no visual- ization for debugging.
Real-time 3D inspection of agent behav- ior.
Recent solutions leverage C++ engines 85 or CPU/GPU parallelization for faster training [3]
and propagation 42 (see Appendix B.3). Scalability remains challenging when using high-fdelity numerical propagation, each body experiences distinct forces that evolve continuously over time.
3.4 Realistic Simulation Environments
Realistic orbital dynamics environments should integrate (1) high-fidelity dynamics that account for the most relevant forces and perturbations (e.g., gravity fields, drag, SRP, third-body effects), (2) body and thruster specific characteristics, including shape and dimensions, reflection and drag coefficients, and specific impulse, (3) flexible state and action spaces, as different missions may require distinct representations, (3) multi-agent reinforcement learning and federated reinforcement learning capabilities, (4) scalability and parallelization to handle an increasing number of bodies, and (5) reproducibility and extensibility. However, most existing MARL environments for orbital dynamics do not fully integrate these capabilities (Tab.
Beyond the orbital domain, similar challenges have been addressed in other fields where partial observability, decentralized control, and coordination are critical. In energy systems, MARL has been used optimize distributed generation and communication under uncertain and partially observable states [56]. In autonomous robotics, macro- actionn (joint policies) and communication-avee aware learning frameworks have been proposed to improve coordination under local observation constraints [8] 55). Similarly, unmanned aerial vehicle (UAV) swarm control has used graph-based methods to address scalability and partial observability large-scale UAVs coordination and confrontation tasks [82].
These efforts highlight the growing importance of standardized, high-fidelity benchmarks across domains.
Novel Reinforcement Learning Challenges in OrbitZoo
In the emerging era of autonomous systems operating the physical world, RL is evolving beyond synthetic benchmarks and addressing real-worke constraints such as actuation limits, uncertainty. safety, and coordination. OrbitZoo is designed to support this shift. It provides a modular testbed to rigorously investigate RL challenges grounded in high-fidelity dynamics while abstracting just enough enable algorithmic insight. Tab. 2 summarizes the main RL challenges addressed by OrbitZoo, which are discussed in detail below.
Sim-to-Real Transfer and Grounded Dynamics. Sim-to-real gaps constitute a significant barrier in deploying RL policies in physical systems. While Orckit handles high-fidelity propagation,

---

OrbitZoo provides the interfaces needed to test RL algorithms in scenarios with uncertain initial conditions, realistic gravity fields historical weather data), SRP (accounting for the occlusion of the Sun by the Moon), and third-body forces (from all planets the solar system, as well as the Sun, Moon, and the barycenters of the Farth-Moon system and the solar system). Validation against Starlink ephemerides (Sec. E) shows that these perturbations can be tuned to match real trajectories, allowing research on robust policy learning under realistic dynamics.
harmonies (35)) drag (computed from
Continuous Control and Thrust Constraints. OrbitZoo supports physically plausible, time- continuous control through polar thrust parameterizzatons and constant-magniuude thrusts. Unlike many benchmarks that assume instantaneous or impulsive maneuvers, OrbitZoo models thrust as continuous acceleration integrated over time, constrained by available fuel and propulsion limits. This formulation forces agents to learn realistic, sustained maneuvers, making the environment well suited for studying constrained RL and planning under resource limitations. Classical maneuvers with simplified dynamics and impulsive actions can still be simulated in OrbitZoo (Appendix [.4).
Multi-Agent Coordination with Partial Observability. Real-world orbital operations often in- volve multiple autonomous spacecraft interacting under limited information for example, in forma- tion flying, debris tracking, or pursuit/evasion scenarios. Each agent must make decisions based only on local or delayed observations, while the overall behavior emerges from their collective interaction. OrbitZoo enables such decentralized and partially observable setups through its PettingZoo-base based architecture, where each agent receives its own observation and action space while sharing the same dynamic environment.
This design enables the study of cooperative and competitive MARL paradigms. Cooperative tasks include coordinated thrusting for formation maintenance or rendezvous, while adversarial tasks involve pursuit, evasion, or interference. OrbitZoo also supports centralized training with decentralized execution (CTDE), where global critic accesses full system information during learning, but agents act autonomously at runtime. With support for parallel trajectory collection, OrbitZoo offers a scalable testbed for studying coordination, communication, and robustness in multi-agent orbital control - areas largely unexplored in standard RL benchmarks.
Reward Shaping under Sparse and Delayed Feedback. Designing informative yet physically grounded reward signals a central challenge in orbital control. Rewards must capture long-term mission objectives while remaining interpretable within physical constraints. OrbitZoo provides flexible reward framework that integrates both inter-body and body-specific metrics, enabling the formulation of rich and diverse learning objectives. Inter-body quantities such relative distance or line-of-sight conditions - allow the definition of cooperative or competitive multi-agent tasks. In parallel, each body exposes individual physical attributes such fuel consumption and mass variation, which can be incorporated into specifc reward functions.
This modular reward interface enables experimentation across dense and sparse regimes, as well as multi-objective trade-offs between performance, safety, and efficiency. supports studies on credit assignment, reward misspecification, and curriculum learning (53) in complex multi-agent orbital environments, where delayed effects and coupled dynamics make the reward landscape inherently nontrivial.
Adversarial and Safety-Critical Learning. OrbitZoo's modular setup supports adversarial and safety-critical scenarios, such as pursuit- evasion, intentional jamming, or collision avoidance. The environment includes inter-body metries like the probability of collision (POC) and body-specific measures of propagation uncertainty, allowing agents to quantify and respond to risk in real time. Combined with partial observability and fuel-aware dynamics, these features enable research on robust policy design, safe exploration, and adversarial resilience in continuous orbital domains.
Visualization and Debugging. OrbitZoo offers real-time, integrated 3D visualization tool that aids policy inspection and failure case diagnosis. Although not a core contribution, this represents, to the best of our knowledge, the first Python- RL framework with a built-in, real-time orbital visualization interface, as summarized in Appendix [ This feature enhances interpretability. repro- ducibility, and debugging by providing an intuitive link between learned behaviors and the underlying orbital dynamics.
7

---

5 Experiments and Results
To evaluate the effectiveness of OrbitZoo in modeling diverse orbital missions and supporting RI. methods, we conducted a series of experiments using several deep learning algorithms (i.c., DQN, DDPG and PPO), as well FRL applied PPO in multi-agent setting. These include the devel- opment of a mission analogous to one created in different environment, orbital transfers (OTs), collision avoidance maneuvers (CAMs), multi-agent geostationary orbit (GEO) constellation coor- dination problem using independent and federated learning, and validation experiment comparing OrbitZoo's 's simulations against real-world Starlink ephemeris data. In the following missions, we consider an RL agent as satellite with limited maneuvering capabilities, learning strategy (policy) to apply thrusts that maximize the expected return (Sec. 3.1) Details can be found Appendix E
5.1 Single-Agent Hohmann Maneuver
The Hohmann transfer experiment was chosen as a benchmark to evaluate OrbitZoo's ability to support RL in high-fidelity orbital dynamics environment. Details of the experiment can be found in Appendix [E.4] As classic problem in orbital mechanics, the Hohmann transfer is analyti- cally solvable and provides clear reference for comparing RL-derived solutions with theoretical optima
33
Setup. The agent observes the satellite's ' current orbital state through equinoctial coordinates and receives a reward based on the reduction of transfer error while minimizing fuel consumption. The action space corresponds to polar thrust parameters (T, 8, Φ), where T is the thrust magnitude and (0, Φ) define the thrust direction. The environment incorporates perturbative forces, including drag.
The RL agent learned near-optimal strategies for a 30 km altitude Hohmann transfer, matching theoretical semi-major axis values. Deviations in other clements (e.g.. inclination) arose from thrust misalignments. Contrary to the classical optimal maneuver, the agent adapted to new perturbations and reached the target successfully. These results validate OrbitZoo's ability to model classical problems in complex orbital dynamies and highlight areas for refining policy and reward design. Figure 17 shows the optimized transfer with minimal fuel use. The trajectory closely matches the theoretical solution (Figure I.. confirming OrbitZoo's high- figh-fidlity simulation.
Results.
5.2 Single-Agent Collision Avoidance Maneuver
CAM missions in RL are often focused on -eeppng while preventing collisions with other objects. Many existing approaches assume perfect knowledge of the current state of all bodies in the enviromment, simplifying the task to maximizing the Euclidean distance from other bodies while maintaining proximity to nominal orbit. In reality, however, the states of these bodies are known with uncertainty, which must be accounted for in operational planning. In this mission, a more realistic approach to CAM is adopted by explicitly modeling state uncertainty and its evolution over time. While the environment handles propagation using high-fidelity dynamics that closely resemble real-world physics, the prediction of future states and the corresponding probability of collision (POC) relies on a simplified Newtonian attraction model, representing a low-fidelity simulation.
Setup. Two bodies - debris and satellite with maneuvering capabilities are synthetically instantiated in the same Cartesian position with some uncertainty, and with symmetric Cartesian velocity, creating short-term encounter at that moment. By propagating both bodies backward in time, we create the initial conditions for the start of each episode, enough for the satellite to perform the needed maneuvers and ultimately avoiding collision. Due to the uncertainty, POC >> 10-6 is considered as high collision risk, and the agent is tasked at lowering while staying near its initial orbit.
Results. In many existing CAM approaches, the RL agent is trained using discrete action spaces, since maneuvers are only required when a collision risk exists. Accordingly, DQN was employed for training the agent. To investigate whether applying thrust in specific directions could lead to improved performance, the continuous version of PPO was also implemented. In this setup. the action space included a decision variable (0 S 8 S 1) indicating whether thrust should be applied > 0.5). For training, the environment used Newtonian attraction with atmospheric drag to model the actual
8

---

dynamics. During evaluation, however, all available perturbations in OrbitZoo were enabled to assess the generalization capability of the trained agents. Results show that both algorithms learned effective policies under the (simplified) training dynamics, but PPO demonstrated superior performance under realistic conditions, exhibiting greater ability reduce the POC. More details are presented in Appendix E.6
5.3 Multi-Agent GEO Constellation Coordination
This experiment demonstrates OrbitZoo's s ability to simulate and train multi-agent systems for satellite constellation management in geostationary orbit (GEO), using both independent and federated learning. More details can be found Appendix E.7
Setup. A four-satellite constellation in GEO aimed to maintain equal angular separation and altitude while minimizing fuel use through small thrusts limited to the GEO orbital plane (ø parameterization). Each agent observes the anomalies of other satellites, its own semi-major axis and eccentricity, and outputs thrust commands via decentralized policy. Training used PPO with generalized advantage estimation (GAE) [68]. At the start of each episode, the satellites are placed at random position along the GEO orbit.
polar
Results. Figure 2 shows a view of the OrbitZoo interface of the constellation's configuration after days for given initial configuration. Generally, the agents successfully obtained a larger angular separation while not drifting away from GEO. The policies are also able generalize to unseen perturbations, such as third body forces (Sun and Moon), SRP and drag. However, the constellation not able to increase the overall angular separation within the given time in some configurations.
X10
X20
agent
agent
agent
agent
agent
agintt
5.0
-7
Distance
0.0
agent
agent 4
-25
L2
day agent_z
days
5
agent 2
-5.0
-75
-10
-5
x [m)
x30*
Figure 1: The L. error between the optimal and Experiment maneuvers stays low over long bits, with most error due to minor inclination shifts. However, agents were trained minimize equinoc- tial element differences, not Euclidean distance.
ägent
agent
days
days
Figure 2: Constellation evolution over 4 days (seed 44). The purple circle represents GEO.
5.4 Validation Against Real-World Data
To evaluate OrbitZoo's ability to bridge the reality gap, we configured the environment to simulate Starlink satellites and compared its output publicly available ephemeris data in Space-Traae Further details on the analysis can found Appendix
Setup. Thirty-one Starlink satellites with publicly available data were selected for the validation
experiment. Bayesian optimization was employed to tune parameters such as the drag coefficient, reflection coefficient, and satellite radius to match the ephemeris data. The RMSE between the DobitZZoo-proppgaeed trajectories and the ephemeris data was used as the primary metric for evalua- tion.

---

Results. As summarized in Table 3] OrbitZoo achieved varying levels of accuracy across 31 satellites, with mean RMSE values ranging from 24.14 meters 1924.90 meters over 16.6-hour propagation. While some satellites closely matched the Starlink ephemeris data, others showed significant deviations, likely due to limited information on their physical properties. Figure 3 illustrates the residuals between the propagated and observed trajectories, demonstrating the accuracy of OrbitZoo's physical modeling for the relevant horizon of two hours.
Group
Mean RMSE (meters)
Low RMSE
24.14 14
Orekit vs Ephemeris (1000 steps)
Medium RMSE
83.75
0.20
High RMSE
1924.90
Residual (km)
0.15
Table 3:
Root mean square error (RMSE) values, in meters, for three groups of
0.10
satellites (a total of 31 satellites), sepa- rated based on an RMSE threshold. The "Low RMSE" group includes satellites with RMSE values below 50, the "Medium RMSE" group includes satellites with val- ues between 50 and 100, and the "High Figure 3: Residuals between OrbitZoo proroooo-r tra-
0.00
1.0
12
14
16
(hours)
RMSE" group is for satellites above 100. jectories and Starlink ephemeris data for satellite 44744 The estimated error is derived from 16.6 in blue. The residuals remain low the validation interval, confirming OrbitZoo's high fidelity.
hour propagation (1000 steps).
5.5 Additional Experiments
Many RL missions and environments developed by other authors (as shown in Table are not publicly available, hindering reproducibility and making challenging assess the complexity of their implementation. Among the few exceptions is the work by D. Kolosa [40 and A. Herrera [3], whose code is openly accessible and supports the results presented. In this context, we replicate both authors first missions in Appendix E.2 and Appendix E.3] respectively, to illustrate how OrbitZoo provides flexible framework for developing and analyzing such missions. For completeness, we present an additional experiment: chase-target scenario (Appendix E.5 where satellite pursues a moving object in a higher orbit. For each mission, we provide a brief overview, followed by the experimental setup, results, key challenges, and directions for future improvement.
Conclusions
Through comparisons with existing environments and diverse of experiments including Hohmann transfer maneuver, geostationary constellation coordination task, and validation using real-world Starlink ephemeris data we demonstrated that RL policies trained OrbitZoo can achieve near-optimal control while remaining consistent with realistic satellite behavior. These results establish OrbitZoo as a reliable benchmark for RL-based autonomy in space operations. To assess generalization and robustness, realism, highlighting how continuous action spaces generally enhance performance under realistic orbital dynamics. Beyond RL, OrbitZoo serves as publicly available platform designed to support researchers aerospace engineering, satellite operations, and machine learning. It provides a modular, fidelityyytty environment for studying RL in realistic orbital settings. While grounded in orbital dynamics, its abstractions and challenge structures are representative of broader autonomy domains. The platform fosters reproducible experimentation and enables the exploration of autonomous decision-making for future space applications.
employed multiple RL algorithms and different levels of
Acknowledgments. We thank the anonymous reviewers for their valuable input and for helping strengthen the manuscript. This work was partially supported by NOVA LINCS (UID/04516) funded by FCT IP. and the Neuraspace AI Fights Space Debris project (project code 0626449889-004666 operation code 2022-00510101-02,, co-funded by Recovery and Resilience Plan and NextGeneration
EU Funds, www.reuupeaapprttgalgg.gg
10

---

References
[1] Tim Abilov. Basic 3D world playground with animations based on 2D. 2021. GitHub repository.
[2] Maruthi R. Akella and Kyle Alfriend. Probability of collision between space objects. Journal of Guidance, Control, and Dynamics, 23(5)766-777,, 2000.
[3] Marcin Andrychowicz, Anton Raichuk, Piotr Stanezyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters in on-policy reinforcement learning? A large-scale empirical study. In Proceedings of the 9th International Conference on Learning Representations (ICLR), Vienna Virtual, Austria, 2021.
[4] A.M. Annaswamy and D.J. Clancy. Adaptive control strategies for flexible space structures. IEEE Transactions on Aerospace and Electronic Systems, 32(3):55--965 1996.
[5] Inc. Ansys. Ansys STK: Systems tool kit for mission modeling and analysis. https: www ansys. mangsss.coo//prdddttt/////iiii// wwtpsssiiiinnnsssssssssss Accessed: 2025-01-16.
[6] Andrea Antolino and Luc Maisonobe. Automatic differentiation for propagation of orbit uncertainties on Orekit. Presented at Stardust Final Conference, 2016.
[7] Muneeb Arshad, Michael C.F. Bazzocchi, and Faraz Hussain. Emerging strategies in close proximity operations for space debris removal: A review. Acta Astronautica, 228:996-1022, 2025.
[8] Nikhil Barhate. Minimal pytorch implementation of proximal policy optimization. //githb.. www.://wihhbbbaaattttee/////.-..
https
2021.
[9] Pierre Bernhard, Mare Deschamps, and Georges Zaccour. Large satellite constellations and space debris: Exploratory analysis of strategic management of the space commons. European Journal of Operational Research, 304(3):1 140-1117,, 2023.
[10] Dimitri Bertsekas. Reinforcement learning and optimal control. Athena Scientific, 2nd edition, 2019.
[11] A.C. Boley and M. Byers. Satellite mogtl-constellations create risks in Low Earth Orbit, the atmosphere and on Earth. Scientific Reports, 11(1):10642, 2021.
[12] Stefano Bonasera, Natasha Bosanac, Christopher J. Sullivan, Ian Elliott, Nisar Ahmed, and Jay W. MeMahon. Designing sun-earth 12 halo orbit stationkeeping maneuvers via reinforce- ment learning. Journal of Guidance, Control, and Dynamics, 46(2)300-331, 2023.
[13] Nicolas Bourriez, Adrien Loizeau, and Adam F. Abdin. Spacecraft autonomous decision- planning for collision avoidance: reinforcement learning approach, 2023.
[14] J.C. Butcher. A history of runge-kutta methods. Applied Numerical Mathematics, 20(3):247- 260, 1996.
[15] Carlos M. Casas, Belen Carro, and Antonio Sanchez-Esguevillas. Low-thrust orbital transfer using dynamics-agnostic reinforcement learning, 2022.
[16] Bryan Cazabonne, Julie Bayard, Maxime Journot, and Paul Cefola. A semi-analytical approach for orbit determination based on Extended Kalman Filter. 08 2021.
[17] Cesium. Cesium: Build the open metaverse with cesium. https: fttpp::/cesium.com Accessed: 2025-01-16.
[18] Hung-Tu Tu Chen. Deep Q-Learning network in Pytorch. httpp://giit //ttthu///ttubbb.. htt.uu..commuuugtuuchnn pytorch-dqn 2017.
[19] Vincent Cucchietti, Maxime Journot, and Pascal Parraud. Orbit and covariance interpola- tion/blending with orekit. 2023.

---

[20] Sydney Dolan, Siddharth Nayak, and Hamsa Balakrishnan. Satellite navigation and coordination with limited information sharing. In Nikolai Matni, Manfred Morari, and George Pappas. editors, Proceedings of The 5th Annual Learning for Dynamics and Control Conference, volume
211 of Proceedings of Machine Learning Research, pages 1058-1071. PMLR, 15-16 Jun 2023.
[21] J.R. Dormand and PJJ. Prince. A family of embedded runge-kutta formulae. Journal of
Computational and Applied Mathematics, 6(1):19-26, 1980.
[22] European Space Agency (ESA). Esa space environment report 2024, 2024. Accessed: 2025-01-
06.
[23] European Space Agency (ESA). Space debris by the numbers, 2025. Accessed: 2025-01-06.
[24] Lorenzo Federici, Andrea Scorsoglio, Alessandro Zavoli, Roberto Furfaro, al. Autonomous guidance for cislunar orbit transfers via reinforcement learning. In AAS/AIAA Astrodynamics Specialist Conference. American Astronautical Society Big Sky, Montana (Virtual), 2021.
[25] Ricardo Ferreira, Cláudia Soares, and Marta Guimarães. Probability of collision of satel- lites and space debris for short-term encounters: Rederivation and fast-to-compute upper and lower bounds. In Proceedings of the 74th International Astronautical Congress (IAC), Baku, Azerbaijan, 2023. IAC-23-A6.3.99345
[26] Tim Flohrer, Holger Krag, Klaus Merz, and Stijn Lemmens. CREAM ESA's Proposal for Collision Risk Estimation and Automated Mitigation. In S. Ryan, editor, Advanced Maui Optical and Space Surveillance Technologies Conference, page 57, September 2019.
[27] Leonid Gremyachikh, Dmitrii Dubov, Nikita Kazeev, Andrey Kulibaba, Andrey Skuratov, Anton Tereshkin, Andrey Ustyuzhanin, Lubov Shiryaeva. and Sergej Shishkin. Space navigator: a tool for the optimization of collision avoidance maneuvers, 2019.
[28] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1861-1870. PMLR, 10-15 Jul 2018.
[29] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI 16, page 2090-2100. AAAI Press, 2016
[30] Conor Hayes, Roxana Ráduleseu, Eugenio Bargiacchi, Johan Källström, Matthew Macfar- lane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, Enda Howley, Athirai A. Irissappane, Patrick Mannion, Ann Nowé, Gabriel Ramos, Marcello Restelli, Peter Vamplew, and Diederik M. Roijers. A practical guide multi-objective reinforcement learning and planning. Autonomous Agents and Mulli-Agent Systems, 36(1) (6)26, 2022.
[31] Armando Herrera III. Reinforcement learning environment for orbital keeping Master's thesis, The University of Texas Rio Grande Valley, 2020. ScholarWorks @ UTRGV.
[32] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 98):11335-180,, 1997.
[33] W. Hohmann, United States. National Aeronautics, and Space Administration. The Attain- ability of Heavenly Bodies. NASA technical translation. National Aeronautics and Space Administration, 1960.
[34] Joshua Holder, Natasha Jaques, and Mehran Mesbahi. Multi agent reinforcement learning for sequential satellite assignment problems, 2024.
[35] S. A. Holmes and W. E. Featherstone. A unified approach to the clenshaw summation and the recursive computation of very high degree and order normalised associated legendre functions. Journal of Geodesy, 76(5):279-20) 2002. Published on 2002/05/01.
12

---

[36] Siyi Hu, Yifan Zhong, Minquan Gao, Weixun Wang, Hao Dong, Xiaodan Liang, Zhihui Li, Xiaojun Chang. and Yaodong Yang. Marllib; A scalable and efficient multi-agent reinforcement learning library, 2023.
[37] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101((-22:99-134) 134,, 1998.
[38] Sajjad Kazemi, Nasser L. Azad, K Andrea Scott, Haroon B. Oqab, and George B. Dietrich. Satellite collision avoidance maneuver planning in low earth orbit using proximal policy optimization. In 2024 IEEE Congress on Evolutionary Computation (CEC), pages 1-9, 2024.
[39] Donald J Kessler, Nicholas L Johnson, JC Liou, and Mark Matney. The kessler syndrome: implications to future space operations. Advances in the Astronautical Sciences, 137(8)20010, 2010.
[40] Daniel S. Kolosa. A Reinforcement Learning Approach Spacecraft Trajectory Optimization. Ph.d. dissertation, Western Michigan University, 2019.
[41] Nicholas LaFarge, Kathleen Howell, and David Folta. Adaptive closed-loop maneuver planning for low-thrust spacecraft using reinforcement learning. Acta Astronautica, 211, 06 2023.
[42] Mathias Lechner, lianhao yin, Tim Seyde, Tsun-Hsuan Johnson Wang, Wei Xiao, Ramin Hasani, Joshua Rountree, and Danicla Rus. Gigastep one billion steps per second multi-agent reinforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 155-170. Curran Associates, Inc., 2023.
[43] Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. 2015.
[44] Airong Liu, Xiaoli Xu, Yongqing Xiong, and Shengxian Yu. Maneuver strategies of starlink satellite based on spacex-released ephemeris. Advances in Space Research, 74(7)33157-319, 2024.
[45] Maisonobe, Bryan Cazabonne, Romain Serra, Evan Ward, Maxime Journot, Sébastien Dinot, Guilhem Bonnefille, Thomas Neidhart, Lue Maisonobe, Clément Jonglez, Mark Rutten, yjeand, Julio Hernanz, lirw1984, Vincent Cucchietti, Andrew Goetz. Guiuux, gaetanpierre0, Lars Næs- bye Christensen, Alberto Fossà, jvalet, Alberto Ferrero, gabb5, liscju, Christopher Schank, Tanner Mills, plan3d, Vyom Yadav, Shiva Iyer, and Petrus Hyvönen. Cs-si/orekit: 12.2.1, December 2024.
[46] Luc Maisonobe, Pascal Parraud, Maxime Journot, and Albert Alcarraz-Gare Garcia. Precise Orbit Determination, an adaptable open-source implementation.
[47] Chiara Manfletti, Marta Guimarães, and Claudia Soares. Ai for space traffic management. Journal of Space Safety Engineering, 10(4):495-504, 2023.
[48] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Effiicee Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 1282. PMLR, 20-22 Apr 2017.
[49] Daniel Miller and Richard Linares. Low-thrust optimal control via reinforcement learning. In Proceedings of the 29th AAS/AIAA Space Flight Mechanics Meeting, Ka anapali, HI, USA, 01 2019.
[50] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1928-1937, New York, New York, USA, 20-22 Jun 2016. PMLR
13

---

[51] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing Atari with deep reinforcement learning. 2013.
[52] Volodymyr Mnih. Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement 2015.
learning. Nature, 518 (1117580):522955555
[53] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: a framework and survey. J. Mach. Learn, Res., (1(1), January 2020.
[54] Siddharth Nayak, Kenneth Choi, Wenqi Ding, Sydney Dolan, Karthik Gopalakrishnan, and Hamsa Balakrishnan. Scalable multi-agent reinforcement learning through intelligent informa- tion aggregation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Confer- ence on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 25817-25833. PMLR, 23 29 Jul 2023.
[55] Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, page 2681-2690. JMLR.org, 2017.
[56] Andrea Ortiz, Tobias Weber, and Anja Klein. Multi-agent reinforcement learning for energy harvesting two-hop communications with a partially observable system state. IEEE Transactions on Green Communications and Networking, S(1):442-456 2021.
[57] Georgios Papoudakis. Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Bench- marking multi-agen aagent deep reinforcement learning algorithms in cooperative tasks, 2021.
[58] Thomas Paulet and Bryan Cazabonne. An open-source solution for tle based orbit determination. In Proceedings of the 8th European Conference on Space Debris, Darmstadt, Germany, 04 2021. ESA Space Debris Office. ESA Space Debris Office Publication.
[59] Jiaju Qi. Qihao Zhou, Lei Lei, and Kan Zheng. Federated reinforcement learning: techniques, applications, and open challenges. Intelligence & Robotics, 1(1), 2021.
[60] Qingyu Qu, Waner Ma, and Kexin Liu. Cooperative spacecraft formation flying based on reinforcement learning. In 2023 China Automation Congress (CAC), pages 5352-5355 2023.
[61] V.G. Rao and D.S. Bernstein. Naive control of the double integrator. IEEE Control Systems Magazine, 21(5):86-9)) 900 2001.
[62] Juan Luis Cano Rodríguez, Yash Gondhalekar, Antonio Hidalgo, Shreyas Bapat, Nikita As- trakhantsev. Chatziargyriou Eleftheria, Kevin Charls, Meu, Dani, Abhishek Chaurasia, Al- berto Lorenzo Márquez, Dhruv Sondhi, Tomek Mrugalski, Emily Selwood, Manuel López- Ibáñez, Orestis Ousoultzoglou, Pablo Rodríguez Robles, Greg Lindahl, Syed Osama Hussain, andrea carballo, Andrej Rode, Helge Eichhorn, Anish, sme, Himanshu Garg, Hrishikesh Goyal, Ian DesJardin, Matthew Feickert, and Ole Streicher. poliastro/poliastro: poliastro 0.17.0 (scipy us '22 edition), July 2022.
[63] Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of Machine Learning Research, 2022.
[64] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1))61-80, 2009.
[65] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations (ICLR), 2016.
14

---

[66] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. 2016.
[67] Moritz Schneider. Pytorch implementation of ddpg for continuous control tasks. https
(rgithub.com//cceeeeee. / pyttrch 2019.
[68] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, 2016.
[69] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[70] David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. Reward is enough. Artificial Intelligence, 299: 103535, 2021.
[71] William D Smart and Leslie Pack Kaelbling. Practical reinforcement learning in continuous spaces. In ICML. pages 903-910, 2000.
[72] Alexandru Solomon and Ciprian Päduraru. Collision avoidance and return manoeuvre optimisa- tion for low-thrust satellites using reinforcement learning. In 75th International Astronautical Congress (IAC), Milan, Italy, 2024. International Astronautical Federation (IAF).
[73] Christopher J. Sullivan, Natasha Bosanac, Rodney L. Anderson, Alinda K. Mashiku, and Jeffrey R. Stuart. Exploring transfers between earth-moon halo orbits via multi-objective reinforcement learning. In 2021 IEEE Aerospace Conference (50100), pages 1-13, 2021.
[74] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
[75] Nektarios Aristeidis Tafanidis, Avijit Banerjee, Sumeet Satpute, and George Nikolakopoulos. Reinforcement bearningbbssed station keeping using relative orbital elements. Advances in Space Research, 76(2):750-763, 2025.
[76] J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez- Vicente, Niall Williams, Yashas Lokesh, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances Neural Information Processing Systems, volume 34, pages 15032-15043. Curran Associates, Inc., 2021.
[77] Massimo Tipaldi, Raffaele Tervolino, and Paolo Roberto Massenio. Reinforcement learning in spacecraft control applications: Advances, prospects, and challenges. Annual Reviews in Control, 54:1-23, 2022.
[78] K. E. Tsiolkovsky and A. A. Blagonravov, editors. Collected Works K. E. Tsiolkovsky, Vol II: Reactive Flying Machines. NASA TT F-237. National Aeronautics and Space Administration, Washington, DC, 1965. A translation of "K. E. Tsiolkovskiy. Sobraniye Sochineniy. Tom II. Reaktivnyye Letatel' nyye Apparaty," Izdatel' stvo Akademii Nauk SSSR, Moscow, 1954.
[79] G. E. Uhlenbeck and L. S. Ornstein. On the theory of the brownian motion. Phys. Rev., 36:823-841, Sep 1930.
[80] David A. Vallado and Wayne D. McClain. Fundamentals of astrodynamics and applications, volume The space technology library. Microcosm Press, 4th ed edition, 2013.
[81] Yuchen Xiao, Platt Robert, Wong Lawson, and Kaelbling Leslie. Macro-Action-Basse Multi- Agent/Robot Deep Reinforcement Learning under Partial Observability. PhD thesis, USA, 2022. AA129390780.
[82] Min Yang, Guanjun Liu, Ziyuan Zhou, and Jiacun Wang. Partially observable mean field multi-agent reinforcement learning based on graph attention network for uav swarms. Drones, 7(7), 2023.
15

---

[83] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and YI WU. The surprising effectiveness of ppo in cooperative multi-agent games. S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems (NeurIPS 2022), volume 35, pages 2022.
Curran Associates, Inc.,
[84] Guohui Zhang, Xinhong Li, Gangxuan Hu, Yanyan Li, Xun Wang, and Zhibin Zhang. Marl- based multi-satellite intelligent task planning method. IEEE Access, 11:135517-13552, 2023.
[85] Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 04 2018.
16

---

Appendix Table of Contents
OrbitZoo: Framework for Multi-Agent RL in Orbital Dynamics
18
A.1 Architecture and Design
18
A.2 Use Cases
19
Orbital Mechanics
20
Coordinate Systems
20
B.2 Thrust Representation
21
B.3 Propagation
22
Formal Model Definitions
24
C.1
Double Deep Q-Network (DDQN)
24
Deep Deterministic Policy Gradient (DDPG)
24
Proximal Policy Optimization (PPO)
25
C.4
Generalized Advantage Estimation (GAL)
25
Federated Leamning (FL)
25
D
Computational Performance
26
D.I
Hardware Specifications
26
D.2 Scalability
26
D.3 Parallelization
27
D.4
Challenges and Future Improvements
28
Experiments
28
E.1
Learning Algorithms and Architectures
29
OrbitZoo SOTA: Kolosa Comparison
30
Orbit/oo vs. SOTA: Herrera Comparison
32
Hobmann Maneuver
35
E.5
Chase Target
40
Collision Avoidance
43
GEO Constellation
48
F
Exploratory Data Analysis of Starlink open data
51
Comparison of Orekit Propagation and Ephemerides
59
G.1 Figures of Residuals and Orbits
59
G.2 Residuals Analysis
59
Applicability Beyond Starlink
65
Broader Impacts
65
17

---

Drifters
Body
PettingZoo
state
{
Rodlus
Intall Po
Vilcety
reset
()
Intaal Uncertaintyy
Orekit
Satellites
step
(
OrbitZoo
Mare
Theurr Action Sowne
drifters
extends
Thrust Speciff
Agant
Insulte
satollites [1
Satellite
Forces
thrust (
Third Bocy Forocs
Grauity Model
JSON
reset ()
Drag Foroes
agjent {
RL Algorithm
Soler Radiation Pressure
step
Interface
reward ()
Activoinactve
Sodics customization
Colore customication
render
Interface
Environment
Propagation
(
list
function
object
Figure 4: High-level overview of OrbitZoo's architecture. A JSON file describing the orbital system provided to OrbitZoo, which then serves as interface for data generation, single- and multi-agent RL mission development, or orbital dynamics analysis.
A OrbitZoo: A Framework for Multi-Agent RL in Orbital Dynamics
OrbitZoo is flexible and modular Python environment for RL designed address the challenges of applying MARL to high-fidelity orbital dynamics. overcomes key limitations in existing tools, such lack of configurability and visualization, restricted support for realistic perturbative forces, and limited multi-agent capabilities. While the vanilla OrbitZoo environment does not implement specific missions, RI. algorithms, and training logic, it can easily be extended and configured for such cases.
A.1 Architecture and Design
OrbitZoo's s architecture is designed to be modular and extensible. Figure shows the primary modules, where darker boxes represent developed classes and lighter colors indicate integrated external components. This modular design allows users to implement and customize each component independently. ensuring compatibility with diverse RL algorithms and experimental setups.
A.1.1
Core Modules
Body: Modular Propagation of Orbital Dynamics. The Body class is the foundation of OrbitZoo and represents physical entities in the environment. Each body instance contains an individual numerical propagator, which is used to compute future states with great accuracy (position, velocity, uncertainty and mass) knowing the body's current state and active forces. Upon initialization, each body must contain unique identifier (name), dry mass, radius, expected Cartesian position and velocity f== (x. y. ., ÿ. 2), and uncertainties associated with each these elements (o., Ty. g. aa Tg. os). which are internally used construct the covariance matrix Σ with these uncertainties as its diagonal entries. Optionally, the initial date can be provided, allowing an accurate representation of perturbative forces acting on the satellite at specific moment, namely drag. When resetting the environment, the actual position and velocity of the body are sampled from multivariate normal distribution Mi,, E). Although numerical propagation performed in Cartesian coordinates in the background, each body instance provides methods to retrieve its current state in multiple representations Cartesian, Keplerian, or equinoctial. Some static methods also provide information related to two given bodies, such as the distance between them, line of sight without Earth's intersection, Time Closest Approach (TCA), or Probability of Collision (POC), which are useful for large range of RL missions.
18

---

(a) A system propagating (b) A system with differ- (c) A single-agent mis- (d) A multi-agent mission: Constellation in GEO.
several Starlink satellites. ent Keplerian orbits.
sion: Chase Target.
Figure 5: Frames of different systems on OrbitZoo's interface.
Satellite: Extending Bodies with Thrust Capabilities. The Satellite class extends the Body class to incorporate propulsion systems and agent parameters, essential for RL tasks requiring control and maneuverability. Satellites are configured with: (1) polar thrust parametrization (T, 0, o) and action space, including maximum thrust (Tmxx, deviation angle (fma), and azimuthal angle ( mmmax) for realistic thrust modeling; (2) initial fuel mass and thrust-specific impulse (Tsp) for long-duration missions; and (3) agent parameters. The agent parameters are arbitrary and serve as a standardization, depending the specifie requirements of the implemented algorithm for initialization. In addition to the instance methods provided by the Body class, a Satellite includes fuel-related methods that can be used both to define episode termination conditions and to shape reward functions.
Interface:
Interactive Visualization. The Interface class provides interactive 3D visualization of the orbital environment, making OrbitZoo particularly useful for debugging, analysis, and presentation. supports: (1) customizable visual components, such as central body, equatorial grids, Keplerian orbits, velocity and thrust vectors, and satellite trails; (2) real-time updates of system states, including timestamps, body names, and orbital parameters; and (3) flexible camera perspectives for inspecting orbital trajectories and multi-agent interactions. Figure demonstrates some visualization capabilities, ranging from single-agent missions to multi-agent constellations.
Additionally, demonstrate this tool through four videos that highlight its capabilities in various scenarios. The first video offers an overview the user interface, followed by demonstrations of the Hohmann maneuver mission, the GEO constellation mission, and a chase-target mission. These videos can be accessed through the following links: Interface Video Hohmann Maneuver Mission, GEO Constellation Mission, Chase Target Mission, The interface is built upon play3d [1], library that leverages 2D perspective projections to create interactive 3D environments.
High-level Interaction. The Environment (class OrbitZoo) serves the primary interface for users. integrates the above modules to provide a streamlined workflow for designing and executing RL missions. Key features include: (1) high-level methods for retrieving orbital state information, managing agents, and configuring scenarios; (2) flexibility to define single-agent multi-agent missions, supporting tasks such station-keeping, orbital transfers, and collision avoidance; and (3) compatibility with PettingZoo, enabling seamless integration with existing RL workflows. At high level, the environment implements the four fundamental methods common to RL frameworks: reset, step, reward, and render.
Environment:
A.2 Use Cases
Currently, OrbitZoo focuses systems orbiting Earth, reflecting the growing challenges posed by the rapid increase in satellites and debris within LEO. The framework simulates realistic gravity fields and perturbations such atmospheric drag, solar radiation pressure, and third-body effects providing a high-fidelity environment for RL in near-Earth operations while also supporting realistic control maneuvers. Nonetheless, by modifying few physical attributes, such as the gravitational parameter of the central body, the environment can be easily adapted to represent other celestial systems.
Typical RI-based use cases (as explored in this paper) include autonomous orbit transfer, formation flying, rendezvous, collision avoidance and debris tracking, where agents must make real-time deci-
19

---

Figure 6: Representation of the Cartesian position (r) and velocity (r) small body with mass m orbiting large central body with mass M.
sions in complex dynamical regimes. While the current focus remains Earth-centric, the modular design of OrbitZoo supports future extensions to other planetary or multiple central body environ- ments, effectively providing a high-fidelity environment for CR3BP scenarios.
Notably, RL is only one component of OrbitZoo. The environment can be used independently RL to test classical maneuver strategies, analyze orbital evolution, study the intrinsic dynamics of multi-body systems under perturbations. This flexibility allows researchers to validate deterministic control laws, benchmark optimal control solutions, simply observe long-term dynamical behavior under realistic forces.
B Orbital Mechanics
This section provides overview of key orbital mechanics concepts support clearer understanding of the concepts presented throughout the paper. We begin by explaining the most commonly used coordinate systems, then discuss our approach to thrust representation, and conclude with how body propagation is usually handled in orbital dynamics.
B.1 Coordinate Systems
Orbital dynamics is typically described using different coordinate systems, each offering unique advantages depending on the problem hand. Considering a large central body (such as Earth) as an inertial frame. Cartesian coordinates provide a straightforward and intuitive representation of body's s position and velocity in space as r (Ir, y, 2) and i = (ár, ÿ. &), respectively, as shown in Figure While useful for direct numerical computations, such as altitude or distance between bodies, this state rapidly changes and often lacks the deeper insights into the actual orbital shape and orientation, which many missions focus such as the closest point (periapsis) farthest point (apoapsis) from the orbiting body.
Keplerian elements, as Figure 7 shows, describe the orbital motion using five parameters: the semi major semi-major
axis (a), eccentricity (c), inclination (i), argument of perigee (w) and longitude of the ascending node (Ω2.. Within this orbit, the anomaly (which can be represented in three ways, seen Figure 8 an angle that indicates the current body position within that orbit, measuring from the periapsis. The mean anomaly is commonly used due to its linear evolution over time. When there are no perturbative forces (including maneuvers) that change the inclination, all points the orbit are included in two-dimensional plane, called the orbital plane. In these cases, Cartesian positions and velocities vectors are always within this plane.
These elements directly relate to the orbital geometry and are particularly useful for characterizing two-body motion. However, w and Ω become undefined ambiguous in special cases, such as circular or equatorial orbits. These scenarios, referred to singularities, pose significant challenges for numerical integration and optimization problems. Equinoctial elements are therefore preferred for RL, as they avoid these singularities by using five parameters:
Eg cos(w !2) ey sin( + ?)) h tan(i/2) (o)() hy tan(i/2) sin(
20

---

orbiba pane
asconding
(vernal equinoi)
node
Figure 7: Representation of Keplerian Orbital Elements. Here, it is assumed that the orbit is perfectly circular (e = 0), and consequently, the semi semi-major axis (a) corresponds to the Euclidean distance to the primary focus (M).
Figure 8: Types of anomaly. A body S orbits central body C' in an elliptical orbit with semi-majj axis a. The true anomaly (0) is the actual angular position of S measured from the periapsis (closest point to C). The mean anomaly ( (M) is the angle assuming uniform motion over the same orbital period (represented by body S'). Eccentric anomaly (E) is an intermediate angle used to relate M and 6, representing the position of S on the circular orbit (Q). Despite evolving at different rates, all types anomaly reach periapsis (0 rad) and apoapsis (æ rad) at exactly the same moment.
where e, and ey represent components of the eccentricity vector, and h and hy describe the inclination vector.
B.2 Thrust Representation
Spacecraft control often employs local reference frames (centered on the body) for thrust actions due to their intuitive representation, such as RSW. Given a satellite's 's Cartesian position and velocity it is straightforward to compute the radial (f), cross-track (W) and along-track ($) unit vectors:
r rX P
R
W
$ = W x R.
(1)
x fl
where represents the cross product and || is the L2 norm. This coordinate system places the spacecraft the origin, with ft pointing towards the central body, W pointing perpendicular the orbital plane, and Š pointing perpendicular to W and R (approximately the same direction as r)).
In this approach, the thrust usually consists of a vector TRsw = (T Ts. Tw) ) that represents the thrust magnitude each axis of the RSW frame. The action space is then limited maximum thrust magnitude Tmux on each component: TRSW E [-Tmaxx Tmax]3
A more realistic and versatile approach to modeling the thrust action space is to adopt polar parametrization, representing the thrust as (T, ()).).)......)))). A visual comparison can be seen in Figure This parameterization limits the thrust to the action space T < (Tmax, l Omax). When
21

---

T
T
e max
T'
TR
4
W
Tw
W
R
R
Cartesian paramctrization.
Polar parametrization.
Figure 9: Comparison of Cartesian and Polar parameterizationn of the thrust vector T.
max = IT rad and 0max = 2x rad, the satellite can apply thrust in any possible direction. This parameterization naturally constrains the thrust cone-shaped region, providing physically realistic limit on the directions which the satellite can apply force, offering a more controlled and flexible representation of the satellite's s maneuvering capabilities. Finally, the thrust vector in the RSW frame can be obtained via the following transformation:
TRSW Teoss 0$ sin ((cos oR + sin oW)).
Now consider that the spacecraft has a thrust system that produces a force T R Similarly to the gravitational acceleration, the actual acceleration provoked by this thrust system comes from Newton's 's second law of motion:
T 70
e
Othrust
This expression indicates that propulsive capability is inversely related to the spacecraft mass; hence, as propellant expended, the spacecraft is capable of higher acceleration values when using the same force. The rate at which mass is lost is given by:
T
m
Ip9
where Ta is the specific impulse (a measure of the efficiency of the propulsion system) and is standard gravity. A higher Tap value means that the spacecraft can generate more thrust per unit of propellant mass, which makes them more efficient by saving fuel.
B.3 Propagation
The motion of celestial bodies is governed by Newton's 's law of universal gravitation, which states that each body exerts gravitational force on every other body. For a two-body system, the force acting satellite or debris with mass due central body with mass M is given by:
GMm
F
f.
(5)
||||
where G the gravitational constant and f is the unit vector of position r. Using Newton's second law, the resulting acceleration of the smaller body is expressed
GM f
FE
(6)
1||
1|
where #r is called the standard gravitational parameter of Earth.
Although this model forms the foundation of orbital mechanics and is commonly used in RI, analytical models like SGP incorporate additional perturbative forces such as atmospheric drag, SRP, and Earth's s oblateness provide fast and relatively accurate method of propagating bodies.
22

---

Numerical propagators provide a powerful alternative to SGP by solving the equations of motion through numerical integration, enabling precise modeling of complex orbital dynamics. Unlike SGP, numerical propagators can incorporate wide range of perturbative forces (0em) ) including higher-order gravitational harmonies, third-body effects, atmospheric drag, and SRP. This flexibility allows them to handle scenarios requiring high precision, such as long-term orbit predictions and mission-critical maneuvers.
Assuming the state of a spacecraft is characterized by its Cartesian position velocity i and mass m, the state vector s (r, rt m) Ie' can propagated in time using integration methods. The Runge- Kutta (RK4) [1] method is numerical integration technique used to solve first-order differential equations, which can be used approximate the unknown function s dependent on time Since the state corresponds to second-order system, it can be rewritten as first-order system:
8(t,8)
F m
H |
T
+
Geny
di
di
By knowing the state of the spacecraft given moment, create the initial conditions and so- To get an approximation of the state after a step size AI, we define:
At (k1 2kg + 21ss + k:a)
SA1 50
(8)
6
where
11 flo, 80);
Al
ki
1/ = f
to
2 80 At
At
At k2
lss f
to
80
2
K4 = f (lo AI, Alka)
The time dependence f(s, t) in equation 7 enables the modeling forces that are active only at specific times, such environmental accelerations dew or thrust maneuvers.
To propagate the uncertainty, one can use the state transition matrix (STM) analytically approximate the expected uncertainty found in Monte Carlo simulations.
only interested in the uncertainty of the position and velocity r. the state can be simplified to = (r, F), therefore representing the dynamics as [t, s) = (f. F). By calculating the Jacobian A E 6 which linearizes the dynamics around to:
Assuming
0s Ös
Or
0
I
A
Ør
aF
0
Dr
OF
it is possible to approximate the STM (& E 6 as:
== AAt.
(10)
By knowing the position and velocity covariance matrix at to (Eo E R6x6), the propagated covariance
(Exr) becomes:
Σat $5
(11)
23

---

C Formal Model Definitions
For completeness, provide the mathematical details of the methods referred to in the main paper.
C.1 Double Deep Q-Network (DDQN)
Double Deep Q-Network (DDQN) 29 is an improvement over the standard Deep Q-Network (DQN) [5]] algorithm used in RL for environments with discrete action spaces. Although powerful, DQN suffers from overestimation bias (the tendency to overestimate action values due to using the same network for both action selection and action evaluation). DDQN addresses this by decoupling these two roles using two networks: an online network with parameters 0 used for selecting actions, Q0 at), and a target network with parameters A- used for evaluating the value of the selected actions without constant deviations, Qp-(s,, at).
After an agent interacts with the environment and stores several experience tuples (st, st+1) 1) in buffer, DDQN samples a batch B of experiences from the buffer and minimizes the mean squared error (MSE) between the current Q-value, (s az), and the estimated real Q-value, (7 oz):
min
Σ (SL 11
(Te + 7Q0- (81+1, To(8++1)) Qo((), ae))²
(12)
B
@7(s).))
where [B] is the size of the batch, 0 < < 1 is the discount factor and To (8(+))1) = max. Q0(s a) corresponds the best action according to the current parameters 9. Periodically, the target parameters 0- are updated with the parameters of the online network, 0.
Exploration in DQN is guided by the epsilon-greedy strategy, where each time step, the agent selects a random action with probability E (0, 1). and with probability - c, it chooses the action with the highest estimated Q-value. This probability typically starts relatively high to encourage exploration early in training and gradually decreases over time, allowing the agent to increasingly exploit the strategies it has learned.
C.2 Deep Deterministic Policy Gradient (DDPG)
DDPG is an off-policy RL algorithm that contains an actor and critic network, as it stores experiences in a buffer gathered in interactions with old policies, similar to DQN. The actor network with parameters 0 receives a state and produces an action (representing the policy, <p (s)), and the critic network with parameters & receives state and an action, and produces single value (representing the Q-value function, Qs(s,, at)). Similarly to DQN, DDPG also contains a target actor with parameters 0 and target critic with parameters 0-
For optimization, both networks are dependent from the value that the other outputs. While the actor network tries to maximize the expected value produced by the critic (equation (13). the latter tries to approximate the Q-values to the true estimated ones by a MSE loss function (equation 14)
max E E1Q2 (Q)(s(,
(13)
min
Σ
(r+ 7Q0 (s1+1, Fo (o1+1)) -Q)()
(14)
B
Q (..,))
The target network be updated using the soft update method, rather than directly copying the online network. This method uses parameter т E 10, to keep a percentage of the old targets, preventing large, abrupt changes in target values:
0 0 () 0
(15)
Exploration DDPG can be achieved using the Ornstein Saieenbe-hlebeckk process 779. which introduces temporally correlated noise to the actions produced by the actor network. This noise evolves over time and updated each time step according to:
I++1 = 2t + 0(p Iz At + o -N(
(16)
where determines how strongly the noise pulled back toward the mean μl, a controls the magnitude of the noise, and At is small time increment used to discretize the continuous process.
24

---

C.3 Proximal Policy Optimization (PPO)
The actor loss in PPO 69 is defined based on the ratio r0)),, which measures the probability of taking an action or under the current policy compared to the old policy:
(T (rg(ar(s))
rx
(17)
nu (as 8t)
The clipped objective ensures stability during training by preventing excessively large updates to the policy:
min
min
(18)
where controls the trust region, and Ag is the advantage function. The advantage A, is computed recursively using the temporal difference (TD) error:
8e rr (o((8t1) Vo((t);
(19)
A, 5 - ALt1,
(20)
where À is a decay factor and (8)) ) is the value function approximated by the critic network.
Since the actor network in PPO outputs the expected action the agent should take given state, exploration naturally handled by sampling from a probability distribution. Specifically, an action sampled from normal distribution centered at the policy output with given standard deviation
(TT((a | st).
(21)
The standard deviation a typically starts high to encourage exploration the early stages of training and is gradually decreased over time to allow the agent to exploit the strategies it has learned.
C.4 Generalized Advantage Estimation (GAE)
Generalized Advantage Estimation (GAE) 68 provides a flexible way to compute the advantage by balancing bias and variance through the À parameter:
A, Σ (7
(22)
For long episodes, such those encountered orbital dynamies, GAE effectively stabilizes training by reducing variance in advantage estimation.
C.5 Federated Learning (FL)
In MARL environments, there are several approaches collaboratively train agents. A common method is independent learning, where each agent is trained separately and treats other agents as part of the environment's dynamics. In this setting, collaboration arises indirectly through shared reward function, which must be identical for all agents to ensure coordinated behavior.
stable and scalable alternative by enabling agents to share knowledge during training while keeping their local data private. FL is machine learning paradigm in which multiple entities (e.g., agents or devices) collaboratively train global model without sharing raw data. Instead, they exchange model parameters or gradients, which are aggregated improve a shared model.
However, Federated Learning (FL) offers
Applied to RL, FL can be used to exchange learned parameters between agents rather than raw experience tuples. This is particularly beneficial in POMDP environments, where each agent has only a limited view of the full state. By federating value functions or policy networks, agents can better estimate the value of given state 8t, leading to more stable and data-efficient learning while maintaining a level of privacy and decentralization.
A specific case of FL. is horizontal federated learning (HFL) 559
which occurs when all agents have the same observation and action spaces. In such scenarios, multiple identical agents operating in parallel can be treated instances of single agent in multiple environments. This setting allows for
25

---

straightforward parameter sharing and can be seen as data augmentation across parallel environments, enhancing generalization and learning speed.
One common HFI. algorithm that can be used for recent RI. algorithms (e.g. DDPG and PPO) is Federated Averaging (FedAvg) [48, by training shared critic through central server (e.g., ground station). In an environment with N agents, the server starts by sending the current global parameters (wg) every agent: w; wgs Vi E {I, .... N (where w; are the local parameters of agent i). Then, agents interact with the environment for an arbitrary number of steps, gathering a buffer of experiences B,, which are then used to locally optimize the weights w; by RL algorithm. After training. these weights are gathered by the server and averaged:
B:l
W/g
Wi:
(23)
BR
k=
where [B] is the number of experiences buffer B. Finally, the process is repeated by sending these new global parameters to each agent. When assume that the experiences gathered by each agent are independent identically distributed (IID,, FedAvg converges.
D Computational Performance
Real orbital systems can consist of hundreds or even thousands bodies, each requiring propagation at every step. Additionally, different forces may act on different bodies, and each body can have unique properties such as shape, attitude, or drag/reflection coefficients.
OrbitZoo imposes no strict limit on the number of bodies in a system, with scalability constrained only by the available hardware (D.I). However, it is essential assess how increasing the number of bodies and introducing complex dynamics impact simulation speed and parallelization efficiency. These aspects are examined in the following subsections (D.2 and and
Although not yet implemented, OrbitZoo experiments could benefit from libraries mentioned in 3.3 to accelerate training.
D.1 Hardware Specifications
OrbitZoo supports systems of varying sizes and complexity, making hardware requirements dependent on the specific system being modeled, particularly in terms of CPU power and memory. For the following experiments and evaluations, the hardware used is detailed Table[4] with the GPU utilized solely for training RI. agents.
Table 4: Hardware specifications.
Hardware
Specification
CPU
Intel(R) Core(TM) i3-8100 CPU @3.60 Hz, 3600 Mhz, Cores, 4 Logical Processors NVIDIA GeForee GTX 1050 Ti
GPU
RAM
16.0 GB, 2933 Mhz
D.2
Scalability
One the simplest metrics for assessing scalability is simulation speed. Specifically, the time required perform single propagation step for all bodies in the system. Since multiple factors influence this speed, we focus on addressing the following questions:
How does the simulation speed evolve with the addition of bodies?
How does the simulation speed evolve with the addition of forces acting on the bodies (more realism)?
To address the first question, we evaluate system performance when propagating 1, 5, 10, 100, 1000, and 10000 bodies. For the second question, we analyze simulation speed starting with simple
26

---

102
Spherical Harmonk Gravity Third Bodies Soler Raciation Drag Sphericall Gravitt
Spherical Harmonic Gravity Third Bodies
Spherical Harmonk Gravity
10a
Sti. Dev.
Gravity
Time per Step (log s)
10"
10-1
10
10-3
10*
Bodies (log)
Figure 10: Time per step (in seconds) by number of bodies and active forces. Each point represents an average of 100 propagation steps of 10 seconds. Shaded regions indicate = standard deviation.
Newtonian gravity model assuming a perfectly spherical central body. We then introduce a more complex gravity model using spherical harmonics (HolmesFeuthhrssonne) and further incorporate perturbative forces, including third-body effects (Sun and Moon), SRP, and drag.
Results in Figure 10 show that OrbitZoo exhibits approximately ((n) time complexity per step, where n is the number of bodies in the system. This trend becomes more evident as n increases. However, for smaller systems (1 to 10 bodies), the effects of parallelization hardware and OrbitZoo distribute computations across multiple threads. Additionally, the inclusion of perturbative forces increases the step time in roughly linear manner.
noticeable, as the
Scalability and performance also depend on user- user-specific implementations. For example, in large LEO satellite constellations, drag is crucial perturbative force for realistic trajectory modeling. However, drag computation relies on body shape, which assumed to be unique for each satellite by default. As result, each propagator stores its atmospheric data, significantly increasing memory usage. By assuming uniform body characteristics (e.g., identical shapes), a shared force instance can be used across propagators, drastically reducing memory consumption.
D.3 Parallelization
As shown in Figure OrbitZoo integrates with two key external components: PettingZoo and Orekit. PettingZoo provides a framework for sequential or parallelized step computations, while Orekit handles propagation of orbital dynamics within each step. Since orbital MARL missions assume all agents act simultaneously rather than in turns, OrbitZoo propagates systems in parallel manner, so it should naturally take advantage of available tools for faster computations.
OrbitZoo leverages PettingZoo's 's parallelization by implementing the ParallelEnv class, and Orekit's parallelization by implementing the PropagatonsParalllliieee class (a way to simultancously propagate several bodies). To assess the impact of this choice, we compared simulation times between parallel and sequential step computations (as it was presented in Figure I0
Figure II shows that when using few active forces, the sequential propagation is faster than the parallel, even for large number of bodies. However, when systems become more computational demanding - with many perturbative forces the parallel mode becomes the fastest. Given that single RI episode can consist of hundreds thousands of steps, this improvement significantly enhances simulation efficiency, where using the parallel mode can decrease single episode by several minutes.
27

---

10a
Force Models
Spherical Harmonic Gravity Third Bldies Solar Raciation + Drag
Spherical Harmonic Gravity Third Bodiess Solar Raciation
Spherical Haronni Gravity Third Bodies
Spherical Harmonk Gravity
Newtonian Gravity
Std. Dev.
Time per Step (log s)
10"
10-1
10 2
Computation Mode Sequential
10-3
Paraliel
10
Bodies (log)
Figure 11: Time per step (in seconds) when using sequential and parallel computations, by the number of bodies and active forces. Each point represents an average 100 propagation steps of 10 seconds. Shaded regions indicate 1 standard deviation.
Because of these results, OrbitZoo allows the change the computation mode upon environment creation (boolean value) according to its needs (through the JSON input file).
D.4 Challenges and Future Improvements
As mentioned earlier, each propagator is currently treated independently. with duplicated instances for similar bodies. Future work regarding scalability may be related to automatically eliminating this redundancy for body propagators, as forces may be similar for different bodies, therefore benifitting from using the same force instances. This effectively reduces memory usage without the need to manually change these settings.
As real-world satellite constellations continue to grow in size and complexity, further research into scalability becomes increasingly important. Ensuring that RL-based control systems scale effectively with the number of satellites, while maintaining performance and reliability, is a critical step toward practical deployment in real space missions.
E
Experiments
To evaluate OrbitZoo's capabilities, we designed series of missions utilizing algorithms ranging from foundational to state-of-the-art (as discussed in Section [..]. The first mission (Section E.2 compares the OrbitZoo environment to D. Kolosa's ' implementation for a general orbit change maneuver using DDPG, highlighting how the tools and flexibility provided by OrbitZoo can support the development of both existing and novel, more complex missions without the technicalities typically present in orbital dynamics. Then, Herrera's (31 propagation method is compared to the used in OrbitZoo, which is used to develop station-keeping mission in LEOO (Section F.3)
Next, two missions (Sections E.4 and E.5 focus orbital transfers (OTs) using PPO, where agents learn strategies to reach static and dynamic targets, respectively, while considering real-world factors such perturbative forces and mass loss due to fuel consumption.
We then present collision avoidance maneuver (CAM) mission (Section E.6). in which a satellite learns to minimize the Probability of Collision with debris in the days leading up to predicted close approach, while maintaining its initial orbit. This showcases how OrbitZoo can also be employed in missions where uncertainty plays central role. In this mission, we directly compare the performance of strategies learned using DDQN and PPO, highlighting how discrete RI. algorithms perform relative to continuous ones in CAM missions.
28

---

Finally, we introduce a multi- reinforcement learning (MARL) mission (Section E.7 set in Geostationary Earth Orbit (GEO). where four satellites learn distribute themselves evenly along the orbit without losing altitude. For this scenario, we compare independent learning and federated learning approaches using the PPO algorithm, demonstrating how knowledge sharing among agents in cooperative settings can be beneficial.
For each experiment, we first provide a brief definition, highlighting the key aspects of the mission. We then describe the environment setup, detailing the objectives, environment characteristics, observation and action spaces, and reward functions. Finally, we present the results, assessing the generalization capabilities of each policy, followed by discussion of the challenges encountered and potential improvements.
All experiments are fully supported by the accompanying code, which available at the following repository: https://gtt thub. com/ or/i tzootoooot.b......
E.1 Learning Algorithms and Architectures
E.1.1
PPO
The chosen algorithm for a large part of the missions was PPO, since it is a state-of-the-art RL algorithm in environments with continuous action spaces, such as in the experiments carried out in this paper. The implementation was inspired by [8] which offers a minimal PPO setup for discrete and continuous action spaces. However, several modifications were introduced to enhance learning performance, as recommended in 33. These include the use of generalized advantage estimation (GAE) instead of Monte Carlo estimation for calculating advantages, recalculating advantages each epoch, and employing batch-based training. Other improvements include changes in the network architecture (with input normalization, tanh activation functions, and smaller initial weights), together with hyperparameter tuning and performance adjustments (such as calculation of expected returns only at training time).
The overall structure of the actor and critic networks used throughout the experiments is similar, with two hidden layers and Tanh activation functions, represented in Table No extensive research was made to find optimal hyperparameters.
Table 5: Network Structure of PPO Actor and Critic.
Layer
Actor Network (Input: state_ding dim)
Critic Network (Input: state_dim)
Input
Batchllormid (state_da)
__im)
Hidden Layer
Linear (state, cectaat__dm_meeteeeeeeeeee dim_actor, 500), Tanh
Linear (state, cicteeddi__rrrreee 500), Tanh
Hidden Layer? 2
Linear (500, 450), Tanh
Linear (500,, 450), Tanh Linear (450, 1)
Output
Linear (450, action_dimg dim), Tanh
E.1.2 DDPG
DDPG was employed for the mission described in Section E.2 aiming to replicate the work presented in 40 within a custom environment. Our implementation builds upon [67], incorporating the Sinsten-UUllenbed noise process (79 for exploration, and enhanced with prioritized experience replay [65), adapted from the implementation in [66].
The architectures of both the actor and critic networks consist of two hidden layers with Tanh activation functions, as detailed in Table No exhaustive hyperparameter tuning was performed. Unlike PPO, the DDPG critic network estimates the Q-value, evaluating the expected return of the specific action taken by the actor in given state, hence receiving the state-action pair.
Table 6: Network Structure of DDPG Actor and Critic.
Layer
Actor Network (Input: state_dim)
Critic Network (Input: state_dim, action_dim)
Input
Batchloruid (state_ddaateedd
Batchlorrid (atate cectcateddmmmeee sam_critic)
Hidden Layer
Linear (state_din, 512). Tanh
Linear aatraateddim_riittee 512 action_dim), Tanh Linear (512 + action_dim. 256), Tanh
Hidden Layer
Linear 256), Tanh
Output
Linear (25e, action_dim dim), Tanh
Linear (256, 1)
29

---

E.1.3 DDQN
DDQN employed as alternative to PPO for the collision avoidance task (see Section E.6). following a similar approach to that in [13]. The implementation is based on the open-source project [8] which builds upon the original DQN algorithm introduced in [52]. In our implementation, we also incorporate prioritized experience replay 665 and a target Q-value network for increased stability.
The Q-network architecture consists of two hidden layers with Tanh activation functions, as detailed in Table No extensive hyperparameter tuning conducted to optimize performance.
Table 7: Network Structure of DDQN Q-value network.
Layer
Q-Value Network (Input: state_dim)
Input
BatchNormid (statt ddm) Linear (state_ (state_din,, 512), Tanh Linear (512, 256), Tanh
Hidden Layer 1
Hidden Layer 2
Output
Linear (256, action_ding _im
E.2
OrbitZoo vs. SOTA: Kolosa Comparison
E.2.1
Definition
Kolosa 40 developed mission in Medium Earth Orbit (MEO) where spacecraft learns to perform a general orbit change maneuver using DDPG. In this section, we replicate Kolosa's mission within OrbitZoo.
F.2.2 Environment Setup
Objective and Environment Characteristics. We design an environment inspired by Kolosa's setup. The scenario involves spacecraft with dry mass of 500 kg and an initial fuel mass of 150 kg, equipped with thruster of specific impulse Ia 3100 s. The initial orbit is defined by the equinoctial elements (a, Cz2 ey, hy by. Md (5500 + Rg km, 0.153, 0.128, 0.041, 0.015, 10%). with the goal of reaching Rg km, 0.154, 0.171, 0.042, 0.019) representing general orbital transfer, where RØ 6378 is the radius of Earth. The dynamics follow Newtonian gravitational attraction. Each episode spans 692 steps, with a time interval of 500 s per step (approximately four days total).
target orbit characterized by (a, Ex, Cg. ha hy) = (6300
Action Space. Departing from Kolosa's Cartesian thrust representation, we employ a polar thrust parameterization, described in Section B.2 The action each time step is defined as a, (T,0,0) o). constrained by (Tmax, Omax, (max) = (0.6, म, 2T).
Observation Space. Building upon Kolosa's observation space, we include an additional realistic and time-varying parameter: fuel mass, which decreases as thrust applied. The observation at time I is defined as oz (st, Mr, ft), where s, (a, ex, Cy, Nz, hy) are the equinoctial elements, M, is the mean anomaly, and fi the remaining fuel.
Reward Function. The reward function also follows the same formulation, computed based on the deviation between the current and target equinoctial elements. Specifically, for each element the difference Ae is scaled by a corresponding weight Ogg, resulting in:
rt = - + ae, Des + ae, Ae, + ah, Ahg - ah,
(24)
where
ce-
=f=
Ac
((e-
if tens hzr kyy
30

---

Table OrbitZoo vs. Kolosa: Training hyperparameters.
Parameter
Value
Actor learning
0.00001
Critic learning rate
0.0001
Epochs
Discount factor (7)
0.99
0.01
0.2
0.15
A
0.01
Memory Capacity
10 000
Initial Standard Deviation
0.5
Batch Size
256
-90
100
Revard
110
-130
100
Episode
Figure 12: OrbitZoo vs. Kolosa: Cumulative reward per episode using the implemented DDPG.
E.2.3 Results
The weights used during training were kept identical to those in Kolosa:
(aa, Ore Oegy Oh2 anh,)) = (109, 10 10 10 10²).
The DDPG training hyperparameters are summarized in Table g The training performance using our DDPG implementation (see Section E. 1.2 is presented in Figure
When comparing our training performance with Kolosa's, 's we observe that our agent initially achieves higher reward score but ultimately converges to lower final score. This difference is likely due to architectural variations between the implementations. Nonetheless, the agent exhibits stable learning dynamics, with performance stagnating after approximately 200 episodes. Analyzing the evolution of the agent's 's orbital elements (Figure 13), we find that the agent correctly learns the objective, as evidenced by the consistent reduction error across all elements except ey. However, similarly to Kolosa, the agent is unable to converge all orbital elements within the predefined tolerance and time constraints.
Moreover, OrbitZoo offers a distinctive feature: a real-time visualization interface that allows users to observe the agent's 's maneuvers as they occur in space. This capability provides valuable insight into the learning dynamics and behavior of the agent, as illustrated in Figure [4]
This mission demonstrates that OrbitZoo can successfully reproduce similar RL missions with ease, while also offering additional configurations and features that facilitate the development and testing of RL agents.
31

---

12100
01532
www.
14050
01531
12000
0328
0.22
0.24
202
100
1.60
150
Step
Figure 13: Orbital elements of the maneuver per step.
Satellite Trajectory
Target trajectery
6000
4000
2000
10000
5000
-15000 110000 5000
-5000gmm
(km)
5000
10000
-15000
(a) OrbitZ environment.
(b) Kolosa's environment. Extracted from
40
Figure 14: Visual comparison of the maneuver in OrbitZoo and on Kolosa's environment.
E.2.4 Challenges and Future Improvements
Since our DDPG implementation differs from the original, the agent learns different strategy, resulting diverging training outcomes. This highlights the need for further hyperparameter tuning to align the behavior more closely with expected results. Future work may focus replicating additional environments and benchmark results, aiming to evaluate and enhance the generalization capabilities of OrbitZoo across wider range of mission scenarios.
E.3 OrbitZoo SOTA: Herrera Comparison
To compare OrbitZoo's 'ss capabilities with those of existing environments that offer publicly available code, we extend our analysis by implementing the mission proposed by Herrera in (3], consisting of station-keeping mission in LEO.
E.3.1
Definition
Herrera developed reinforcement learning environment for station-keeping in Low Earth Orbit (LEO). While Keplerian orbits deseribe well-defined elliptical path around a central body, real- world satellites are influenced by perturbative forces beyond gravity, causing them to drift from their nominal orbits over time. Station-keeping missions aim to apply strategic control through attitude adjustments thrust maneuvers to maximize the duration satellite stays within its designated
32

---

orbit. This task becomes especially challenging and critical in LEO due to the significant impact of atmospheric drag.
E.3.2 Environment Setup
Objective and Environment Characteristics. In this mission, Herrera employed PPO to train a satellite to control its thrust for station-keeping. The environment's 's dynamics were manually implemented using 4th-order Yoshida integrator, which computes the satellite's next position and velocity based on gravitational and drag forces. The satellite began on circular orbit at an altitude of 550 km above Earth's s surface, with the objective of maintaining that altitude for as long as possible without deviating by more than meter.
Herrera also clearly defined the episode termination conditions: (1) each episode has maximum duration of 800 steps, with each step representing I second; (2) if the satellite does not perform any maneuvers, it exceeds the threshold after approximately 200 steps due natural drift; and (3) the satellite is provided with enough fuel to apply maximum thrust for to 125 steps. For mission requiring this level of precision, the differences dynamic modeling are critical. As stated by Herrera: "The 4th Order Yoshida integrator is used as the default integrator as there is a balance of performance and error. RK4, while generally having less error, is not semantic integrator and requires the calculation of the acceleration 4 times while the Yoshida integrator only requires 3.". Given the difference in integration methods Herrera employing the faster but less precise Yoshida integrator, and OrbitZoo using the more accurate Dormand-Prince integrator (a RK4 integrator with adaptive time steps) we conduct direct comparison between the values produced by each environment over the course of full episode, starting from the same initial position and velocity. As shown in Figure [3] the difference in position after approximately 13 minutes around 25 meters, while the difference in velocity is around 0.08 meters per second. With this comparison, the one meter threshold initially defined for episode termination extremely strict under realistic dynamic conditions. The code needed to run this comparison is provided in the supplementary material.
note that
Poeten kom bifferetee
0.05
0.04
10-
0.03
(a) Fuclidean distance between Herrera's and Orbit- Zoo's propagated position, per second.
(b) Euclidean distance between Herrera's 's and Orbit-
Zoo's 's propagated velocity, per second.
Figure 15: Comparison between Herrera's and OrbitZoo's natural propagation: position and velocity differences over time.
Following Herrera's 's setup, we designed a similar mission in which the agent has an initial mass of 100 kg, including 75 kg of fuel. The satellite is modeled a perfect sphere with radius of 16.8 meters, a drag coefficient of 2.123 and is equipped with a thruster characterized by a specific impulse of Isp 0.0067 While this configuration is not physically realistic, it was chosen maintain consistency with Herrera's 's termination conditions (2) and (3). Notably, in OrbitZoo, atmospheric drag is not solely determined by the satellite's 's current state (as in Herrera's 's implementation) but also incorporates historical data, allowing drag to vary dynamically based on the current moment.
Action Space. To simplify the problem, Herrera limits the action space to the orbital plane, and considers a polar thrust representation (T,0), However, the action the agent performs is not directly the thrust being applied, but the variation applied to the current thrust, creating smoother and more realistic maneuver: a, (AT, A0). While the maximum thrust is char-
33

---

Table 9: OrbitZoo vs. Herrera: Training hyperparameters.
Parameter
Value
Actor learning rate
0.0001
Critic learning rate
0,001
GAE
0.95
Epochs
5
Discount factor (7)
0.99
Clip (c)
0.03
Initial Standard Deviation
0.5
Standard Deviation Decay Rate
10000 steps
Standard Deviation Decay Amount
0.05
Minimum Standard Deviation
0.05
Experiences for Training
800
Batch Size
64
acterized by (Tmax, mmax) (0.04, 2x), the action space is limited to fraction of that change:
(AT mux: A0max) = (Tmax/ max
//ma/50) /50, fmmx/66.
Observation Space. Similarly to Herrera,
Cartesian coordinates represent the current position E 2 and velocity i E I² of the satellite. Additionally, we also define the distance to the nominal altitude, I'arget
Tnominal and nominal velocity, Inareet |||l||
||| nomiinal where
PE
7 nomiinl
550 km
Nomiinal
Toominal
with ng representing the gravitational parameter of Earth. The observation or E IR. consists of: (r/Tnomina / ///nominul: lurggt, Flarget 0, T).
Reward Function. In order to encourage the agent to both run out of fuel and stay within the orbital termination threshold, the reward function is similar to Herrera's:
if Itarget 1m V Vf=0
rh
(25)
+ 0.5
otherwise
S00
where f is the current available fuel in kg, and is the current step within the episode.
E.3.3 Results
Similarly to Herrera, employed PPO trained the agent, with the algorithm hyperparameters being shown in Table 9 If the agent can consistently exceed the baseline of 200 steps representing the duration a satellite remains within tolerance without maneuvers station-keeping. Figure [6] illustttsss that, despite the narrow 1-meter tolerance from the nominal orbit (together with the more realistic perturbations), the agent effectively learned maintain station-keeping by surpassing those 200 steps consistently.
demonstrates successful
E.3.4 Challenges and Future Improvements
The agent did not achieve the same level of performance as in Herrera's environment, primarily due to differences in the dynamics, as demonstrated earlier. Additionally, the PPO implementations differed one using TensorFlow and the other Py Torch with variations hyperparameters.
This attempt to replicate Herrera's environment suggests that OrbitZoo could benefit from using faster propagators. While these may less precise, they are suitable RL missions where extreme precision is not critical. As Herrera points out, continuous but deterministic algorithms (such as DDPG and TD3) may better suited for this mission due to the precision required in controlling thrust maneuvers something that is challenging achieve with PPO.
34

---

M. Momaa M Ma
1
250
Steps
200
100
100
200
300
400
500
600
700
800
Episode
Training
No Maneuvers
Figure 16: Steps per episode, averaged by sliding window of 10 episodes. If no maneuvers executed, the satellite exits the allowed tolerance range by step 200.
E.4 Hohmann Maneuver
The Hohmann transfer experiment 333 was chosen due its well-established relevance in orbital mechanics and its suitability as a benchmark for evaluating reinforcement learning frameworks in this domain. Furthermore, this maneuver is rarely addressed using reinforcement learning, highlighting the importance of investigating it. As one of the most fuel-efficient orbital transfer maneuvers, it provides clear and analytically solvable problem that allows for direct comparison between RL- derived solutions and theoretical optima. Moreover, the Hohmann transfer incorporates key aspects of orbital dynamics, such as thrust application, trajectory optimization, and state transitions, making it ideal testhed to validate the realism and effectiveness of OrbitZoo's 's high-fidelity environment.
E.4.1 Definition
Consider spacecraft on a nearly circular orbit, where its semi-major axis approximately corresponds to the distance of the spacecraft to the primary focus (R). If this spacecraft has the objective of ending up on an orbit with an increased distance R' while maintaining all other elements constant, the Hohmann transfer maneuver can achieve this in a highly fuel-efficient manner by applying two very specific impulsive thrusts in the along along-track track direction (AV and AV') 80 The first establishes the transfer orbit (with semi-major axis At < aи < Rt), and the second adjusts the elements to match target orbit. When considering instantaneous impulses and conservation of energy, these changes in velocity can be easily calculated:
AV
1 R
2R' R
AV'
1 R
1
2R R
(26)
V
where is the standard gravitational parameter of Earth (μI GM ~ 3.986 m²s 3), and R and R' are the radii of the departure and arrival orbit, respectively. According to the Tsiolkovsky rocket equation (78). a body with total mass and m, of fuel mass, and with thrusting capabilities that have specific impulse Tp. can only perform the Hohmann maneuver if AV AV" Isp9o In ma where my go is standard gravity. In a real scenario, the force (Fi) that is required to change the velocity by AV applying that force (Al):
depends the mass of the body and the time that
AV
F
(27)
At
After this maneuver, the body loses some mass (m that inversely proportional to the specific impulse of the thrust:
F
in
(28)
I To
35

---

AV
AV
R
Figure 17: Representation of the Hohmann Maneuver. spacecraft is initially in an orbit with semi-major axis R. It applies thrust AV which positions it the eccentric orbit. Later, it performs a second thrust AV' which positions on the intended orbit, with a semi-major axis of R'.
Although small, this difference in mass will impact the force (F2) that should be applied to change the velocity by AV:.
AV
(mg- m))
F2
(29)
At
Finally, the time between AV and AV' (transfer time, (x) is given by Kepler's third law:
2 3
Hu = 2\
(30)
where usually it is assumed that an = (R + R22 A representation is shown in Figure 17
E.4.2 Environment Setup
Objective and Environment Characteristics. We define a spacecraft with a mass of my = 200 kg
and 50 kg of fuel, starting near-circular orbit with equinoctial elements (a, Cuy Euy h, hyy (2000 + Rg km, 0.007, 0.006, 0.041, 0.015) and ending up on (a, Ez, Ey. has hy) = (2030- -
Rg km, 0.007, 0.006, 0.041, 0.015). Therefore, the objective is raise the semi-major axis by 30 km. The spacecraft is equipped with a thruster that provides a specific impulse of Lap = 310 and capable performing thrusts every direction.
According to the values provided, using the Hohmann transfer, the optimal maneuver requires AV AV' 6.16 m/s and transfer time of t, 3826.1 Therefore, define that each episode contains 1000 steps of 5 seconds, giving extra time for the agents to reach the target. Instead of assuming instantaneous burns, which unrealistic, the thrust is applied for the entire step (5 seconds). For this interval, the optimal forces are F 308 N and Fz 307.9 .9 N in the along-track direction.
For simplicity, we consider a Newtonian attraction model with no perturbations, and the agents to always start in the same anomaly with little uncertainty. The agent completes its objective if reaches the target orbit within a tolerance for every equinoctial element: (+:00 m, +0.055, +0.005, 10,0001,
Action Space. To account for the requirement that thrust should be applied only at specific moments during episode rather than continuously, an additional component included the action space: = (T, 0, Φ, 5). Here, 8 represents the decision to apply thrust, where values close to I or 0 indicate high certainty to apply or retain thrust, respectively. The action space limited by (I'mxx 0max, Qmax mmax) (500, π, 2n, 1).
36

---

Table 10: Hohmann maneuver: Training hyperparameters.
Parameter
Value
Actor learning rate
0.0001
Critic learning rate
0,001
GAE
0.95
Epochs
5
Discount factor (7)
0.99
Clip (c)
0.1
Initial Standard Deviation
0.5
Standard Deviation Decay Rate
40000 steps
Standard Deviation Decay Amount
0.05
Minimum Standard Deviation
0.05
Experiences for Training
4096
Batch Size
64
Observation Space. The observation or is defined vector comprising the current equinoctial elements = (at, Cat, Eyts fit,, hyt), the mean anomaly Mr, and the remaining fuel ft: oz = (st, Mc. fi) E IR?.
Reward Function. The reward function is designed to encourage the agent to apply large but accurate thrusts. First, the absolute difference between the current orbit time and the target orbit, is defined as As, js+- 8l. Next, the scalar progress is computed as P. = w (As,,- As.) st. where w is a vector of constant weights assigned to each orbital element. The reward function is then formulated as:
T P, a1 T
0
Ls50.5
022
(31)
O
where 1..... is an indicator function that activates the reward only when the thrust decision d exceeds 0.5, and an and ag are the importance given to the progress, and actions the along- track direction, respectively.
E.4.3 Results
The agent training hyperparameters are shown in Table TO Two experiments were conducted to analyze the agent's behavior, with two different reward functions as shown in Table []] However, there was no extensive research to find optimal coefficients.
Table 11: Values of an and 02 used in each experiment to analyze the agent's behavior. The first experiment does not penalize thrust deviations from the along-track direction, while the second experiment includes a penalty for such deviations.
Experiment
01
O22
Experiment
1
0
Experiment 1
0.5
For training, the weights used to measure the progress Pt were (103, 100 100 1 10 1 10-3). without extense research to find optimal parameters. The agent was trained until successfully adopted strategy achieve the target orbit within the specified tolerance. Figure T8a shows the reward achieved on each experiment by episode, illustrating that Experiment I took more time to reach good policy, explained by the lack of signaling regarding the along along-track direction (02)). The fuel, as shown in Figure 18b] shows that the agent learned to save fuel in both experiments.
For evaluation, in the second experiment, the generalization of the agent was tested by switching from a Newtonian attraction model to a spherical harmonic gravity, and using third body forces (Sun and Moon), SRP, and drag force.
37

---

mmmm
-20
Dolynce
Squiterr13
Orparnmn::
-Igorimant
Caparmmer:
(a) Cumulative Reward for Experiment and Experi-
(b) Final fuel for Experiment 1 and Experiment 2.
ment 2.
Figure 18: Hohmann Maneuver: Cumulative reward and fuel consumption across episodes.
Figure 19 illustrates that in both experiments, the agent achieved the target semi- major axis while also correcting some other orbital elements. However, in Experiment 1, the agent failed to bring the by component within tolerance. In contrast, the results of the second experiment show a closer approximation to the optimal maneuver. Notably, when using more realistic dynamics by adding perturbative forces, the agent is also able reach the target orbit, even with constant change in orbital parameters created by these perturbations.
12
1.0
tog on
0.8
0.8
0.5
044
10P
0.-
200
50.0
202
800
200
400
820
1000
$t..
50.0
5000
1.70
4.25
1.80
4.20
4.15
1.55
1.50
4.00
1.40
200
200
600
1000
Broermment1
Beperment
Excerment 2 (wo serturtations)
Optimal
Target
Terranc Range
Figure 19: Evolution of equinoctial elements (Aa, Ez Ey hay hyy in the Hohmann maneuver for the first experiment (red), second experiment (yellow), second experiment with realistic perturbations (purple) and optimal maneuver (green). Aa corresponds to the absolute difference between the target and current semi-major axis.
Nevertheless, both experiments demonstrate that the agent incorrectly learned to apply thrusts that alter the inclination, shown in Figure 20 with thrusts being applied in the radial (r) and cross-track (w) directions. Finally, Figure 21 illustrates that the agent's decision policy ō is nearly optimal, with thrust activations occurring both at the beginning of the episode and near tH-
An additional analysis related to the error between the optimal control approach and five successful attempts was performed when using the agent trained in Experiment I and Experiment 2 with different levels of perturbations. In Table T2, we measure the average fuel error (in kg) and standard deviation of these attempts the optimal approach under simplified dynamics. Results suggest that both experiments are able to achieve near optimal control while using, on average, no more than 1 kg of additional fuel, which small portion of all the available fuel.
Additionally, we compare the altitude error (in km) of those attempts by step, as seen Table The steps shown in the table were selected based the intervals where agents perform thrusts (as
38

---

x100
x100
x100
3
(1N)
5
200
400
602
500
2.0
400
502
600
200
600
800
Skep
Step
Expedment:
Experiment
Experment 2 (el perturtal on)
Optimal
Figure 20: Comparison of the applied thrust (r, 8, wt), in Newtons, for the first experiment (red), second experiment (yellow), second experiment with realistic perturbations (purple) and optimal maneuver (green).
Ecpsstment
0.7
Expustment pertutittams
Threshole
02
600 1000
Figure 21: Comparison of the agents decisions (6) to apply thrust during a full episode.
seen in Figure These results suggest that after the first optimal thrust (step 0), Experiment generally better approximates the optimal altitude compared to Experiment 1, but with more variance. After the second optimal thrust (step 765), while Experiment contains smaller error in terms of altitude until the end the episode, Experiment contains a lot more variance, suggesting that in this case the agent arrives the target orbit (given in equinoctial elements) through different trajectories, indicating a better understanding of the dynamics than Experiment These results also suggest that shaping the reward to give importance to the semi-major axis may be needed, since the agents where trained to minimize the error of all equinoctial elements.
Table 12: Average fuel difference (in kg) and standard deviation between each experiment and the optimal maneuver under simplified dynamics (Newtonian attraction), using five successful attempts.
Exp 1 No Perturbations Exp 1 Perturbed
Exp No Perturbations
Exp 2 Perturbed 0.86
(.34
0.36
0.85
Overall, this experiment demonstrates that when trained without perturbations. the policy successfully generalizes to real dynamics, adapting to complex gravity fields, third-body forces, SRP, and atmospheric drag. This is particularly significant LEO, where gravity and drag are the dominant influences on a satellite's trajectory.
Additionally, the impact of providing more informative reward function is evident, as Experiment more quickly learns to avoid performing maneuvers outside the orbital plane.
E.4.4 Challenges and Future Improvements
Despite the agent's ability to generalize to realistic dynamics, it still encounters difficulties reaching the target orbit consistently, as actions in PPO are stochastic by nature, whereas the Hohmann maneuver requires precise execution. Future improvements for this mission include employing rein-
39

---

Table 13: Average altitude difference (in km) and standard deviation between each experiment and the
optimal maneuver under simplified dynamics (Newtonian attraction), using five successful attempts.
Step
Exp No Perturbations
Exp Perturbed
Exp No Perturbations
Exp Perturbed
(.00
(.00
(.00
(.00
(..00
(.01(o.00)
0.01(0.00)
(.00()
2
(o.0))
0.03
0.02(o.02)
0.02 (a.01)
0.07 (0.00)
0.06(..01)
0.04(0.04)
0.04(..01)
4
0. (o.00)
(.10
0.06(0.06)
0.06(o.01)
553
(157
10.13
2.43
(3.0
554
(1.1(
10.13(1.60)
2.35
8.07 (3.58)
555
(.65(
10.12
(.28
8.04
765
(0.73
(.10(1.12)
17 (2.97)
5.66(4.63)
766
10.78
(1.
17. 17.59 17.59 (2.97)
(473
767
(0.83(1.29)
(.00
17. (7.69
5.80
998
(6.30
(.77 3.76(..75)
30.74(3.58)
13.33
999
(6.90
30.74(3.58)
(3.32(6.09)
Table 14: Chase Target: Training hyperparameters.
Parameter
Value
Actor learning rate
0.00001
Critic learning rate
0.0001
GAE A
0.95
Epochs
5
Discount factor (7)
0.99
Clip (e)
0.1
Initial Standard Deviation
0.4
Standard Deviation Decay Rate
10000 steps
Standard Deviation Decay Amount
0.05
Minimum Standard Deviation
0.05
Experiences for Training
4096
Batch Size
64
forcement learning algorithms specifically designed for discrete actions and exploring hyperparameter tuning, since thrusts were not always applied at optimal moments.
F.5
Chase Target
E.5.1
Definition
Orbital transfer missions typically involve placing satellite into specific stationary orbit, seen in the Hohmann transfer, or performing station-keeping maneuvers to maintain a nominal orbit. However, targets in such missions can also be nonstationary, as in scenarios where the objective is to rendezvous with moving bodies such as debris or active satellites, Many missions focus on approaching dynamic targets, like comets, particularly when the goal is scientific observation and study.
E.5.2 Environment Setup
Objective and Environment Characteristics. In this experiment, we have an agent/satellite, called follower, that starts on an orbit characterized by the following Keplerian elements Additionally,
(up, er, ir,ir, there a non-maneuverable body, called target, that starts on an orbit with a much larger altitude and less inclination than the follower: cer,er,i,,u wt, Ut My) (40000
Mr) (10000 RS km, 0.1, 5.0% 10.0%, 10.0", 11.00,10.00 10.0%).
40

---

mmmmmm
Raward
-250
-300
-350
100
120
140
Episade
Figure 22: Chase Target: Cumulative reward per episode.
RE km, 0.001,5.09 5.0%, 10.0 .10.0, 10.0%). To avoid singularities, represent the orbits of the follower (af, Exp Syp,, hgg hgp) and the leader (ØL, Caz Cyy hag hgg) The objective is for the follower to approximate the leader.
equinoctial parameters
The satellite is characterized by being spherical with radius of 5 m, mass 650 kg (from which 150 kg are fuel), and has propulsion system containing specific impulse of Ep 3000 s. Finally, the dynamics for training are composed Newtonian attraction without additional perturbations, and each episode contains 2000 steps of 500 seconds.
Action Space. The satellite thrust system is capable of performing thrusts any direction with maximum magnitude of 30 N, that is, limited by (Tmax 0 0max) (30, π, 2T).
Observation Space. Since the orbital elements of the leader remain static throughout the episode (besides the anomaly Mr.), the observation space of the agent becomes smaller, as it does not need to know those elements: or (af, Cugy Cye Nag hyg MF, ML; Jt), where Le represents the current fuel.
Reward Function. The reward function aims to minimize the error between each equinoctial element, including the anomaly. We first define the difference between each element as:
/ (eL- "
if
Ae
V(el- eFr er), er), cos (ez - er)) if M
if E {ez, ey, hg
(32)
atan2(sin (EL -
The final reward function weights each difference by a factor az:
(aa + dee, Aen + ac,, Reyl + an, A onyy |hy am AM))
(33)
The error for the semi-major axis is normalized by the target element (leader) due of its large scale, and the error in the anomaly needs to account for the periodical nature of the element, where errors vary between [0, r).
E.5.3 Results
For training, the following weights were used, without extensive research for optimal parameters: (Oia Deg ae at Oh,,, a.) (10", 10-3 10-3, 10-20- 110-2 1.1099 10.)). The evolution of training is shown in Figure 22
To assess the performance and generalization ability of the follower, conducted three experiments incorporating all perturbative forces available OrbitZoo, including harmonic gravity fields, third- body forces (Sun and Moon), SRP, and atmospheric drag.
As detailed in T5] the first experiment evaluates the model under the same initial orbital elements used during training but with all forces applied. In the second experiment, the agent starts in an orbit
41

---

Table 15: Chase Target: Initial Keplerian elements of the follower for the different experiments.
Experiment
i
w
M
Experiment
10000 + Rg km
0.1
5.0%
10.0
10.0
10.0
Experiment 2 15000 + Rg km
0.5
8.0
10.0
10.0
90.0P
Experiment
5000 + Rg km
0.001
5.0
10.0
180.0
180.0%
-0.4
0.5
Supp
Tapedmmen:
Experimen2
Drpartmene
Figure 23: Reward per step, for each experiment. A reward of 0.0 indicates that the follower contains exactly the same orbital elements the target, including anomaly.
with a larger semi-major axis, higher eccentricity and inclination, and a different initial anomaly. Finally, the third experiment tests the agent in lower orbit (smaller semi-major axis), with reduced eccentricity and a new initial anomaly. While Experiment 1 assesses the agent's ' generalization to different perturbative forces, Experiment evaluates its adaptability when starting an orbit with sigmificantly different characteristics. Lastly, Experiment 3 tests the agent's 's ability to escape Earth's 's stronger gravitational potential by applying larger thrusts, which requires greater fuel consumption.
The most straightforward way to assess the agent's performance is by tracking the immediate reward full episode. Since the reward represents error, the agent's objective is to minimize it. As shown in Figure 23 the agent effectively reduces the error in all experiments, demonstrating that has learned successful policy for reaching the target. In Experiment 3, the reward remains constant because the agent exhausts all available fuel, maintaining the same orbit until the episode ends. This outcome expected, as escaping Earth's 's gravitational pull requires significantly more force when starting at an altitude of 5000 km compared to 10000 km.
To gain deeper understanding of the agent's 's policy and its impact the reward, examine the differences in each orbital element, as shown in Figure 24 Across all experiments, the agent prioritizes increasing the semi-major axis while keeping the remaining elements as close possible to the target.
This behavior is particularly evident in Experiment 3, where the agent initially struggles to increase the semi-major axis because simultaneously attempts to minimize deviations in other elements while gaining altitude. In contrast, Experiment is the only case where the anomaly closely matches that of the leader, whereas in the other experiments, the agent places greater emphasis on minimizing all elements. This strategy aligns with the weight distribution in the reward function, reflecting the relative importance assigned to each orbital parameter.
This experiment demonstrates that the agent learns an effective policy capable of generalizing thrust maneuvers initial conditions did not encounter during training, while also adapting to realistic orbital dynamics with pon-conservative forces.
E.5.4 Challenges and Future Improvements
Despite the agent demonstrating good generalization capabilities, it fails to perform maneuvers effectively when the target orbit's inclination differs significantly from those encountered during training. Additionally, fuel consumption is not explicitly accounted for in the reward function, which negatively impacts the agent's 's long-term strategy, often resulting in rapid fuel depletion.
Future improvements to this mission could include introducing additional penalties for fuel consump- tion the reward function, training the agent with initial random orbits, and further exploring the limits of the agent's 's strategy.
42

---

-101
0.0
5 0.30
1000
S0op
2003
5000
303m
0.12
w
0.08
0.00
1.0
0.00
1000
10:00
500
1000
1302
2000
25:00
3303
2500
4002
500
2000
2000
4006
9049
50mg
Expertment11
Experiment2
Experiment
Figure 24: Absolute difference between the follower and leader orbital elements per step, for each experiment.
E.6
Collision Avoidance
E.6.1
Definition
Collision avoidance missions often rely probabilities, particularly the Probability of Collision (PoC) 2 between two objects. These include maneuverable body, referred to as the target with radius Rr, and non-mancuueeaale body, referred to the chaser with radius Rc. As the propagation of these bodies is simulated, inherent uncertainties their current positions and velocities grow over time. These uncertainties form the basis for calculating PoC and generating CDMs.
The PoC assumes that the trajectories of the bodies follow linear motion during the very short period of possible collision, neglects uncertainties in their current velocities, and considers that position uncertainties remain constant over the short timeframe [80. By defining the relative position (r TC) and relative velocity (f = èT ic) vectors, the exact Time of Closest Approach (TCA) within the step can calculated as:
T TCA
(34)
simplified coordinate system, known as the RTN frame, then introduced, with the chaser positioned at the origin. The unit vectors defining the RTN frame are computed as follows:
R= rc [[cll
Ñ TC to Ñ= R.
(35)
Ire x fell
where fř, / and Ñ represent the radial, transverse, and normal directions, respectively. These unit vectors form the columns of the 3 x transformation matrix Q = [R, T. NN. which is used to transform the original relative position, velocity, and position covariances of the target and chaser (Er and Ee) into the RTN reference frame:
FRIN Qr
FRIN
Qi
ΣRII = QT
ΣCRIN = Q0.Q=
(36)
Next, an orthogonal coordinate system is established based on the relative vectors:
i ÀRTN
ÀTrTN x NCRTN
j
(37)
|RTS
FTRRN PCRTIN|
where i, J and K are the unit vectors forming the orthogonal system. Similarly to orbital plane encountered in Keplerian orbit (as explained in [B.I), given the assumption of negligible velocity uncertainty, only the j and K axes (defining the conjunction plane) relevant for the analysis. This simplifies the problem
dimensions by introducing projection matrix C = cj K).
Under the assumption that uncertainties follow a Gaussian distribution, the distribution's parameters given by μ C FRTN and Σ C(ETRIN + ΣERTT) The PoC is then calculated as the
43

---

integral over a circle centered at the origin with radius R = Rr + Ro:
1
R2
e 18 dzdy,
PoC
(38)
2T 3
R2²-1²
where
8
p
and [Σ| represents the determinant of matrix E.
In this approach, we use a faster computation method compared to the traditional one, as outlined in 25 According to NASA, a PoC value greater than 10 indicates high collision risk, requiring maneuvers to address it. However, SpaceX places its threshold at 10-6, which is the one used in this experiment.
E.6.2 Environment Setup
Objective and Environment Characteristics. We consider scenario involving two spherical and isotropic bodies; a satellite (target) with mo = 250 kg. 50 kg of fuel, Lep 3100 and Ry = 10 meters- and drifter (chaser) with Rc 5 meters. The real propagation of the bodies follows a model incorporating Newtonian gravitational attraction and atmospheric drag. However, the satellite equipped with a system capable of simulating the propagation of both bodies (and their associated uncertainties) using simplified model based solely on Newtonian gravitational attraction, neglecting drag effects. As expected, this simulator introduces inaccuracies since it does not fully represent the real propagation dynamics. Nonetheless, it provides valuable estimates of key parameters, including the PoC, the TCA, and the miss distance, at any given moment.
The setup of the initial conditions for each episode is critical, as the satellite must begin two days prior to the initial TCA detected by its system, with a PoC exceeding 10 To achieve this, the initial positions of both bodies are set at the same expected location, but with opposing velocities and small uncertainties. At the start of each episode, the states of both bodies are sampled from their respective expected positions and uncertainties, followed by propagating their trajectories backward in time. The objective is for the target to learn a strategy that effectively minimizes the PoC while maintaining trajectory close to its nominal orbit.
In practice, both bodies are considered start with the same Keplerian elements (a, e, i, Ω, M) (2000 + Rg km, 0.01, 5.0% 20.0", 20.0", 10.0%). However, they start with opposite velocity vectors. There is also an uncertainty of 0.1 m and 0.1 m/s when sampling the initial Cartesian position and velocity of both bodies, respectively.
At each step, the satellite simulates the dynamics of both bodies until the end of the episode, considering their current states and uncertainties. The simulation outputs the TCA, miss distance, and PoC. Each step lasts 15 minutes, with termination occurring 10 steps after the initial TCA. Thrust is applied only during the first 10 seconds of each step.
Action Space. As in the Hohmann maneuver mission E..4 mission, apart from the polar repre- sentation of the thrust, the action space also contains the decision (6) to perform the given thrust: a, (T, 0, 6, d). This action space is limited by (Tmax, 0max, Omax 8max) (5, π, 2n,, 1).
Observation Space. The observation o, = (St, Mr, s/c, My. Pt) 14 consists of the current equinoctial elements of the satellite (sx) and the drifter (s)), both bodies mean anomaly (M₃ and M(), together with the current fuel mass (ft) and PoC (Pt).
Reward Function. The reward function is built in a way to not apply thrusts when there's a small PoC, and apply thrusts that do not deviate too much from the satellite's nominal orbit (8) when is higher than 10-6, while reducing the PoC:
aj 1820.5
P 10-6
rt (As: + az 10))
P 2 10 10
(39)
where As; = w (st - 8). with w. representing a vector of weights, Pt- and P. are respectively the PoC before and after the action, and aj and a2 weights given to each parameter.
44

---

-20
Reward
-00
-100
20
40
60
80
100
120
Episede
DOON
PPO
Figure 25: Collision Avoidance: Cumulative reward per episode using DDQN and PPO.
E.6.3 Results
The weights used to measure the deviation from the nominal orbit (As,) were set to
w = (10' 1100",10= 10- ,01,0,000010101111111
and the weights assigned to each term in the reward function were (01, = (10", 10-1). No extensive tuning was performed to find optimal parameter values.
For collision avoidance maneuvers, algorithms that work with discrete actions, such as DQN, work well, as shown by [13]. Because of this, we used both the DDQN (with the implementation shown in
section E.L.3 and PPO (with the implementation shown in section E.I.I algorithms. The training
hyperparameters for both algorithms are shown in Table 16
Table 16: Collision Avoidance: Training hyperparameters for PPO and DDQN.
PPO
DDQN
Parameter
Value
Parameter
Value
Actor learning rate
0.0001
Learning rate
0.00005
Critic learning rate
0.001
Epochs
1
GAE A
0.95
Discount factor (7)
0.95
Epochs
5
T
0.001
Discount factor (r)
0.95
Memory Capacity
10 000
Clip (e)
0.5
Initial e
0.5
Initial Standard Deviation
0.2
Decay Rate
1000 steps
Standard Deviation Decay Rate
5000 steps
Decay Amount
0.05
Standard Deviation Decay Amount
0.05
Minimum
0.05
Minimum Standard Deviation
0.05
Batch Size
256
Experiences for Training
256
Update Target Frequency
10 Online updates
Batch Size
64
Since DDQN only handles discrete actions, we chose to consider the maximum application of thrust in each of the directions in 3D space, together with the non-action, shown in Table 17
The training progression for both algorithms is illustrated in Figure 25 where PPO demonstrates more stable convergence compared to DQN. To evaluate the agent's pertormance, we examine its decision- making behavior through key metrics such as the PoC, miss distance, and TCA. Furthermore, to assess the agent's generalization capabilities, we activate all perturbative forces available in OrbitZoo, including higher-order gravitational harmonics, third-body perturbations from the Sun, Moon and Jupiter, SRP, and atmospheric drag. Importantly, the internal satellite simulator dynamics remain unchanged, preserving the realistic limitation that the agent operates under incomplete information about the environment.
45

---

Table 17: Possible actions in DDQN.
Action
Thrust (T,0,0,
Direction
0
0,,0,,11 1)
Forward
1
T /, 0,
Left (radial out) Behind
2
75,
3
(5, x.
Right (radial in)
4
Up
5
(5, 15,T/2,3m/2.1
Down
6
(0,0,0,0)
Nothing
Thrust
0 0.5
Thrust
0.0
POC POC
High Hilk
20.5
ml
100
days
dynamics
Figure 26: Collision Avoidance using DDQN: Agent decision to apply thrust (6 > 0.5), Probability of Collision (PoC), Miss Distance, and Time of Closest Approach (TCA) by step.
The DDQN results (Figure 26 show that, under unrealistic dynamics, the agent effectively reduces the collision risk a low level before the Time Closest Approach (TCA) by occasionally activating thrust. Under realistic dynamics, although the agent experiences consistently low collision risk throughout the episode, applies thrust almost continuously. detailed examination of the orbital elements in Figure reJrell that, under the unrealistic dynamics encountered during training, the agent performs extremely well by maintaining the nominal orbit. In contrast, under realistic dynam- ics, the agent reduces its semi-major axis by approximately 40 km while keeping the eccentricity components close those the nominal orbit.
In contrast to DDQN, the results from PPO (Figure 28 indicate that the agent learns a strategy of applying thrust early in the episode- precisely when the collision risk highest. This early intervention effectively reduces the risk for the remainder of the episode. Notably, the agent's
2000
0.0005
0.0055
0.0050
0.0045
1960
0.0040
60
60
100
Step
Step
Unrwaistic Dynamics
Dealistic Dynamics
Nominall Orba
Figure 27: Collision Avoidance using DDQN: Evolution of the semi-major axis (a) and eccentricity components (e, and e,y) of the orbit by step, using unrealistic and realistic dynamics.
46

---

0.2
Treust
030
0.40
Thrust
(hpl
10
High
40-9
Low Risk
20*
200
100
says before inttal TCA
Unrealesti dynamics
Resllctcc dynamics
Initaal TIA
Figure 28: Collision Avoidance using PPO: Agent decision to apply thrust (5 > 0.5). Probability of Collision (PoC), Miss Distance, and Time of Closest Approach (TCA) by step.
2002.5
0.00765
2002.0
0,006425
2001.5
0.00760
0.006400
0.006375
0.00755
w/0
0.006350
2000.5
0.00750
2000.0
40
100
40
80
100
20
60
80
100
Step
50eg
Step
Unrwalistic Dynamics
Bealisik: Dynamits
Nominnll Drba
Figure 29: Collision Avoidance using PPO: Evolution of the semi-major axis (a) and eccentricity components (Cz and ey) of the orbit by step, using unrealistic and realistic dynamics.
behavior closely resembles what would be expected under realistic orbital dynamics, demonstrating the effectiveness of the learned strategy. A closer examination of the orbital elements Figure 29 shows that the agent increases the semi-major axis by approximately 3 km while maintaining the cccentricity components near those of the nominal orbit.
From these experiments, we conclude that although both agents learn effective policies under the unrealistic dynamics used during training, PPO demonstrates superior performance under realistic dynamics. This improvement is largely attributed to the continuous action space in PPO, which provides greater flexibility executing maneuvers.
E.6.4 Challenges and Future Improvements
While both algorithms demonstrate some ability to generalize to realistic dynamics, they struggle with the inherent instability of the satellite simulation, particularly the sharp spikes in probability of collision (PoC) near the Time of Closest Approach (TCA). Additionally, the agents have difficulty generalizing their strategies when initialized on orbits that differ from those encountered during training. Although DDQN is unable to fully generalize to realistic dynamics, it shows potential by maintaining consistent eccentricity components, suggesting partial understanding the objective.
Future improvements could include training with randomized time horizon for maneuver com- pletion, initializing agents from broader range orbits, and enhancing the observation space to include more informative features. These modifications would reduce uncertainty and support more effective decision-male making.. Additionally, providing more detailed information to each body's 's internal simulation could further enable robust strategy learning. The DDQN architecture may also benefit
47

---

from the integration of recurrent networks, demonstrated by [13,, which are capable of capturing temporal dependencies critical for decision-makee makinng dynamic environments.
E.7
GEO Constellation
E.7.1 Definition
Unlike LEO and MEO, which occupy defined altitude ranges, the Geostationary Earth Orbit (GEO) is specific orbit located 35786 km above Earth's 's equator. This orbit is significant because any object within it maintains an orbital period equal to Earth's 's rotational period, making it appear stationary to observers on Earth.
There are various possible formations for constellations satellites in geostationary orbit (GEO). In this work, we focus on mission scenario where all agents learn to maintain equal angular separations along the orbit. Specifically, with n satellites, each pair adjacent satellites should be separated by an anomaly difference of 2n/n rad.
Given that the satellites share similar observations and are cooperating, we assume that agents can exchange knowledge after training, as discussed in Section C.5 In the context of PPO, this enables agents to share their critic networks after local training by aggregating and averaging model parameters operation that could be performed, for example, by a ground station. By forming a global critic, each agent gains a more comprehensive understanding of the environment's s overall state, even though it has only partial observations locally.
E.7.2 Environment Setup
Objective and Environment Characteristics. We define an environment with 4 agents in GEO, characterized by semi-major axis of aGEO 35786 + Rg = 42164 km, where Rg is Earth's s equatorial radius, in a near perfect circular orbit. Each agent a satellite with my 250 kg. 50 kg of which is fuel, and equipped with thrust with sp = 3100
Each episode contains 500 steps with step sizes of 360 (approximately 2 revolutions/dayss) and the dynamics are purely based on Newtonian attraction. At the start of each episode (t = 0), agents are initialized with random mean anomalies (Mj. the orbit. The objective for the agents is to learn to distribute themselves uniformly around the orbit, maintaining angular separations of 2T/ /n radians (90 degrees for n 4).
{1, ..., n))) representing their positions along
Action Space. Since GEO lies in a single orbital plane near the equator, the problem's 's dimen- sionality can be reduced. The action space is simplified to two dimensions: a, (T, 0). This configuration allows agents to perform mancuvers affecting the semi semi-major axis and eccentricity components of their equinoctial elements while leaving inclination unchanged. The actions limited to (Tmax, mmx)) = (5, 2T).
Observation Space. The observation space is also reduced. Each agent observation (oe, Ext; Eyt, Mf, Mf. M², Mf) receives its current semi-major axis, eccentricity components, remaining fuel, and the anomalies of all agents in the environment.
Reward Function. To calculate the collective penalty related the difference in anomalies (PMr). we first normalize the anomalies to be within [0, 2Tr, followed by:
max (0,
2T 2m/n Aij
PM
(40)
CT
2T n
where
Ai min (IM, M,1,2 |M, M,1).
The reward function of each agent penalizes differences in altitude (norm of Cartesian position r), magnitude of thrust (T), and close anomalies:
Ti = - -aa (aGBO - |||||| + azT + agPM),
(41)
where (1,, 02 and 0g work as customizable weights. This reward function extensible to an arbitrary number of agents.
48

---

-0000
-4000
Reward
-6000
-8000
10000
12000
144000
25
50
75
100
125
150
175
200
Episode
Federated Learning
Independent Learning
Figure 30: GEO Constellation: Average Cumulative Reward per episode, using independent and federated learning.
0,279
Anomalles Penalty
0.278
0.277
0.276
0.275
100
300
Stop
Federated Learning
Independent Learning
Figure 31: Average Anomaly Penalty (PM) per step over thirty Monte Carlo simulations with realistic dynamics, comparing independent and federated learning. these simulations, initial anomalies are random but equal for both federated and independent constellations.
E.7.3 Results
The agents were trained both independently (without communication) and with a centralized critic using the Federated Averaging (FedAvg) algorithm, with the training hyperparameters provided in Table [8
For training, a combination of parameters that ended up performing rather well was (a1, 022 as)) (10-8, ",10", (0112) The training evolution using these parameters for both experiments is shown at Figure 30
For evaluation, we tested the generalization capability of the policies by introducing realistic dynamics, which include harmonic gravity fields, third body forces, SRP, and drag. To evaluate the performance of both constellations, we conducted Monte Carlo simulations analyzing the evolution of the anomaly penalty, illustrated in Figure 31
The results show that agents are capable reducing the collective anomaly penalty in both scenarios. However, the rate and effectiveness of this reduction depend on the initial constellation configuration. Federated learning demonstrates a faster decrease in the anomaly penalty, highlighting its potential advantage independent learning. Through the OrbitZoo interface, shown in Figure 2 it is evident that the agents are successfully learning the intended objective, albeit with some imperfections and rather slowly.
In conclusion, although the agents were initially trained independently without sharing information, they successfully learned to maintain equal spacing around the GEO orbit under realistic
49

---

Table 18: GEO Constellation: Training hyperparameters.
Parameter
Value
Actor learning rate
0.00001
Critic learning rate
0,0001
GAE
0.95
Epochs
3
Discount factor (7)
0.99
Clip (c)
0.2
Initial Standard Deviation
0.5
Standard Deviation Decay Rate
10000 steps
Standard Deviation Decay Amount
0.05
Minimum Standard Deviation
0.05
Experiences for Training
1024
Batch Size
64
dynamics not encountered during training. The federated approach, however, demonstrated clear advantages, enabling more stable training and a better understanding of the global objective in cooperative setting.
E.7.4 Challenges and Future Improvements
In some initial configurations particularly those where agents begin well separated the anomaly penalty fails to decrease significantly. This limitation appears in both the independent and federated constellations. Addressing it may require tuning hyperparameters more effectively, including adjust- ments to the reward function or exploring alternative RL architectures. Another challenge stems from the current reward setup: the agents must balance reducing the anomaly penalty with maintaining the nominal altitude, which slows the rate of improvement.
Future work includes implementing vertical federated learning (VFL) using split learning, as the FedAvg approach can become unstable due to its simplistic averaging mechanism. Another key direction is scaling to larger constellations to better reflect realistic operational scenarios.
50

---

F Exploratory Data Analysis of Starlink open data
Starlink's ephemeris files are uploaded three times a day, every hours, with each file containing three days of propagated data. This means that consecutive files have overlapping predictions, with the first 8 hours of each file being considered the most accurate. By analyzing the overlapping predictions, we can evaluate Starlink's & own propagation error.
Figures 32 through 35 compare the predictions of a three-day propagation starting on November 27th at around 5:00 AM and ending three days later. The aggregation of 10 consecutive ephemeris files allows for the residual analysis of the initial three-day prediction (from the first ephemeris file) and the subsequent predictions made in the following 9 ephemeris files. This analysis shows that Starlink's data is not error free with prediction errors increase significantly after the 48-hour mark. This is due to a change in propagator by the ephemeris provider. Due to the high computational cost, it shifts to a simpler model that only accounts for the oblateness of the Earth 44
Figures 36 throughh 39 show the evolution of the three positional vectors for four different satellites. Since the plotted position is derived from combination of ephemeris files, using only the most accurate segments from each file, a discontinuity occurs at the transitions between files when one ends and another begins. As expected, the position is oscillatory in nature, reaching its maximum value when the satellite is at apogee (the farthest point from Earth along the major axis of an elliptical orbit) and its minimum value when the satellite is perigee (the closest point to Earth along the same axis). Figures 36 to 44 illustrate the satellites' positional evolution and orbital trajectories over three days.
Ephemeris
Nov 27, 2024, 13:19 UTC
Nov 27. 2024, 21:05 UTC
Nov 28, 2024, 04:40 UTC
13:20 UTC
Residual (km)
Now 28, 2024,21.00 21:09 UTC.
04:34 UTC
Now 29, 2024, 13:13 UTC.
20:59 UTC
Nov 30, 2024, 04:55 UTC
1.0-
0.5
0.0
10
20
30
40
50
60
70
Time (hours from start)
Figure 32: Comparison of predictions between the first ephemeris file, starting on November 27, 2024. 04:50 UTC, with the subsequent nine ephemeris files to form a complete three-day prediction for the satellite with NORAD ID 44744. The residuals represent the distance between the corresponding points in each comparison.
51

---

Ephemens
10
Nev 2024 13:11 VTC Nev 2024, UTC
Nov 28, 2024, 05.05 UTC.
Nov 2024,
Residual (km)
Nov 2024, 21:13 UTC Nov
Nov 29, 2024, 13:06 UTC.
Nov 30, 2024, 04:49 UTC
10 20
30
40
50
60
70
Time (hours from start)
Figure 33: Comparison of predictions between the first ephemeris file, starting on November 27, 2024, 04:54 UTC, with the subsequent nine ephemeris files to form a complete three-day prediction for the satellite with NORAD ID 44748. The residuals represent the distance between the corresponding points in each comparison.
Ephemeris
2024,
Nov 27, 2024, 20:46 UTC
2024,
Nov 28, 2024, 13:13 UTC
Resiual (km)
2024,
Nov 29, 2024, 04:27 UTC
Nov 2024, 13:16 UTC
Nov 2024, 20:51 UTC
Nov 2024, UTC.
30
40 50
Time (hours from start)
Figure 34: Comparison of predictions between the first ephemeris file, starting on November 27, 2024, at 04:21 UTC, with the subsequent nine ephemeris files to form complete three-day prediction for satellite with NORAD ID 44753. The residuals represent the distance between the corresponding points in each comparison.
Ephemeris
Nov 27, 2024, 12:32 UTC
27, 20:4,21:12
Nov 28, 2024, 05:01 UTC
Residual (km)
2.5
Nov 28. 2024, 20:55 UTC
2024, 12:59
29, 2024, 04:51
2.0
Nov 29, 2024, 13:00 UTC
1.5
Now 29, 2024, 21:17 UTC
05:05 UTC
1.0
0.5
10
20
60
(hours start)
Figure 35: Comparison of predictions between the first ephemeris file, starting on November 27, 2024, at 05:05 UTC, with the subsequent nine ephemeris files to form a complete three-day prediction for the satellite with NORAD IDD 44921. The residuals represent the distance between the corresponding points in each comparison.
52

---

Evolution of Position (x) for 44744
Ephemeris
Nov 2024, 04:50 UTC
6000
Nov 27, 2024, 13:19 UTC
Nov 27, 2024, 21:05 UTC
2024, 04:40
4000
Nov 28, 2024, 13:20 UTC
Nov 28, 2024, 21:09 UTC
Position (x) in km
Nov 2024, 04:34
2024, 13:13
Nov 2024, 20:58 UTC
-2000
-4000
6000
Evolution of Position (y) for 44744
4000
Position (y) in km
2000
4000
Evolution Position (2) 44744
6000
4000
Position (2) in km
-2000
-6000
11-27 12
11-28 00
11-28 12
11-29 00
11-29 12
11-30 00
Date
Figure 36: Visualization the satellite's x, y. and z positional evolution over three days for NORAD ID 44744. Since the plotted position is derived from a combination of ephemeris files, using only the accurate segments from each file, discontinuity occurs the transitions between files when one ends, and another begins.
53

---

Evolution of Position (x) for 44748
Ephemeris
Nov 2024, 04:54 UTC
6000
Nov 27, 2024. 13:11 UTC
Nov 27, 2024, 21:08
2024, 05:05
4000
Nov 28, 2024, 13:13 UTC
Nov 28, 2024, 21:13 UTC
Position (x) in km
Nov 2024, 04:59
2024, 13:06
Nov 2024, 21:14 UTC
Nov 30, 2024, 01:49 UTC
2000
-4000
6000
Evolution of Position (y) for 44748
Position (y) in km
2000
4000
Evolution Position (2) 44748
6000
4000
Position (2) in km
-2000
-6000
11-27 12
11-28 00
11-28 12
11-29 00
11-29 12
11-30 00
Date
Figure 37: Visualization the satellite's x, y. and z positional evolution over three days for NORAD ID 44748. Since the plotted position is derived from a combination of ephemeris files, using only the accurate segments from each file, discontinuity occurs the transitions between files when one ends, and another begins.
54

---

Evolution of Position (x) for 44753
Ephemeris
Nov 2024, 04:21 UTC
Nov 27, 2024. 12:50 UTC
4000
Nov 27, 2024, 20:46 UTC
2024, 04:11
Nov 2024, 13:13 UTC
Nov 28, 2024, 20:51 UTC
Position (x in km
2000
Nov 29, 2024, 04:27 UTC
2024, 13:16
Nov 2024, 20:51 UTC
4000
Evolution of Position (y) for 44753
6000
4000
Position (y) in km
2000
-2000
-400
6000
Evolution Position (2) 44753
6000
4000
Position (2) in km
-2000
-6000
11:27 12
11-28 00
11-28 12
11-29 00
11-29 12
11-30 00
Figure 38: Visualization the satellite's x, y. and z positional evolution over three days for NORAD ID 44753. Since the plotted position is derived from a combination of ephemeris files, using only the accurate segments from each file, discontinuity occurs the transitions between files when one ends, and another begins.
55

---

Evolution of Position (x) for 44921
Ephemeris
Nov 2024, 05:05 UTC
4000
Nov 27, 2024. 12:32 UTC
Nov 27, 2024, 21:12
2024, 05:01
Nov 2024, 12:59 UTC
2000
Nov 28, 2024, 20:55 UTC
Position (M in km
Nov 29, 2024, 04:51
2024, 13:00
Nov 2024, 21:17 UTC
-2000
4000
Evolution of Position (y) for 44921
6000
Position (y) in km
2000
-2000
4000
Evolution Position (2) 44921
4000
Position (2) in km
2000
-200
4000
11:27 12
11-28 00
11-28 12
11-29 00
11-29 12
11:30 00
Date
Figure 39; Visualization the satellite's x, y. and z positional evolution over three days for NORAD ID 44921. Since the plotted position is derived from a combination of ephemeris files, using only the accurate segments from each file, discontinuity occurs the transitions between files when one ends, and another begins.
56

---

Ephemeris 27,
Nov 27, 2024, 13:19 UTC
Nov 27, 2024, 21:05 UTC
28,
Nov 28, 2024, 13:20 UTC.
Nov 28, 2024, 21:09 urc
29,
Nov 29, 2024, 13:13 UTC.
Nov 29, 2024, 20:59 UTC.
2000
0
2000
5000
4000
2000
2000
fkml
2000
2000
4000
4000
Figure 40: Visualization of the satellite's orbit evolution over three days for satellite with NORAD ID 44744. To display the most accurate dhree-day trajectory. a combination of consecutive ephemeris files was used, selecting only the most accurate segment from each ephemeris file, roughly the first 8 hours of each file.
Ephemer's
27. 04:54 Nov 27, 2024, 13:11 UTC
27, 2024,
28, 2024, 05:05
Nov 28, 2024, 13:13 UTC
28, 2024, 21:13
29, 2024,
Nov 29, 2024, 13:06 UTC
4000
29, 2024,
Nov 30, 2024, 04:49 UTC
1000
0
2000
4000
40000
4000
2080
2000
+ 7(0m)
(km)
2000*
2000
4000
4000
6000
Figure 41: Visualization of the satellite's orbit evolution over three days for satellite with NORAD ID 44748. To display the most accurate three-day trajectory, combination of consecutive ephemeris files was used, selecting only the most accurate segment from each ephemeris file, roughly the first hours of each file.
57

---

Ephemeris 27,
Nov 27, 2024, 12:50 UTC
Nov 27, 2024, 20:46 UTC
28,
Nov 28, 2024, 13:13 UTC.
Nov 28, 2024, 20:51
29,
Nov 29, 2024, 13:16 UTC.
Nov 29, 2024, 20:51 UTC
2000
0
2000
4000
4000
4000
2000
2000
ffml
x
2000
2000
4000
4000
6000
Figure 42: Visualization of the satellite's orbit evolution over three days for satellite with NORAD ID 44758. To display the most accurate dhree-day trajectory, a combination of consecutive ephemeris files was used, selecting only the most accurate segment from each ephemeris file, roughly the first hours of each file.
Ephemeris
27, 2024,
27, 2024, 12:32
Nov 27, 2024, 21:12 UTC
28,
Nov 28, 2024, 12:59 UTC
Nov 28, 2024, 20:55 UTC
29, 2024,
Nov 29, 2024, 13:00 UTC.
4008
Nov 29, 2024, 21:17 UTC
8000
2000
4000
6000
4000
4000
2000
-2000
x ffm1
/8
2000
2000
4800
6000
4000
Figure 43: Visualization of the satellite's orbit evolution over three days for satellite with NORAD ID 44921. To display the most accurate three-day trajectory, combination of consecutive ephemeris files was used, selecting only the most accurate segment from each ephemeris file, roughly the first hours of each file.
58

---

G Comparison of Orekit Propagation and Ephemerides
To validate Orekit's propagator, physical parameters were optimized using Bayesian optimization. The parameters included:
Drag coefficient (Cp).
Reflection coefficient (Cn).
Satellite radius.
Third-body gravitational forces (enabled/disabled))
The optimization minimized the Root Mean Square Error (RMSE) between Orekit's propagated trajectory and the ephemeris data over 1000 steps (16.6 hours). The RMSE was calculated as:
N
1
RMSE
Σ
rephemeris (t) Forekit [oreki(t)]l?
(42)
N
G.1 Figures of Residuals and Orbits
Figures 49 to 52 present the propagated orbits compared to the ephemeris data. Oscillatory residuals were observed in the y, and components, in Figure 48 increasing over time due to unmodeled perturbative forces and numerical integration errors.
G.2 Residuals Analysis
In this section present further analysis validate the precision of the OrbitZoo propagation against ephemeris data from Starlink. Residuals between real orbital data were computed by comparing overlapping predictions from ephemeris files. The residual each time step was calculated as the Euclidean norm of the difference between the position vectors:
Residual Tephemeris (t) Torekir (t)]]-
(43)
The following scenarios were evaluated understand the discrepancies between predictions:
Scenario 1:
Full Propagation vs. Initial Ephemeris File. Residuals computed for the entire 1000-step propagation against the first ephemeris file.
Scenario 2: Second Half Propagation vs. Second Ephemeris File. Residuals were computed for the second half of the propagation (500 steps) against the second ephemeris file.
Scenario 3: Overlapping Region Between Consecutive Files. Residuals were computed for the overlapping period (approximately hours) between two consecutive ephemeris files.
Figures to illustrate the residuals for these scenarios. The overlapping region provides the most reliable assessment of the Starlink propagator's accuracy, it is based on the most recent ephemeris data. These figures demonstrate the high accuracy of Orekit's propagator when compared to Starlink's ephemeris predictions, particularly in the overlapping regions.
As mentioned earlier, the first hours these ephemeris files are the most relevant as they contain the most up-to-date information at the time of their release. To assess the accuracy of our predictions using Orekit, we must also evaluate the accuracy Starlink's own predictions. If Starlink's error is significant, potentially due to unforeseen events, our propagation may also exhibit large error as result.
propagation of 16.6 hours (1000 steps) was performed using Orekit, Figures through 47 display the residuals of this propagation for four different satellites.
The first case shown in the figures corresponds to the residuals between the propagated data and the first ephemeris file. This case is labeled Orekit Ephemeris 1" because it uses the initial
59

---

ephemeris file for comparison. The propagation was initiated with the first position of this first ephemeris file.
The second case shows the residuals between the same propagated trajectory and the second ephemeris file, labeled "ephemeris 2." This second file contains up-to-date information compared the first ephemeris file, as it was released 8 hours later.
Finally, the third case examines the residuals between the two consecutive ephemeris files (ephemeris and ephemeris 2). These residuals represent Starlink's propagator's 's error, as the discrepancies arise solely from Starlink's own predictions over the overlapping time frame. If the residual from this third case is higher than the error in the first case, means that Orekit's propagation outperformed Starlink's.
As final remark, multiple factors beyond unknown satellite characteristics can contribute to higher RMSE values.
First, limited information on how Starlink produces its ephemeris data can lead inconsistencies when attempting to approximate the satellites trajectories. Researchers [44] suggest that Starlink may use multiple propagators and even switch among them propaaaaan,,,on, introducing further uncertainty into a single ephemeris file. Consequently, reconstructing each satellite's 's real orbital trajectory from Starlink's ephemeris data may vary across different satellites.
Second, our propagation relies exclusively on the first recorded position and velocity from each Starlink ephemeris file, under the assumption that this initial state has the least accumulated error. If that first state inaccurate, our predicted trajectory will be skewed, causing higher RMSE values.
Lastly, Starlink's propagation benefits from detailed space weather information. As result, some of our trajectory predictions may happen to align more closely with real conditions than others, contributing to variability in RMSE values. This same reasoning extends to the previously mentioned satellite characteristics, as well as other forces acting on the satellite.
0.30
Case 1: Orekit vs Ephemeris (1000 steps)
Case 2: Orekit Ephemeris (491 steps)
Case 3: Ephemeris Ephemeris (491 steps)
0.25
Residual (km)
0.20
0.15
0.10
0.05
10
12
14
16
Time (hours)
Figure 44; Visualization of the residuals in three different cases for NORAD ID 44744. The residuals represent the distance between the corresponding points in each comparison. Case (in blue) shows the residuals between the ephemeris data from the first ephemeris file and the Orekit propagation over the entire 1000 steps. Case 2 (in yellow) shows the residuals between the second ephemeris file and the Orekit propagation over the later 491 steps. Case 3 (in green) shows the residuals between the two consecutive ephemeris files during their overlapping region within the 1000 steps. The first and second ephemeris files provide position data for different time spans, with the intersection capturing the transition between them.
60

---

0.175
Case 1: Orekit vs Ephemeris (1000 steps)
Case 2: Orekit Ephemeris (503 steps)
Ephemeris Ephemeris (503 steps)
0.125
Residual (km)
0.100
0.050
0.025
10
12
14
16
Time (hours)
Figure 45: Visualization of the residuals in three different cases for NORAD ID 44753, The residuals represent the distance between the corresponding points in each comparison. Case 1 (in blue) shows the residuals between the ephemeris data from the first ephemeris file and the Orekit propagation over the entire 1000 steps. Case 2 (in yellow) shows the residuals between the second ephemeris file and the Orekit propagation over the later 503 steps. Case 3 (in green) shows the residuals between the two consecutive ephemeris files during their overlapping region within the 1000 steps. The first and second ephemeris files provide position data for different time spans, with the intersection capturing the transition between them.
2.00
Case 1; Orekit Ephemeris (1000 steps)
Case Orekit Ephemeris 2 (491 steps)
1.75
Case Ephemeris Ephemeris (491 steps)
1.50
(km)
1.25
0.75
0.50
0.25
0.00
10
12
14
16
Time (hours)
Figure 46: Visualization of the residuals in three different cases for NORAD ID 44923. The residuals represent the distance between the corresponding points in each comparison. Case (in blue) shows the residuals between the ephemeris data from the first ephemeris file and the Orekit propagation over the entire 1000 steps. Case 2 (in yellow) shows the residuals between the second ephemeris file and the Orekit propagation over the later 491 steps. Case 3 (in green) shows the residuals between the two consecutive ephemeris files during their overlapping region within the 1000 steps. The first and second ephemeris files provide position data for different time spans, with the intersection capturing the transition between them.
61

---

Case 1: Orekit vs Ephemeris (1000 steps)
3.0
Case 2: Orekit Ephemeris (553 steps)
Ephemeris Ephemeris (553 steps)
2.5
Residual (km)
2.0
10
12
14
16
Time (hours)
Figure 47: Visualization of the residuals in three different cases for NORAD ID 44921. The residuals represent the distance between the corresponding points in each comparison. Case (in blue) shows the residuals between the ephemeris data from the first ephemeris file and the Orekit propagation over the entire 1000 steps. Case 2 (in yellow) shows the residuals between the second ephemeris file and the Orekit propagation over the later 553 steps. Case 3 (in green) shows the residuals between the two consecutive ephemeris files during their overlapping region within the 1000 steps. The first and second ephemeris files provide position data for different time spans, with the intersection capturing the transition between them.
Mean X Residual
Mean Y Residual
0.6
Mean Z Residual
Residual (km)
0.2
-0.2
-0.4
2
6
10
14
166
Time (hours)
Figure 48: Visualization of the mean position residuals between the ephemeris data and Orekit propagation over 1000 steps for the satellites with NORAD ID 44744, 44748, 44753 and 44921.
62

---

Z Ezoomwad
4000
2000
Z
2000
40000
1900
-4000
-2000
1000
6200
X
8400 (2oomed)
2000
4000
4000
6000
66:2400
Figure 49: Comparison of the propagated orbits and the ephemeris data, for NORAD ID 44744, illustrating the overlap between the two. The cube in the left plot indicates the approximated region that is magnified in the right plot for closer view. The Orekit propagation represented by solid red line, while the ephemeris data is shown as a dashed blue line.
100
Lonommed
2000
750
2000
4000
2300
000
5000
-400
-2000
2000
y
4000
2000
(aoomed)
4000
6000
6762800
Figure 50: Comparison of the propagated orbits and the ephemeris data, for NORAD ID 44748, illustrating the overlap between the two. The cube in the left plot indicates the approximated region that is magnified in the right plot for closer view. The Orekit propagation is represented by solid line, while the ephemeris data is shown as a dashed blue line.
4000
2000
Z
2000
4000
5500
6000
00g
-2000
4000
3400
yo
2000
2000
X
3000 (2somed)
4000
6000
4000
3850100
Figure 51: Comparison of the propagated orbits and the ephemeris data, for NORAD ID 44753, illustrating the overlap between the two. The cube in the left plot indicates the region approximated that is magnified in the right plot for closer view. The Orekit propagation represented by solid red line, while the ephemeris data is shown as a dashed blue line.
63

---

1000
(c00med)
4000
2000
Z
2000
4008
5600
4000
(bodined)
6000
2000
2hog
x
200g
4000
2000
(zosmed)
6000
4000
26004G0
Figure 52: Comparison of the propagated orbits and the ephemeris data, for NORAD ID) 44921 illustrating the overlap between the two. The cube in the left plot indicates the region approximated that is magnified in the right plot for closer view. The Orekit propagation is represented by solid red line, while the ephemeris data is shown as a dashed blue line.
64

---

G.3 Applicability Beyond Starlink
Since no RL environments offering quantitative propagation analysis were found, the evaluation was extended beyond Starlink data. The Root Mean Squared Error, expressed in meters, was compared over 16 hours of propagation between OrbitZoo (via Orekit) and real satellite data from multiple sources, including OneWeb, as shown in Table 53 Additional datasets were obtained from the EUROLAS Data Center, encompassing organizations such as the Deutsches Geodätisches Forschungsinstinut (DGFI), the Joint Center for Earth Systems Technology (ICET), and the NERC Space Geodesy Facility (NSGF).
Source
Satellite ID
RMSE (m)
One Web
19
3945.9
OneWeb
2540.5
One
20
554.0
OneWeb
3360.3
One Web
14
2007.7
One OneWeb
1581.9
One Web
18
5745.6
Web
4797.2
One Web
23
3067.1
One Web
247.5
One Web
42
1050.8
One Web
39
1552.8
OneWeb
30
5653.4
One Web
21
4228.9
One Web
17
3648.1
One Web
15
1526.9
One Web
46
1095.0
One Web
48
4606.5
One Web
702.8
One Web
49
327.1
Wne
410.4
DGFI
ajisai
1801.2
DGFI
lageosl
4509.7
DGFI
lageos2
3531.3
JCET
etalon1
2096.1
JCET
etalon2
4624.4
NSGF
lares
1553.1
NSGF
lares2
5006.4
NSGF
larets
1169.8
NSGF
starlette
1573.2
NSGF
stella
826.6
Figure 53: Orbital propagation errors (RMSE)
between the Orekit propagation and reference satellites from different organizations.
For some of the tested satellites in Table 53 OrbitZoo maintains errors below km after 16 hours of propagation. We note that satellites outside the One Web constellation experience corrective forces (e.g.. empirical accelerations) modeled by Orekit, which may explain discrepancies.
H Broader Impacts
This work advances RL research in nigh-dimensional, discrete and continuous control domains relevant to space operations, incorporating perturbative forces, model uncertainties, and ephemeris- based validation. Notably, OrbitZoo includes support for federated learning, enabling decentralized training across multiple agents or environments while preserving data locality, an important step toward scalable and privacy-aware coordination in space missions.
65

---

Potential positive impacts include increased autonomy and efficiency in satellite operations, reduced reliance on ground- based control, and more resilient responses to unexpected events like debris conjunctions. OrbitZoo can accelerate the development of safe, adaptive control strategies for real- world missions, and democratize access to space-focused RI. research by offering an open, extensible platform.
central challenge for deployment the sim-to-real transfer the reliable application of policies trained in simulation to real spacecraft. Closing this gap requires models that capture orbital perturbations, actuator limits, and sensor noise while remaining computationally efficient for on-board use. Real-time operation further demands low-latency inference and synchronization with existing flight software. Integrating RL-based control with established attitude and orbit control systems, telemetry pipelines, and ground operations is crucial for achieving certifiable autonomy.
Potential negative impacts include over-reliance black-hox models that may behave unpredictably in safety-critical situations. The deployment of learned policies without robust interpretabilltyy or certification frameworks could lead to unintended behaviors mission failures. Additionally, advances in autonomous maneuvering might be misused in military or competitive commercial contexts without adequate regulatory oversight.
Our experiments demonstrate that policies trained in OrbitZoo can closely replicate real-world satel- lite behavior, including Starlink trajectories, positioning the platform a benchmark for trustworthy and autonomous space systems.
66

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper presents OrbitZoo, a multi-agent reinforcement learning environ- ment built on the high-fidelity Orekit library, designed to enable realistic and accurate orbital simulations. It addresses key challenges in applying RL to orbital dynamics such as the lack of standardization, oversimplified models, and limited reproducibility and highlights how OrbitZoo overcomes these limitations by offering a robust, flexible, and validated platform. The environment is compared against other publicly available frameworks, and capabilities are demonstrated across several representative use cases, including collision avoidance, station-keeping, orbital transfers, and multi-agent missions using both indepen- dent and federated learning and several RL algorithms. These experiments align closely with the abstract's 's claims. Additionally, the paper includes a validation against real Starlink orbital data, showing a MAPE of 0.16%, which substantiates the claim of high-fidelity simulation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the developed framework in terms of environment configurability and computational performance. For each experiment, we evaluate the specific challenges encountered and outline potential directions for future improvement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
67

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There theoretical results presented in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In addition sharing all the code used provide the trained models. Furthermore, the appendix includes a detailed description of the environment setup and the specific configurations used in each experiment, complementing the provided code and facilitating full reproducibility.
the experiments, we also
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
68

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b)
If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide open access to all code developed for this work, including the OrbitZoo environment and all experiment scripts. To facilitate understanding and usage of the environment, comprchensive README. file is also included. The code can be accessed here: code repository
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/ public/guides/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.co maablic/guidess Soaeee/CCSSuubmssiii.. ssiiiooiccyi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
The main paper and appendix contain, for each experiment, detailed descrip- tions of the environment setup, the specific configuration that used, training evolution, and test results. To further support clarity and reproducibility, the accompanying code contains all remaining implementation details.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
69

---

The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Regarding the computational performance of OrbitZoo, we report I-sigma error margins to capture variability in execution time. Specifically, we evaluate how increasing the number of bodies and enabling realistic force models affect simulation speed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide dedicated section on the computational performance of the environment, analyzing how it scales with the addition of bodies and perturbative forces. From these results, along with training data, one can infer the computational resources required for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htt.a//hiiittie//////oooooooooaa http:///wwwpppppccpc///cc/ccccca
70

---

Answer: [Yes]
Justification: Our research complies with the NeurIPS Code of Ethics by using only public simulated orbital data, ensuring privacy and legal compliance. OrbitZoo is only designed for scientific applications and avoids use cases involving surveillance, weaponization, or deception. All datasets and models are documented, licensed, and shared support repro- ducibility and appropriate use. We address environmental concerns by optimizing simulation efficiency and limiting computational overhead. No sensitive personal data used, and our work promotes fairness by enabling flexible evaluation across diverse mission profiles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: The paper includes a dedicated section on the broader impacts of our research, outlining both the potential benefits such as advancing autonomous space operations and possible risks, including the lack of interpretability of learned strategies and policy misuse.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not involve the release of high-risk models datasets. All assets used are publicly available and widely used in the research community. No additional safeguards were necessary.
Guidelines:
71

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In this paper. the assets used include OrbitZoo's interface, Starlink ephemeris data, and publicly available implementations of reinforcement learning algorithms. All assets openly accessible and have been properly cited throughout the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce the OrbitZoo environment as new asset, accompanied by spe- cific use cases and reinforcement learning implementations. This environment is thoroughly documented both in the repository and throughout this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
72

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve human subjects and entirely focused on a multi-agent reinforcement learning environment applied to orbital dynamics.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs not component of our research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy httpp://reerrissccc//offfeeeeeee for what should or should not be described.
73


Reviews:
[{"review_id": "NdhENloPzb", "reviewer": "Reviewer_LSvo", "summary": "The article introduces OrbitZoo, a modular, high-fidelity, and multi-agent reinforcement learning (MARL) environment for orbital dynamics. Built on top of the industry-standard Orekit library and interfaced via Python, OrbitZoo enables realistic satellite simulations, including perturbative forces and customizable thrust models. The framework integrates with the PettingZoo MARL API and provides interactive 3D visualization. Empirical demonstration of its capabilities through various experiments, including single-agent Hohmann transfers, multi-agent constellation coordination, and validation against Starlink ephemeris data. The paper aims to address the lack of standardized, high-fidelity environments for applying reinforcement learning to realistic space scenarios.", "questions": "This is a higher number of questions than expected, but these are rather short and well-targeted based on the weaknesses listed above:\n\n1. Related work taxonomy: What principles guided the choice and separation of the four categories in Section 2? Could the classification be reorganized to improve clarity and avoid overlap?\n2. Feature specification: The main text lacks a detailed overview of OrbitZoo's configurable parameters and environment structure (e.g., thrust constraints, observation/action spaces, default models). Could this be summarized in a table or example use case?\n3. Challenge articulation (Section 3.2): The listed challenges often read as system properties rather than research obstacles. Could the authors reframe these to highlight RL-relevant aspects (e.g., uncertainty, scalability, delayed rewards)?\n4. Fidelity-performance trade-off: The choice of Orekit implies significant computational cost. Appendix C benchmarks this, but this critical trade-off is omitted from the main discussion. Could a summary be added to Section 4 or 5?\n5. Experiment scope vs. depth: The experiments aim to cover a broad range of missions but are presented succinctly, often without learning curves or ablation details. Are these meant to be illustrative use cases or benchmarkable evaluations? Could this section be reorganised?\n6. Federated learning: How is federated learning implemented in OrbitZoo, and what experimental configuration is used in Section 5.2? This concept appears abruptly and is not clearly contextualized.\n7. Multi-objective RL: Have the authors considered extending the environment or its reward structure to support multi-objective optimization? This is especially relevant in scenarios involving fuel efficiency, time, and risk trade-offs.\n8. Appendix content: Could selected content from Appendix D (e.g., D.2 comparison, D.6 validation) be promoted to the main paper? These elements are essential to support the framework’s claims of realism and generality.", "limitations": "The manuscript does not very explicitly discuss limitations or potential risks. While the primary use case is scientific, autonomy in orbital environments raises questions around safety, robustness, and possible misuse. A brief discussion of intended scope, deployment assumptions, or responsible use (e.g., civilian vs. defense) would be appropriate. Scalability to large constellations or real-time applications is also not addressed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The work addresses a significant and timely gap in the reinforcement learning community by providing a standardized, high-fidelity, and publicly accessible environment for satellite operations.\n-  OrbitZoo builds upon the validated capabilities of Orekit, ensuring accurate modeling of orbital propagation, including atmospheric drag, solar radiation pressure, and third-body effects.\n- The modular architecture, integration with the standard library PettingZoo, and availability of interactive visualization are important assets.\n- The validation using real Starlink ephemerides empirically demonstrates the applicability of the simulation engine to real-world scenarios.\n- Several experimental case studies illustrate the flexibility of the environment across different classes of orbital control problems, including both single-agent and multi-agent settings.\n- The appendices contain a significant number of additional contributions (e.g., computational performance benchmarks, comparison with state-of-the-art environments, detailed mission setups) that could benefit from greater visibility in the main text.\n\nWeaknesses:\n- The taxonomy introduced in Section 2 lacks conceptual coherence, mixing levels of abstraction and including overlapping or loosely defined categories.\n- Section 3.2 aims to discuss challenges in MARL for orbital dynamics but largely describes system components without framing them as reinforcement learning challenges (e.g., reward sparsity, partial observability, exploration complexity).\n- While the use of Orekit introduces realism, the trade-off between physical fidelity and computational cost is insufficiently discussed in the main text. Appendix C provides relevant data that should be included in the main paper.\n- Many core features of OrbitZoo, such as agent observation and action space definitions, configurable parameters, and default settings, are only briefly touched upon or deferred to supplementary material.\n- Section 5.1 introduces an “RL agent” but omits details about the specific algorithm used. More generally, the empirical results prioritize breadth over depth: while multiple use cases are presented, each is described briefly, limiting insight into policy behavior, convergence, or comparative performance.\n- Federated learning appears in Section 5 without prior mention or sufficient explanation of its implementation or relevance within the OrbitZoo framework.\n- No discussion is provided on multi-objective formulations, despite their clear relevance for many orbital control problems and knowing the growing importance of MORL and MOMARL.\n- Too much of the most valuable technical content, including benchmark comparisons, advanced scenario configurations, and realism validation, is relegated to the appendix, reducing its impact.\n- (Detail): Several criteria in Table 1 are undefined (e.g., “continuous control”) and the terminology is inconsistent (e.g., \"open-source\" vs. \"publicly available\").", "was_revised": true, "final_justification": "I have no further remarks or questions. While I will maintain the overall score based on the level of contribution, I have increased the clarity score to reflect the enhancements made during the rebuttal phase.", "rebuttal": "We sincerely appreciate your thorough review. We are especially grateful for the recognition of the contributions of OrbitZoo, including its **high-fidelity orbital modeling** via Orekit, integration with PettingZoo to **support both single- and multi-agent reinforcement learning**, and overall **modular architecture**. We also appreciate the positive remarks on the **empirical validation** using real Starlink ephemerides, the **inclusion of diverse case studies**, and the additional resources provided in the appendices. These elements reflect our intent to offer a comprehensive and extensible platform for advancing RL research in the space domain.\n\nSeveral changes have been made to our paper that reflect some of your concerns, which we will mention first. Then, we will address your questions, and remaining weaknesses and limitations, which are clearly **very relevant** for our work.\n___\n## Changes\n### Change 1\nL70 (Section 2) - Reorganization of the previous four categories into three: (1) Tools for Generating and Simulating Orbital Data; (2) Single-Agent Reinforcement Learning in Orbital Dynamics; (3) Multi-Agent Reinforcement Learning in Orbital Dynamics. We also provide the missing context to the terminologies provided in Table 1.\n### Change 2\nL148 (Section 3.2) - Although the structure remains unchanged, for each challenge we explicitly state its implication for reinforcement learning, especially in the multi-agent context.\n### Change 3\nL173 (Section 3.3) - We now address the challenge of exploration in Section 3.2., and change the text to mention the importance of having a modular framework that supports different types of RL algorithms, not just those with continuous action spaces. The ability for OrbitZoo to model multiple objectives (MORL) by using one or several agents through its modular design is now also mentioned in that section (3.2).\n### Change 4\nL179 (Section 3.4) - We extend our discussion on MARL to include both fully cooperative scenarios (using the Dec-POMDP framework) and partial/fully competitive scenarios (using the MA-POMDP framework), together with the importance of Federated Learning (FL) and Federated RL (FRL) in orbital dynamics in these types of scenarios.\n### Change 5\nL246 (Section 4.3) - We now include “Realistic Thrust Modeling” into the “Bridging the Reality Gap” challenge, and discuss how scalability challenges (including the trade-off between computational cost and realism) are handled by OrbitZoo in “Multi-Agent Coordination”. Aditionally, we now mention OrbitZoo’s ability to simulate FRL.\n### Change 6\nL277 (Section 5.1) - In the “Setup” paragraph, we briefly characterize what is the agent (a satellite) and its strategy to apply thrusts (RL algorithm).\n### Change 7\nL1375 (Appendix H) - We have extended our disscussion in sim-to-real transfer, including real-time applications and integration with existing systems.\n### Change 8\nWe have added a new section \"*Framework Architecture and Extensibility*\" before Appendix A, focused on the details of the OrbitZoo framework, including classes, methods, and use-cases.\n___\n## Questions\n\n> Q1. Related work taxonomy: What principles guided the choice and separation of the four categories in Section 2? Could the classification be reorganized to improve clarity and avoid overlap?\n\nYes it could, and we thank you for this helpful observation. **We agree that those four categories can be reorganized into three**, as mentioned in the first change. Since OrbitZoo attempts to create a bridge between orbital dynamics and the RL areas, we first focus on tools for generating such data, followed by how this data is commonly used in RL/MARL missions, and finally how there is no standardization due to the nature of the needed fidelity for each mission. While we previously created separate sections to address single- and multi-agent missions, we are of the opinion that merging these sections improves clarity. **[Change 1](#change-1) reflects this concern**.\n\n> Q2. Related work taxonomy: What principles guided the choice and separation of the four categories in Section 2? Could the classification be reorganized to improve clarity and avoid overlap?\n\nYes, and we thank you for your advice. Although we think that configurable parameters are too extensive to include in the main paper, **we agree that it is useful to add a completely new section in the appendix** that includes the detailed architecture and methods, together with the detailed description of each parameter and some visual use-cases that clearly promote the usage of OrbitZoo. **We wrote this section before appendix A, as stated by [Change 8](#change-8)**.\n\n> Q3. The listed challenges often read as system properties rather than research obstacles. Could the authors reframe these to highlight RL-relevant aspects (e.g., uncertainty, scalability, delayed rewards)?\n\nYes. According to [Change 2](#change-2), **we have reframed the challenges** to create a connection between orbital dynamics and RL challenges.\n\n> Q4. The choice of Orekit implies significant computational cost. Appendix C benchmarks this, but this critical trade-off is omitted from the main discussion. Could a summary be added to Section 4 or 5?\n\nDefinitely. According to [Change 5](#change-5), **we now talk about scalability issues in Section 4.3**.\n\n> Q5. The experiments aim to cover a broad range of missions but are presented succinctly, often without learning curves or ablation details. Are these meant to be illustrative use cases or benchmarkable evaluations? Could this section be reorganised?\n\nThese are **illustrative experiments meant to demonstrate the range of scenarios OrbitZoo supports**, rather than benchmark performance. **Some ablations are included** (e.g., reward shaping for the Hohmann maneuver, RL algorithms for the CAM mission), but in-depth comparisons are not the focus. We acknowledge that further analysis (e.g., on observation/action spaces or perturbation levels) would be valuable. To reflect our current focus on multi-agent setups, **we reorganized the experiments into single-agent and multi-agent categories**.\n\n> Q6. How is federated learning implemented in OrbitZoo, and what experimental configuration is used in Section 5.2? This concept appears abruptly and is not clearly contextualized.\n\n**[Change 4](#change-4) and [Change 5](#change-5) reflect your concerns here**. While FL is typically performed on several devices, OrbitZoo can simulate learning in this scenario by applying network constraints, where — applied to orbital dynamics — vertices correspond to satellites and edges to active communication between satellites, and where communication constraints are normally related to distance and visibility between satellites. In the GEO mission, we assume that visibility remains constant throughout the episode, since at every moment each satellite observes the anomalies of all other satellites without constraints. In this scenario, agents are partially cooperating (similar to a MA-POMDP), where they need to find a balance between individual (orbit) and collective (anomalies) goals. **Horizontal FL is applied by simply averaging network parameters after local updates**, which in practice would be done, for example, by a ground station. **More complex RL algorithms** that tackle challenges like heterogeneity between satellites and dynamic observation of agents **can also be developed** and easily implemented and tested in OrbitZoo, as it is currently being done by us.\n\n> Q7. Have the authors considered extending the environment or its reward structure to support multi-objective optimization?\n\n**OrbitZoo already supports multi-objective RL (MORL) due to its reward structure**. In the *vanilla* OrbitZoo (as seen in Figure 1), reward is just a non-implemented standardized function that expects to output a dictionary of values for each agent. The actual value for each agent can be anything — including vectors [1] — which should be properly handled by the developer in the mission they are working on. Because of this, **it is possible to implement MORL in several ways**, from using just one agent with a vector of rewards, to using several agents with scalar rewards. This is a very interesting area of research.\n___\n## Weaknesses\n- All identified weaknesses but the fifth (\"*Section 5.1 introduces an “RL agent” but omits details about the specific algorithm used.*\") are directly addressed in our answers to the questions you posed. [Change 6](#change-6) aims to clarify what \"RL agent\" means in that context.\n___\n##  Limitations\n- **We provide an overview on the potential positive and negative impacts of OrbitZoo on appendix H**, where we highlight the possible misuse and safety concerns that naturally result form learning RL policies in a simulated environment. However, we agree that the issues you mention should be further discussed. OrbitZoo is not optimized for strict real-time execution, but its modular design and the use of Orekit enable integration with real-time systems in principle. **We have extended section H to include these limitations**, as reflected in [Change 7](#change-7);\n- As mentioned in the fourth question, we now address **scalability within OrbitZoo in Section 4.3, in addition to the quantitative evaluation in appendix C**.\n___\n## Formatting Concerns\n\n- The text shown in those images (timestamp) is a customizable component of the interface. **We have removed that timestamp from images in Figure 2**, as they are not relevant for what is being shown, and **Figure 4 was changed** for a better understanding of the time evolution;\n- Together with the major changes applied in Section 2 ([Change 1](#change-1)), **we have corrected non-defined acronyms** on first mentions, and simultaneously cited relevant papers for them.\n\n[1] Hayes, C.F., Rădulescu, R., Bargiacchi, E. et al. A practical guide to multi-objective reinforcement learning and planning. Auton Agent Multi-Agent Syst 36, 26 (2022)."}, {"review_id": "jM0kRLscyQ", "reviewer": "Reviewer_paFx", "summary": "This paper introduces OrbitZoo, an environment designed for high-fidelity orbital data generation and RL development. The main contribution of this environment includes high-fidelity orbital data generation built on Python, supporting multi-agent reinforcement learning and modular design for customization and visualization. The authors also present some experiments to validate effectiveness of OrbitZoo.", "questions": "See above", "limitations": "See above", "rating": 2, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "The paper is well organized and easy to follow. The main issue of this paper to me is that this paper feels more like a technical report on a specialized tool for studying orbital dynamics. There is no general benchmarking results or new development of RL algorithms. As such, NeurIPS is probably not the best venue for this work and its contribution to general ML community is limited. It's likely that a more domain-specific conference would be more appropriate.", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review, and appreciate your comment on the **organization and clarity shown in the paper**, since orbital dynamics is usually a challenging topic that may require some prior knowledge.\n\nWe also appreciate your perspective regarding the positioning of this work. We would like to clarify that OrbitZoo is **not merely a specialized simulator**, but rather an **infrastructure designed to enable a new area of machine learning research in orbital dynamics** (L61-63). \n\nWe recognize that NeurIPS provides a dedicated Datasets and Benchmarks track, typically intended for standardized datasets or clearly defined benchmarking suites aimed at algorithm comparison. However, OrbitZoo is fundamentally different: it is neither merely a static dataset nor primarily a standardized benchmarking suite. Instead, **OrbitZoo is a novel, modular RL environment purpose-built to support ML-driven innovation and experimentation specifically within orbital dynamics**—an area demanding high-fidelity modeling, realistic physical constraints, and challenging RL-specific aspects (e.g., multi-agent coordination, partial observability, sparse reward structure, and sim-to-real transfer).\n\nThe core motivation behind OrbitZoo lies in **enabling the ML community to define, study, and rigorously test** entirely new classes of RL problems that were previously inaccessible or inadequately explored due to the lack of sufficiently realistic, standardized, and extensible environments. By bridging ML algorithmic innovation with realistic orbital physics through an intuitive MARL interface (PettingZoo), OrbitZoo advances foundational RL research questions rather than solely evaluating established algorithms against fixed benchmarks.\n\nWe thus strongly believe OrbitZoo aligns with NeurIPS’s main track objectives of fostering new ML research directions, rather than the more evaluation-focused nature typically associated with the Datasets and Benchmarks track.\n\nIf there are remaining questions or concerns regarding OrbitZoo, we are available for clarifications."}, {"review_id": "l0g4oqtuGD", "reviewer": "Reviewer_w3KN", "summary": "This paper introduces OrbitZoo, a novel multi-agent reinforcement learning (MARL) environment for satellite operations grounded in high-fidelity orbital dynamics. Built atop the Orekit library, OrbitZoo supports realistic modeling of gravitational, atmospheric, and third-body forces. It integrates with PettingZoo to provide MARL capabilities and supports interactive 3D visualization. The system is evaluated through tasks such as Hohmann transfers, constellation coordination, and validation against Starlink ephemerides.", "questions": "Can the authors provide more quantitative comparisons with existing RL environments or classical control approaches to establish baseline performance?\n\nHow scalable is OrbitZoo in multi-agent settings beyond 4–5 satellites in practice?\n\nCould the interface support integration with onboard learning frameworks for spacecraft? Does the environment offer support for simulating noisy sensor models or communication constraints?\n\nWhat measures are taken to ensure physical plausibility of agent-generated trajectories under unbounded thrust policies?", "limitations": "The limitations are reasonably well-discussed. However, implications of applying such agents in real satellite systems under uncertainty or adversarial conditions could be elaborated further.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe integration of Orekit ensures industry-grade fidelity in orbital simulation, a clear advancement over simplified environments.\nThe inclusion of multi-agent support, real-world data validation, and realistic perturbations within a single open-source platform is a strong contribution. Enables reproducible, high-fidelity MARL experimentation in the space domain, with applications in satellite traffic management and autonomous operations. Overall well-written, with informative figures and detailed architectural breakdowns.\n\nWeakness:\nThe RL training code is not part of the core environment, limiting direct reproducibility.\nExperiments are illustrative but not benchmarked against baseline policies or environments quantitatively.\nReal-time applications, potential safety concerns, and fail-safe guarantees are not addressed in depth.", "was_revised": true, "final_justification": "Discussion complete with the authors", "rebuttal": "We deeply appreciate your thorough review, and the recognition of the **importance of integrating high-fidelity dynamics, support for multi-agent learning, and our focus on realism and reproducibility**. We are glad the overall **clarity and potential applications** came through, and we're grateful for your positive assessment.\n\nWe will first address your questions, followed by additional concerns as effectively as possible.\n___\n## Questions\n\n> Q1. Can the authors provide more quantitative comparisons with existing RL environments or classical control approaches to establish baseline performance?\n\nYes. Since we found no RL environments offering quantitative propagation analysis, **we extended our evaluation beyond Starlink data**. Specifically, we compared the Root Mean Squared Error (RMSE), in meters, over 16 hours of propagation between OrbitZoo (via Orekit) and real satellite data from various sources, including OneWeb. Additional data came from the EUROLAS Data Center, covering organizations such as the Deutsches Geodätisches Forschungsinstitut (DGFI), the Joint Center for Earth Systems Technology (JCET), and the NERC Space Geodesy Facility (NSGF).\nSource|Satellite|Date|RMSE\n-|-|-|-\nOneWeb|19|2020-01-01|3945.9\nOneWeb|1|2020-01-01|2540.5\nOneWeb|20|2020-01-01|554.0\nOneWeb|28|2020-01-01|3360.3\nOneWeb|14|2022-01-01|2007.7\nOneWeb|16|2022-01-01|1581.9\nOneWeb|18|2022-01-01|5745.6\nOneWeb|22|2022-01-01|4797.2\nOneWeb|23|2022-01-01|3067.1\nOneWeb|24|2022-01-01|247.5\nOneWeb|42|2024-01-01|1050.8\nOneWeb|39|2024-01-01|1552.8\nOneWeb|30|2024-01-01|5653.4\nOneWeb|21|2024-01-01|4228.9\nOneWeb|17|2024-01-01|3648.1\nOneWeb|15|2024-01-01|1526.9\nOneWeb|46|2024-01-01|1095.0\nOneWeb|48|2024-01-01|4606.5\nOneWeb|20|2030-01-01|702.8\nOneWeb|49|2024-01-01|327.1\nOneWeb|57|2024-01-01|410.4\nDGFI|ajisai|2018-01-11|1801.2\nDGFI|lageos1|2025-06-07|4509.7\nDGFI|lageos2|2025-07-12|3531.3\nJCET|etalon1|2012-05-12|2096.1\nJCET|etalon2|2012-05-12|4624.4\nNSGF|lares|2025-07-26|1553.1\nNSGF|lares2|2025-07-26|5006.4\nNSGF|larets|2025-07-26|1169.8\nNSGF|starlette|2025-07-25|1573.2\nNSGF|stella|2025-07-26|826.6\n\nFor some tested satellites, OrbitZoo maintains errors below 1 km after 16 hours of propagation. We note that satellites outside the OneWeb constellation experience corrective forces (e.g., empirical accelerations) not modeled by Orekit, which may explain some discrepancies.\n\nIn terms of comparing the RL algorithm with a baseline, we note that the classical control solution for the Hohmann maneuver (described in **Appendix D.3.1**) assumes **simplified dynamics and instantaneous thrusts**. Under **more realistic conditions**—such as continuous thrust and additional perturbations—**an analytical solution is no longer feasible**, and the classical approach fails to reach the target orbit.\n\nIn contrast, **RL algorithms adapt to these realistic dynamics and can still achieve successful transfers**. To support this claim, we performed a quantitative analysis comparing the fuel usage error (in kg) between the optimal maneuver under simplified dynamics and five successful attempts using the agent trained in both Experiment 1 and Experiment 2 (differences discussed in Table 9), across different levels of realism. The table below reports the mean and standard deviation of the fuel error for each configuration:\n\nExp 1 No Perturbations|Exp 1 Perturbed|Exp 2 No Perturbations|Exp 2 Perturbed\n-|-|-|-\n$0.34_{(0.02)}$|$0.36_{(0.01)}$|$0.85_{(0.31)}$|$0.86_{(0.16)}$\n\nThese values suggest that **both experiments are able to achieve near optimal control while using, on average, no more than 1 kg of additional fuel**, which corresponds to only 1/50th of all the available fuel for maneuvers. Additionally, we compare the altitude error (in km) of those attempts by step:\n\nStep|Exp 1 No Perturbations|Exp 1 Perturbed|Exp 2 No Perturbations|Exp 2 Perturbed\n-|-|-|-|-\n**0**|$0.00_{(0.00)}$|$0.00_{(0.00)}$ | $0.00_{(0.00)}$|$0.00_{(0.00)}$\n1|$0.01_{(0.00)}$|$0.01_{(0.00)}$|$0.01_{(0.01)}$|$0.01_{(0.00)}$\n2 |$0.03_{(0.00)}$|$0.03_{(0.00)}$|$0.02_{(0.02)}$|$0.02_{(0.01)}$\n3 |$0.07_{(0.00)}$|$0.06_{(0.01)}$|$0.04_{(0.04)}$|$0.04_{(0.01)}$\n4 |$0.11_{(0.01)}$|$0.10_{(0.01)}$|$0.06_{(0.06)}$|$0.06_{(0.01)}$\n...|...|...|...|...\n553|$2.57_{(1.51)}$|$10.13_{(1.66)}$|$2.43_{(2.07)}$|$8.10_{(3.60)}$\n554|$2.61_{(1.51)}$|$10.13_{(1.66)}$|$2.35_{(2.07)}$|$8.07_{(3.58)}$\n555|$2.65_{(1.51)}$|$10.12_{(1.67)}$|$2.28_{(2.07)}$|$8.04_{(3.55)}$\n...|...|...|...|...\n**765**|$10.73_{(1.30)}$|$5.10_{(1.12)}$|$17.49_{(2.97)}$|$5.66_{(4.63)}$\n766|$10.78_{(1.29)}$|$5.05_{(1.12)}$|$17.59_{(2.97)}$|$5.73_{(4.63)}$\n767|$10.83_{(1.29)}$|$5.00_{(1.11)}$|$17.69_{(2.98)}$|$5.80_{(4.63)}$\n...|...|...|...|...\n998|$16.30_{(0.96)}$|$3.74_{(0.75)}$|$30.74_{(3.53)}$|$13.33_{(6.08)}$\n999|$16.30_{(0.96)}$|$3.76_{(0.75)}$|$30.74_{(3.53)}$|$13.32_{(6.09)}$\n\nThe steps in the table correspond to intervals where agents apply thrust (see Figure 19). After the first optimal thrust (step 0), Experiment 2 typically achieves altitudes closer to the target than Experiment 1, though with higher variance. After the second thrust (step 765), Experiment 1 maintains lower altitude error, while Experiment 2 shows more diverse trajectories, suggesting a better grasp of orbital dynamics. These findings also indicate that modifying the reward to place greater emphasis on the semi-major axis could improve performance, as the current shaping balances errors across all equinoctial elements. **We have included this analysis in the revised version.**\n\n> Q2. How scalable is OrbitZoo in multi-agent settings beyond 4–5 satellites in practice?\n\n**OrbitZoo can easily simulate systems with hundreds or thousands of bodies**, with propagation time largely dependent on the desired level of realism. **We address scalability in Appendix C**, where we analyze how propagation time and memory usage scale with the number of bodies. Simulation time generally increases linearly as more bodies are added, assuming the number of active perturbations remains fixed. For large, high-fidelity systems, **OrbitZoo supports parallelization via Orekit**, which can reduce the time per simulation step by several seconds. Users can enable this feature through a JSON configuration option.\n\n> Q3. Could the interface support integration with onboard learning frameworks for spacecraft? Does the environment offer support for simulating noisy sensor models or communication constraints?\n\nYes. While OrbitZoo is currently intended for ground-based simulation and training, **its modular design allows for adaptation to onboard learning frameworks**. The environment follows standard RL conventions, enabling trained agents to be exported and deployed on embedded systems. Although high-fidelity dynamics are handled by Orekit in the step function, this component can be replaced with simplified or onboard models. In future work, we plan to support reduced-fidelity dynamics with GPU acceleration, enabling fast propagation of large-scale systems for onboard inference or non-control MARL tasks.\n\nOrbitZoo does not yet include built-in models for noisy sensors, but **its architecture supports easily adding such functionality**—an important direction for improving realism. **For communication constraints, OrbitZoo already includes methods for simulating distance and visibility limitations**, including line-of-sight checks that account for Earth’s obstruction, enabling the modeling of scenarios such as communication not only with other satellites, but also with ground-stations by considering them as stationary bodies.\n\n> Q4. What measures are taken to ensure physical plausibility of agent-generated trajectories under unbounded thrust policies?\n\nWhile unbounded policies are dependent on the specific RL algorithm that is being used (e.g., PPO can have them, but DQN can not), and we assume that developers clip the actual thrust to the action space, **OrbitZoo is prepared to handle unbounded policies in several ways**. The *Satellite.change_thrust()* function expects to receive a thrust in polar parametrization (refer to appendix A.2) for each body with thrusting capabilities. **Magnitudes are validated to have a positive force** (if they are not, it is assumed the agent is not performing any thrust). Since the two remaining dimensions correspond to angles indicating the direction of thrust in a 3D space, any real value is accepted.\nIf, by any chance, bodies intersect the central body or orbits become unbounded (hyperbolic trajectory) as a result of bad policies, **Orekit automatically throws an exception that can be handled by the developer**. Additionally, while Orekit does not inherently distinguish between fuel mass and dry mass for propagating bodies (as calculations are only based on wet/total mass), **we provide functions related to levels of fuel that may be used to control unbounded policies** by terminating episodes early (*Satellite.get_fuel()*, *Satellite.has_fuel()*).\n___\n## Limitations\n\n> L1. (...) implications of applying such agents in real satellite systems under uncertainty or adversarial conditions could be elaborated further.\n\nWe agree that real-world deployment introduces significant challenges. In particular, uncertainty in sensing (as you mentioned), thrust delays, and unmodeled dynamics can degrade agent performance. However, **these uncertainties can all be simulated in OrbitZoo**. Additionally, **adversarial scenarios are naturally supported in OrbitZoo**. For instance, in the MARL experiment we developed, the mission does not perfectly follow a Dec-POMDP framework given that agents contain an adversarial component, where they need to find a balance between individual (orbit) and collective (anomalies) rewards. That is, at each time step they receive different rewards, introducing some level of competition. In the revised version, **we have expanded our discussion in Section 3.2** (L148) to highlight the issue and importance of uncertainty and adversarial training frameworks."}, {"review_id": "zklqXnjG1x", "reviewer": "Reviewer_4WXn", "summary": "This paper proposes a new multi-agent RL environment for orbital dynamics called OrbitalZoo. OrbitalZoo can support data generation/simulation, RL algorithm research and visualization, etc. It addresses some key challenges in this area, including Bridging the Reality Gap, Realistic Thrust Modeling, Multi-Agent Coordination, and Interactive Visualization for Debugging and Analysis. Experimental results including single agent, multi-agent as well as real world data validation show OrbitZoo as a reliable benchmark for RL-based autonomy in space operations.", "questions": "1. What's the plan for open source?\n2. What is the baseline algorithm? Compared w/ RL algorithms, is RL much better than baseline algorithm?\n3. Is the MARL support general to all RL algorithms? Or does it only work for PPO?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written.\n2. It seems OrbitZoo supports a variety of RL algorithms and experiments provide comparison among these algorithms for the same task.\n3. Various of interesting tasks related to orbital dynamics are included in this benchmark, such as chase target, collision avoidance, Hohmann Maneuver, etc.\n\nWeakness:\n1. regarding \"Bridging the Reality Gap.\" it seems it is only supported by the Starlink example, is there any other evidence to show OrbitZoo can bridge the reality gap better than other environments?\n2. Is there any example of OrbitZoo's visualization capabilities?", "was_revised": true, "final_justification": "Very solid work and authors have addressed all my concerns.", "rebuttal": "We thank you for your positive feedback. We are glad to hear that the **clarity of the writing**, the **variety of reinforcement learning algorithms supported**, **multi-agent capabilities**, and the **diversity of orbital tasks included in OrbitZoo** were well received. We appreciate your recognition of the **benchmark's breadth and relevance** to real-world orbital dynamics problems.\n\nWe will first address your questions and related weaknesses, followed by a discussion of the remaining limitations and areas for improvement as thoroughly as possible.\n___\n## Questions\n\n> Q1. What's the plan for open source?\n\n**We already provide full access** to the OrbitZoo environment, trained models, and code needed to reproduce our experiments in both the **supplementary material** and on the **anonymized repository** (appendix D, in the end of the introduction). In the near future, we do **intend to provide a public repository** so that people can independently contribute in their areas of expertise (e.g., use cases, RL algorithms, propagation and integration, scalability, more methods for body interactions, interface components, etc.), building upon what was already developed.\n\n> Q2. What is the baseline algorithm? Compared w/ RL algorithms, is RL much better than baseline algorithm?\n\nAnswering your first question, **we disagree with the assumption that there is a defined baseline throughout the paper**, since **there are comparisons that evaluate OrbitZoo with other environments** (Section D.2) not only in terms of **RL algorithms** (Kolosa’s environment), but also regarding **data fidelity** (Herrera’s environment), and more nuanced differences, such as the **actual code** needed to develop such missions and **easily evaluate** them. We highlight that **it is not our intent here to achieve state-of-the-art results** within each of these missions, but simply **showcase the large range of missions that OrbitZoo supports**, which can indeed be used to easily develop new and more robust algorithms, fundamentally working as a **RL research enabler**. Nonetheless (and concerning your second question), the Hohmann maneuver contains an optimal classical control approach (as shown in D.3.1) that can benefit from a more detailed comparison to the strategy learned by PPO. We have performed an additional analysis on the error between the optimal control approach and five succesfull attempts when using the agent trained in Experiment 1 and Experiment 2 (refer to Table 9) with different levels of perturbations. In the table below, we measure the average fuel error (in kg) and standard deviation of these attempts to the optimal approach under simplified dynamics:\n\n| Exp 1 No Perturbations | Exp 1 Perturbed | **Exp 2** No Perturbations | **Exp 2** Perturbed |\n| --- | --- | --- | --- |\n| $0.34_{(0.02)}$ | $0.36_{(0.01)}$ | $0.85_{(0.31)}$ | $0.86_{(0.16)}$ |\n\nAdditionally, we compare the altitude error (in km) of those attempts by step:\n\n| **Step** | **Exp 1** No Perturbations | **Exp 1** Perturbed | **Exp 2** No Perturbations | **Exp 2** Perturbed |\n| --- | --- | --- | --- | --- |\n| **0** | $0.00_{(0.00)}$ | $0.00_{(0.00)}$ | $0.00_{(0.00)}$ | $0.00_{(0.00)}$ |\n| 1 | $0.01_{(0.00)}$ | $0.01_{(0.00)}$ | $0.01_{(0.01)}$ | $0.01_{(0.00)}$ |\n| 2 | $0.03_{(0.00)}$ | $0.03_{(0.00)}$ | $0.02_{(0.02)}$ | $0.02_{(0.01)}$ |\n| 3 | $0.07_{(0.00)}$ | $0.06_{(0.01)}$ | $0.04_{(0.04)}$ | $0.04_{(0.01)}$ |\n| 4 | $0.11_{(0.01)}$ | $0.10_{(0.01)}$ | $0.06_{(0.06)}$ | $0.06_{(0.01)}$ |\n| ... | ... | ... | ... | ... |\n| 553 | $2.57_{(1.51)}$ | $10.13_{(1.66)}$ | $2.43_{(2.07)}$ | $8.10_{(3.60)}$ |\n| 554 | $2.61_{(1.51)}$ | $10.13_{(1.66)}$ | $2.35_{(2.07)}$ | $8.07_{(3.58)}$ |\n| 555 | $2.65_{(1.51)}$ | $10.12_{(1.67)}$ | $2.28_{(2.07)}$ | $8.04_{(3.55)}$ |\n| ... | ... | ... | ... | ... |\n| **765** | $10.73_{(1.30)}$ | $5.10_{(1.12)}$ | $17.49_{(2.97)}$ | $5.66_{(4.63)}$ |\n| 766 | $10.78_{(1.29)}$ | $5.05_{(1.12)}$ | $17.59_{(2.97)}$ | $5.73_{(4.63)}$ |\n| 767 | $10.83_{(1.29)}$ | $5.00_{(1.11)}$ | $17.69_{(2.98)}$ | $5.80_{(4.63)}$ |\n| ... | ... | ... | ... | ... |\n| 998 | $16.30_{(0.96)}$ | $3.74_{(0.75)}$ | $30.74_{(3.53)}$ | $13.33_{(6.08)}$ |\n| 999 | $16.30_{(0.96)}$ | $3.76_{(0.75)}$ | $30.74_{(3.53)}$ | $13.32_{(6.09)}$ |\n\nResults from the first table suggest that **both experiments are able to achieve near optimal control while using, on average, no more than 1 kg of additional fuel**, which is a small portion of all the available fuel. The steps shown in second table were selected based on the intervals where agents perform thrusts (as seen in Figure 19). These results suggest that after the first optimal thrust (step 0), Experiment 2 generally better approximates the optimal altitude compared to Experiment 1, but with more variance. After the second optimal thrust (step 765), while Experiment 1 contains a smaller error in terms of altitude until the end of the episode, Experiment 2 contains a lot more variance, suggesting that in this case the agent arrives at the target orbit (given in equinoctial elements) through different trajectories, indicating a better understanding of the dynamics than Experiment 1. These results also suggest that shaping the reward to give more importance to the semi-major axis may be needed, since the agents where trained to minimize the error of all equinoctial elements. **We have added this analysis in the revised version**. We hope this addresses you question.\n\n> Q3. Is the MARL support general to all RL algorithms? Or does it only work for PPO?\n\n**MARL support within OrbitZoo is general for all RL algorithms, not just PPO**. This is possible due to the modular architecture shown in Figure 1, where the actual RL algorithm for each satellite is not strictly defined. In fact, when using more complex MARL algorithms that contain centralized training decentralized execution (CTDE) properties (which is common in missions with homogeneous agents), we find that **it is easy to even change the initial architecture that OrbitZoo provides for specific missions*. **Centralizing RL algorithms that contain shared parameters across satellites** (e.g., as seen in IPPO and MAPPO) **is straightforward in OrbitZoo**.\n___\n## Weaknesses\n\n> W1. Regarding \"Bridging the Reality Gap.\" it seems it is only supported by the Starlink example, is there any other evidence to show OrbitZoo can bridge the reality gap better than other environments?\n\nYes. Since we did not find existing RL environments providing quantitative propagation analysis, **we extended our evaluation beyond Starlink data**. We compared the 16-hour propagation Root Mean Squared Error (RMSE), in meters, between OrbitZoo (using Orekit) and real satellite data from multiple sources, including OneWeb. Additional datasets were obtained from the EUROLAS Data Center, representing organizations such as the Deutsches Geodätisches Forschungsinstitut (DGFI), the Joint Center for Earth Systems Technology (JCET), and the NERC Space Geodesy Facility (NSGF):\nSource|Satellite|Date|RMSE\n-|-|-|-\nOneWeb|19|2020-01-01|3945.9\nOneWeb|1|2020-01-01|2540.5\nOneWeb|20|2020-01-01|554.0\nOneWeb|28|2020-01-01|3360.3\nOneWeb|14|2022-01-01|2007.7\nOneWeb|16|2022-01-01|1581.9\nOneWeb|18|2022-01-01|5745.6\nOneWeb|22|2022-01-01|4797.2\nOneWeb|23|2022-01-01|3067.1\nOneWeb|24|2022-01-01|247.5\nOneWeb|42|2024-01-01|1050.8\nOneWeb|39|2024-01-01|1552.8\nOneWeb|30|2024-01-01|5653.4\nOneWeb|21|2024-01-01|4228.9\nOneWeb|17|2024-01-01|3648.1\nOneWeb|15|2024-01-01|1526.9\nOneWeb|46|2024-01-01|1095.0\nOneWeb|48|2024-01-01|4606.5\nOneWeb|20|2030-01-01|702.8\nOneWeb|49|2024-01-01|327.1\nOneWeb|57|2024-01-01|410.4\nDGFI|ajisai|2018-01-11|1801.2\nDGFI|lageos1|2025-06-07|4509.7\nDGFI|lageos2|2025-07-12|3531.3\nJCET|etalon1|2012-05-12|2096.1\nJCET|etalon2|2012-05-12|4624.4\nNSGF|lares|2025-07-26|1553.1\nNSGF|lares2|2025-07-26|5006.4\nNSGF|larets|2025-07-26|1169.8\nNSGF|starlette|2025-07-25|1573.2\nNSGF|stella|2025-07-26|826.6\n\nAs shown, **OrbitZoo achieves errors under 1 km after 16 hours of propagation for several tested satellites**. However, satellites beyond the OneWeb constellation are subject to corrective forces (such as empirical accelerations) that Orekit does not model, which likely accounts for some of the observed discrepancies. **We have added this analysis to the revised version.**\n\n> W2. Is there any example of OrbitZoo's visualization capabilities?\n\nYes. In section 4.2. (in the Interface paragraphs), **we provide four different video links that showcase how OrbitZoo can be used** to provide a clear analysis of systems of bodies and evaluate policies in different kinds of missions. Additionally, there is a link to the anonymized code repository in appendix D (in the end of the introduction) containing a **step-by-step example on how to render a system, together with an “*example_visualization.py*” file**."}, {"review_id": "Rtl1elg25K", "reviewer": "Reviewer_8aNh", "summary": "This paper presents OrbitZoo, a multi-agent RL environment to control\nsatellites in earth's orbit. OrbitZoo adheres to the PettingZoo API for the\nmulti-agent RL and uses Orekit for the flight dynamics.\n\nThe paper discusses general challenges of multi-agent RL for orbital dynamics,\npresents OrbitZoo as a framework in this domain, and gives a few experiments\n(or rather demonstrations).\n\nIt is unclear what the research question was that is to be tackled by this\npaper. However, it mentions three contributions: (1) generation of\nhigh-fidelity orbital data (which is not done by OrbitZoo itself), (2) that it\nfollows PettingZoo for a standardized API and (3) that it can be configured in\na flexible way (e.g., arbitrary number of bodies, etc.) and features a 3D\nvisualization.\n\nIn a sense, this paper is a technical report on the design the OrbitZoo\nframework rather than a research paper.", "questions": "In §5.1 you say you have trained \"the agent\". What agent? How trained? How does\nyour framework lead to improvements compared to what baseline or state of the\nart?\n\nIn l.213 you construct a covariance matrix Σ. It is probably the diagonal\nmatrix formed by the sigmas mentioned before, but do not let the reader guess.\nSo what is it? You explain that this is used to sample position and velocity\nupon resetting the environment, but when is this again becoming relevant later\nin the paper? It seems like a random details.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 1, "strengths_and_weaknesses": "## Significance\n\nIt remains unclear in what sense this constitutes a novel, scientific\ncontribution to the field of RL, rather than an engineering artifact (which\nmight be very useful). The treatments in the paper are rather located in the\nfield of software engineering (e.g., architectural design, JSON configuration,\nUML-like diagrams, description of modules) and orbital mechanics simulation\n(e.g., representations of coordinates).\n\nThis also explains why the works in Table 1, to which this paper compares, are\nor centered around venues in the aforementioned venues and corresponding Master\nand PhD theses.\n\nHowever, the actual multi-agent RL problem setting of controlling a multitude\nof satellites is interesting.\n\n## Clarity\n\nIt is unclear how OrbitZoo per se solves the key challenges mentioned in §3, as\ndiscussed in §4. Saying that OrbitZoo uses Orekit is not a scientific\ncontribution, but a report.\n\nConsequently, it is also unclear how the experiments in §5 are experiments\naddressing a research hypothesis or a research question posed by the paper.\nWhen, for instance, the Hohmann Maneuver experiments runs satisfactory, how or\nin what sense is this attributed to the proposed framework OrbitZoo rather than\nthe components it orchestrates? How does it compare to alternative methods? Why\nwould your solution improve upon what by what mechanism? To sum up, the\nexperiments are rather demonstrations than experiments: They are not explaining\nor revealing insight to explicit questions concerning the prior discussion of\nyour framework.\n\n## Quality\n\nThis also becomes apparent in §5 (discussion). Demonstrating that RL policies\nwere trained in OrbitZoo is sold as a result of OrbitZoo rather than the\nunderlying RL algorithms. The use of continuous action spaces is mentioned as\nan insight to improve performance, but this is not really part of the treatment\nin this paper, which is about a framework. The final conclusion, that OrbitZoo\nwould pave the way to trustworthy AI-driven autonomy in future space missions\nis, well, blunt overselling.\n\n## Originality\n\nIt is unclear in what terms an originality of this work is given.\n\n## More comments\n\nIn §3.3 the exploration not only concerns the action space, but the joint\nstate-action space.\n\nIn §3.4 you should explain what \"constellation\" and \"propagation\" means, i.e.,\nadd context.\n\nIn §3.3, your argument that deterministic policies and Gaussian noise for\nexploration may be suboptimal sounds hand wavy.\n\nIn §3.5 you phrase \"realistic simulation environments, such as OrbitZoo\" is\nimprecise: It should be \"as _in_ OrbitZoo\", as this is actually done by Orekit,\nright? In fact, say that this is provided _by_ OrbitZoo is an argument on thin\nice.\n\nIn §4 you phrase \"to address the challenges\". Name the challenges, e.g., C1 to\nC4, and explicitly explain _how your framework_ addresses each C1 to C4. Also,\nwhat is \"vanilla OrbitZoo\"; it indicates that there is a vanilla version and an\nextended/patched version that could do more? This is not the case, but you\nattempt to sell the merits of the third-party components as merits of the\nOrbitZoo framework.\n\nIn §4 you speak of a \"Python environment\". Note that \"environment\" is now very\nambiguous as it could also be a (Python virtual) environment or a (RL)\nenvironment (in the sense of Barto Sutton) or a (simulation) environment (as\ndiscussed earlier). Be more precise.", "was_revised": true, "final_justification": "My remaining critique has very much been addressed in extensive discussions, including other reviewers. The remaining reasons why I am not going to accept is that still the application domain is a certain niche.", "rebuttal": "We sincerely thank you for your thoughtful review and for **highlighting the potential of multi-agent reinforcement learning in satellite control**. We especially appreciate your recognition that OrbitZoo could be **a valuable engineering contribution**, which is central to our goal.\n\nWhile we acknowledge the Datasets and Benchmarks (D&B) track, **our primary aim is to support exploration of fundamental RL challenges in orbital dynamics**, such as partial observability, coordination, communication, sparse rewards, fidelity-efficiency tradeoffs, and sim-to-real transfer. We believe this practical utility lays the groundwork for future scientific advances.\n\nWe agree that some sections of the paper could be clearer, and we have made changes to address several of your concerns. First, we outline those changes. Then, we respond to your specific questions, followed by a discussion of the identified weaknesses and remaining comments.\n___\n## Changes\n### Change 1\nL151 (Section 3.2) - We fix the typo at the end of the sentence: “(…) are explored in appendix A.”.\n### Change 2\nL173 (Section 3.3) - We now address the challenge of exploration in Section 3.2, highlighting the importance of a modular framework that supports various RL algorithms—not limited to continuous action spaces. We also specifically discuss joint state and action spaces in cooperative and adversarial multi-agent RL settings.\n### Change 3\nL179 (Section 3.4) - We explicitly define a constellation as a group of satellites cooperating to achieve a partial or complete collective objective, and refer readers to the propagation details in Appendix A.3 for further understanding.\n### Change 4\nL188 (Section 3.5) - We change the initial sentence to “Reinforcement learning environments with external components that provide high-fidelity dynamics, such as OrbitZoo, (…)”.\n### Change 5\nL194 (Section 4) - We revised the first sentence to: “OrbitZoo is a flexible Python RL environment designed to address (…)”. Challenges are referred to Section 3.2.\n### Change 6\nL213 (Section 4.1) - We rewrite the sentence as “(…) which are internally used to construct a diagonal covariance matrix $\\Sigma$ with these uncertainties as its diagonal entries.”.\n### Change 7\nL246 (Section 4.3) - We updated paragraphs to follow the new structure, moving “Realistic Thrust Modeling” under “Bridging the Reality Gap” and addressing scalability and computational trade-offs in “Multi-Agent Coordination.” We also highlight OrbitZoo’s capability to simulate FRL.\n### Change 8\nL277 (Section 5.1) - In the “Setup” paragraph, we briefly characterize what is the agent (a satellite with thrusting capabilities that interacts with the environment) and its strategy to apply thrusts (RL algorithm).\n___\n## Questions\n> Q1. In 5.1 you say you have trained \"the agent\". What agent? How trained? How does your framework lead to improvements compared to what baseline or state of the art?\n\nThe agent refers to **a thrust-capable satellite trained with PPO to reach a target orbit**. Due to limited context, **we made [Change 8](#change-8)**. As described in Appendix D.3, the agent learns to reach the target within a small tolerance. **The framework does not guarantee improvement over a baseline**, as results depend on factors like the **RL algorithm**, **dynamics realism**, **implementation**, and available **validation tools**. In the Hohmann maneuver, classical control offers a fuel-optimal solution under simplified, ideal assumptions. However, **RL agents adapt to perturbations and continuous thrust**, reaching the target under more realistic settings and **outperforming classical methods when ideal assumptions no longer hold**. We compare errors between the optimal maneuver (simplified dynamics) and five successful RL runs (Experiments 1 and 2, Table 9), across varying realism levels. The table reports the mean and standard deviation of fuel error (in kg):\nExp 1 No Perturbations|Exp 1 Perturbed|Exp 2 No Perturbations|Exp 2 Perturbed\n-|-|-|-\n$0.34_{(0.02)}$|$0.36_{(0.01)}$|$0.85_{(0.31)}$|$0.86_{(0.16)}$\n\nAdditionally, we compare the altitude error (in km) of those attempts by step:\nStep|Exp 1 No Perturbations|Exp 1 Perturbed|Exp 2 No Perturbations|Exp 2 Perturbed\n-|-|-|-|-\n**0**|$0.00_{(0.00)}$|$0.00_{(0.00)}$ | $0.00_{(0.00)}$|$0.00_{(0.00)}$\n1|$0.01_{(0.00)}$|$0.01_{(0.00)}$ $0.01_{(0.01)}$|$0.01_{(0.00)}$\n2 |$0.03_{(0.00)}$|$0.03_{(0.00)}$|$0.02_{(0.02)}$|$0.02_{(0.01)}$\n3 |$0.07_{(0.00)}$|$0.06_{(0.01)}$|$0.04_{(0.04)}$|$0.04_{(0.01)}$\n4 |$0.11_{(0.01)}$|$0.10_{(0.01)}$|$0.06_{(0.06)}$|$0.06_{(0.01)}$\n...|...|...|...|...\n553|$2.57_{(1.51)}$|$10.13_{(1.66)}$|$2.43_{(2.07)}$|$8.10_{(3.60)}$\n554|$2.61_{(1.51)}$|$10.13_{(1.66)}$|$2.35_{(2.07)}$|$8.07_{(3.58)}$\n555|$2.65_{(1.51)}$|$10.12_{(1.67)}$|$2.28_{(2.07)}$|$8.04_{(3.55)}$\n...|...|...|...|...\n**765**|$10.73_{(1.30)}$|$5.10_{(1.12)}$|$17.49_{(2.97)}$|$5.66_{(4.63)}$\n766|$10.78_{(1.29)}$|$5.05_{(1.12)}$|$17.59_{(2.97)}$|$5.73_{(4.63)}$\n767|$10.83_{(1.29)}$|$5.00_{(1.11)}$|$17.69_{(2.98)}$|$5.80_{(4.63)}$\n...|...|...|...|...\n998|$16.30_{(0.96)}$|$3.74_{(0.75)}$|$30.74_{(3.53)}$|$13.33_{(6.08)}$\n999|$16.30_{(0.96)}$|$3.76_{(0.75)}$|$30.74_{(3.53)}$|$13.32_{(6.09)}$\n\nBoth experiments reach the target orbit using less than 1 kg of additional fuel on average. This last table highlights thrust steps (see Figure 19), with bold marking optimal thrusts. Experiment 1 yields lower final altitude error, while Experiment 2 shows greater trajectory variance—suggesting a deeper understanding of the dynamics. These findings also indicate that shaping the reward to emphasize the semi-major axis could improve performance. **This analysis has been added to the revised version**.\n> Q2. In l.213 you construct a covariance matrix Σ. It is probably the diagonal matrix formed by the sigmas mentioned before, but do not let the reader guess. So what is it? You explain that this is used to sample position and velocity upon resetting the environment, but when is this again becoming relevant later in the paper?\n\nPrecisely. The covariance matrix $\\Sigma$ uses uncertainties as diagonal entries. We thank the reviewer for pointing this out, which led to [Change 5](#change-5). Sampling from this multivariate normal defines the body’s initial state, hence the importance of mentioning it. In the collision avoidance mission (Appendix D.5), this sets the miss distance at time of closest approach (TCA), necessary for the agent to experience slightly different scenarios during training.\n___\n## Strengths and Weaknesses\n### Significance\nAs mentioned earlier, OrbitZoo was designed to **advance RL research in orbital dynamics** by offering a **standardized and accessible framework**.\n### Clarity and Quality\nAs addressed in your first question, **the results depend on external components beyond OrbitZoo itself**, which may have impacted clarity in some sections.\n### Originality\nWhile somewhat subjective, **OrbitZoo offers a unique interactive 3D interface** for visualizing multi-body dynamics and mission strategies — **a distinctive validation feature absent in other environments**. Additionally, **Orekit developers have expressed interest in the OrbitZoo framework**.\n### Additional Comments\n> C1. In 3.3 the exploration not only concerns the action space, but the joint state-action space.\n\nNow we address joint state and action spaces in **Section 3.2**, as [Change 2](#change-2) states.\n> C2. In 3.4 you should explain what \"constellation\" and \"propagation\" means, i.e., add context.\n\n**We clarify the terms *constellation* and *propagation*** in that context, reflecting [Change 3](#change-3).\n> C3. In 3.3, your argument that deterministic policies and Gaussian noise for exploration may be suboptimal sounds hand wavy.\n\nAs noted in [Change 2](#change-2), **we now highlight the importance of a modular framework** **supporting various RL algorithms**, not just those with continuous actions.\n> C4. In 3.5 you phrase \"realistic simulation environments, such as OrbitZoo\" is imprecise: It should be \"as in OrbitZoo\", as this is actually done by Orekit, right?\n\nYes, **propagation is handled by Orekit, not OrbitZoo**, as noted throughout the paper (e.g., Figure 1, L11, L53, L60, L121, L248). Orekit propagates bodies but **does not provide a standardized RL environment**. It lacks methods for applying thrusts based on immediate policies, though it supports pre-defined thrusts via event detectors. OrbitZoo builds on this by dynamically adding/removing thrust during propagation. **[Change 4](#change-4) reinforces this distinction**.\n> C5. In 4 you phrase \"to address the challenges\". Name the challenges, e.g., C1 to C4, and explicitly explain how your framework addresses each C1 to C4. Also, what is \"vanilla OrbitZoo\"; it indicates that there is a vanilla version and an extended/patched version that could do more?\n\n**[Change 5](#change-5) and [Change 7](#change-7) address the challenges you've raised.**  \n*Vanilla* OrbitZoo refers to the environment in Figure 1, **without a built-in RL implementation**, allowing flexibility for different mission types. It supports multi-body setups, thrust, perturbations, and various propagation settings, enabling control strategy design, data generation, or dynamics analysis. OrbitZoo is easily extensible for RL or MARL missions by integrating custom algorithms, rewards, training routines, and communication protocols, as shown.  \n**We do not attempt, *in any way*, to sell the merits of third-party components**, as clarified earlier.\n> C6. In 4 you speak of a \"Python environment\". Note that \"environment\" is now very ambiguous as it could also be a (Python virtual) environment or a (RL) environment (in the sense of Barto Sutton) or a (simulation) environment (as discussed earlier).\n\nWe thank you for your suggestion. **We have rephrased that sentence to be more precise**, as seen in [Change 5](#change-5).\n___\n## Paper Formatting Concerns\n[Change 1](#change-1) aims to correct the typo."}]

Author rebuttal:

Meta-review: The initial reviews were very mixed (ratings 1–5), with concerns expressed as to whether this was more of a technical report than a scientific paper.
After extensive discussion and due to the extensive revisions made by the authors, it is now clear that the revised manuscript OrbitZoo is positioned as a research tool for investigating fundamental RL challenges in physically grounded environments. 
With these changes, the work makes a valuable contribution to the RL community by providing a modular, high-precision environment for investigating multi-agent coordination in physically grounded environments.

I agree with the majority of the reviewers and recommend accepting the manuscript.
Decision: Accept (poster)