Title: OrbitZoo: Real Orbital Systems Challenges for Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Agent Reinforcement Learning

Orbital Dynamics

Data Generation
Abstract: The increasing number of satellites and orbital debris has made space congestion a critical issue, threatening satellite safety and sustainability. Challenges such as collision avoidance, station-keeping, and orbital maneuvering require advanced techniques to handle dynamic uncertainties and multi-agent interactions. Reinforcement learning (RL) has shown promise in this domain, enabling adaptive, autonomous policies for space operations; however, many existing RL frameworks rely on custom-built environments developed from scratch, which often use simplified models and require significant time to implement and validate the orbital dynamics, limiting their ability to fully capture real-world complexities.  To address this, we introduce OrbitZoo, a versatile multi-agent RL environment built on a high-fidelity industry standard library, that enables realistic data generation, supports scenarios like collision avoidance and cooperative maneuvers, and ensures robust and accurate orbital dynamics. The environment is validated against various real satellite constellations, including Starlink, achieving a Mean Absolute Percentage Error (MAPE) of 0.16% compared to real-world data. This validation ensures reliability for generating high-fidelity simulations and enabling autonomous and independent satellite operations. This project is open source and has a dedicated project page.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents OrbitZoo, a multi-agent RL environment to control\nsatellites in earth's orbit. OrbitZoo adheres to the PettingZoo API for the\nmulti-agent RL and uses Orekit for the flight dynamics.\n\nThe paper discusses general challenges of multi-agent RL for orbital dynamics,\npresents OrbitZoo as a framework in this domain, and gives a few experiments\n(or rather demonstrations).\n\nIt is unclear what the research question was that is to be tackled by this\npaper. However, it mentions three contributions: (1) generation of\nhigh-fidelity orbital data (which is not done by OrbitZoo itself), (2) that it\nfollows PettingZoo for a standardized API and (3) that it can be configured in\na flexible way (e.g., arbitrary number of bodies, etc.) and features a 3D\nvisualization.\n\nIn a sense, this paper is a technical report on the design the OrbitZoo\nframework rather than a research paper.\n\nSTRENGTHS AND WEAKNESSES:\n## Significance\n\nIt remains unclear in what sense this constitutes a novel, scientific\ncontribution to the field of RL, rather than an engineering artifact (which\nmight be very useful). The treatments in the paper are rather located in the\nfield of software engineering (e.g., architectural design, JSON configuration,\nUML-like diagrams, description of modules) and orbital mechanics simulation\n(e.g., representations of coordinates).\n\nThis also explains why the works in Table 1, to which this paper compares, are\nor centered around venues in the aforementioned venues and corresponding Master\nand PhD theses.\n\nHowever, the actual multi-agent RL problem setting of controlling a multitude\nof satellites is interesting.\n\n## Clarity\n\nIt is unclear how OrbitZoo per se solves the key challenges mentioned in §3, as\ndiscussed in §4. Saying that OrbitZoo uses Orekit is not a scientific\ncontribution, but a report.\n\nConsequently, it is also unclear how the experiments in §5 are experiments\naddressing a research hypothesis or a research question posed by the paper.\nWhen, for instance, the Hohmann Maneuver experiments runs satisfactory, how or\nin what sense is this attributed to the proposed framework OrbitZoo rather than\nthe components it orchestrates? How does it compare to alternative methods? Why\nwould your solution improve upon what by what mechanism? To sum up, the\nexperiments are rather demonstrations than experiments: They are not explaining\nor revealing insight to explicit questions concerning the prior discussion of\nyour framework.\n\n## Quality\n\nThis also becomes apparent in §5 (discussion). Demonstrating that RL policies\nwere trained in OrbitZoo is sold as a result of OrbitZoo rather than the\nunderlying RL algorithms. The use of continuous action spaces is mentioned as\nan insight to improve performance, but this is not really part of the treatment\nin this paper, which is about a framework. The final conclusion, that OrbitZoo\nwould pave the way to trustworthy AI-driven autonomy in future space missions\nis, well, blunt overselling.\n\n## Originality\n\nIt is unclear in what terms an originality of this work is given.\n\n## More comments\n\nIn §3.3 the exploration not only concerns the action space, but the joint\nstate-action space.\n\nIn §3.4 you should explain what \"constellation\" and \"propagation\" means, i.e.,\nadd context.\n\nIn §3.3, your argument that deterministic policies and Gaussian noise for\nexploration may be suboptimal sounds hand wavy.\n\nIn §3.5 you phrase \"realistic simulation environments, such as OrbitZoo\" is\nimprecise: It should be \"as _in_ OrbitZoo\", as this is actually done by Orekit,\nright? In fact, say that this is provided _by_ OrbitZoo is an argument on thin\nice.\n\nIn §4 you phrase \"to address the challenges\". Name the challenges, e.g., C1 to\nC4, and explicitly explain _how your framework_ addresses each C1 to C4. Also,\nwhat is \"vanilla OrbitZoo\"; it indicates that there is a vanilla version and an\nextended/patched version that could do more? This is not the case, but you\nattempt to sell the merits of the third-party components as merits of the\nOrbitZoo framework.\n\nIn §4 you speak of a \"Python environment\". Note that \"environment\" is now very\nambiguous as it could also be a (Python virtual) environment or a (RL)\nenvironment (in the sense of Barto Sutton) or a (simulation) environment (as\ndiscussed earlier). Be more precise.\n\nQUESTIONS:\nIn §5.1 you say you have trained \"the agent\". What agent? How trained? How does\nyour framework lead to improvements compared to what baseline or state of the\nart?\n\nIn l.213 you construct a covariance matrix Σ. It is probably the diagonal\nmatrix formed by the sigmas mentioned before, but do not let the reader guess.\nSo what is it? You explain that this is used to sample position and velocity\nupon resetting the environment, but when is this again becoming relevant later\nin the paper? It seems like a random details.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-18"}

comments: SUMMARY:
This paper proposes a new multi-agent RL environment for orbital dynamics called OrbitalZoo. OrbitalZoo can support data generation/simulation, RL algorithm research and visualization, etc. It addresses some key challenges in this area, including Bridging the Reality Gap, Realistic Thrust Modeling, Multi-Agent Coordination, and Interactive Visualization for Debugging and Analysis. Experimental results including single agent, multi-agent as well as real world data validation show OrbitZoo as a reliable benchmark for RL-based autonomy in space operations.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well written.
2. It seems OrbitZoo supports a variety of RL algorithms and experiments provide comparison among these algorithms for the same task.
3. Various of interesting tasks related to orbital dynamics are included in this benchmark, such as chase target, collision avoidance, Hohmann Maneuver, etc.

Weakness:
1. regarding "Bridging the Reality Gap." it seems it is only supported by the Starlink example, is there any other evidence to show OrbitZoo can bridge the reality gap better than other environments?
2. Is there any example of OrbitZoo's visualization capabilities?

QUESTIONS:
1. What's the plan for open source?
2. What is the baseline algorithm? Compared w/ RL algorithms, is RL much better than baseline algorithm?
3. Is the MARL support general to all RL algorithms? Or does it only work for PPO?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces OrbitZoo, a novel multi-agent reinforcement learning (MARL) environment for satellite operations grounded in high-fidelity orbital dynamics. Built atop the Orekit library, OrbitZoo supports realistic modeling of gravitational, atmospheric, and third-body forces. It integrates with PettingZoo to provide MARL capabilities and supports interactive 3D visualization. The system is evaluated through tasks such as Hohmann transfers, constellation coordination, and validation against Starlink ephemerides.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe integration of Orekit ensures industry-grade fidelity in orbital simulation, a clear advancement over simplified environments.\nThe inclusion of multi-agent support, real-world data validation, and realistic perturbations within a single open-source platform is a strong contribution. Enables reproducible, high-fidelity MARL experimentation in the space domain, with applications in satellite traffic management and autonomous operations. Overall well-written, with informative figures and detailed architectural breakdowns.\n\nWeakness:\nThe RL training code is not part of the core environment, limiting direct reproducibility.\nExperiments are illustrative but not benchmarked against baseline policies or environments quantitatively.\nReal-time applications, potential safety concerns, and fail-safe guarantees are not addressed in depth.\n\nQUESTIONS:\nCan the authors provide more quantitative comparisons with existing RL environments or classical control approaches to establish baseline performance?\n\nHow scalable is OrbitZoo in multi-agent settings beyond 4–5 satellites in practice?\n\nCould the interface support integration with onboard learning frameworks for spacecraft? Does the environment offer support for simulating noisy sensor models or communication constraints?\n\nWhat measures are taken to ensure physical plausibility of agent-generated trajectories under unbounded thrust policies?\n\nLIMITATIONS:\nThe limitations are reasonably well-discussed. However, implications of applying such agents in real satellite systems under uncertainty or adversarial conditions could be elaborated further.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces OrbitZoo, an environment designed for high-fidelity orbital data generation and RL development. The main contribution of this environment includes high-fidelity orbital data generation built on Python, supporting multi-agent reinforcement learning and modular design for customization and visualization. The authors also present some experiments to validate effectiveness of OrbitZoo.

STRENGTHS AND WEAKNESSES:
The paper is well organized and easy to follow. The main issue of this paper to me is that this paper feels more like a technical report on a specialized tool for studying orbital dynamics. There is no general benchmarking results or new development of RL algorithms. As such, NeurIPS is probably not the best venue for this work and its contribution to general ML community is limited. It's likely that a more domain-specific conference would be more appropriate.

QUESTIONS:
See above

LIMITATIONS:
See above
TITLE: Official Review
CLARITY: 2
IMPACT: 1
ORIGINALITY: 2
DATE: 2025-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article introduces OrbitZoo, a modular, high-fidelity, and multi-agent reinforcement learning (MARL) environment for orbital dynamics. Built on top of the industry-standard Orekit library and interfaced via Python, OrbitZoo enables realistic satellite simulations, including perturbative forces and customizable thrust models. The framework integrates with the PettingZoo MARL API and provides interactive 3D visualization. Empirical demonstration of its capabilities through various experiments, including single-agent Hohmann transfers, multi-agent constellation coordination, and validation against Starlink ephemeris data. The paper aims to address the lack of standardized, high-fidelity environments for applying reinforcement learning to realistic space scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The work addresses a significant and timely gap in the reinforcement learning community by providing a standardized, high-fidelity, and publicly accessible environment for satellite operations.\n-  OrbitZoo builds upon the validated capabilities of Orekit, ensuring accurate modeling of orbital propagation, including atmospheric drag, solar radiation pressure, and third-body effects.\n- The modular architecture, integration with the standard library PettingZoo, and availability of interactive visualization are important assets.\n- The validation using real Starlink ephemerides empirically demonstrates the applicability of the simulation engine to real-world scenarios.\n- Several experimental case studies illustrate the flexibility of the environment across different classes of orbital control problems, including both single-agent and multi-agent settings.\n- The appendices contain a significant number of additional contributions (e.g., computational performance benchmarks, comparison with state-of-the-art environments, detailed mission setups) that could benefit from greater visibility in the main text.\n\nWeaknesses:\n- The taxonomy introduced in Section 2 lacks conceptual coherence, mixing levels of abstraction and including overlapping or loosely defined categories.\n- Section 3.2 aims to discuss challenges in MARL for orbital dynamics but largely describes system components without framing them as reinforcement learning challenges (e.g., reward sparsity, partial observability, exploration complexity).\n- While the use of Orekit introduces realism, the trade-off between physical fidelity and computational cost is insufficiently discussed in the main text. Appendix C provides relevant data that should be included in the main paper.\n- Many core features of OrbitZoo, such as agent observation and action space definitions, configurable parameters, and default settings, are only briefly touched upon or deferred to supplementary material.\n- Section 5.1 introduces an “RL agent” but omits details about the specific algorithm used. More generally, the empirical results prioritize breadth over depth: while multiple use cases are presented, each is described briefly, limiting insight into policy behavior, convergence, or comparative performance.\n- Federated learning appears in Section 5 without prior mention or sufficient explanation of its implementation or relevance within the OrbitZoo framework.\n- No discussion is provided on multi-objective formulations, despite their clear relevance for many orbital control problems and knowing the growing importance of MORL and MOMARL.\n- Too much of the most valuable technical content, including benchmark comparisons, advanced scenario configurations, and realism validation, is relegated to the appendix, reducing its impact.\n- (Detail): Several criteria in Table 1 are undefined (e.g., “continuous control”) and the terminology is inconsistent (e.g., \"open-source\" vs. \"publicly available\").\n\nQUESTIONS:\nThis is a higher number of questions than expected, but these are rather short and well-targeted based on the weaknesses listed above:\n\n1. Related work taxonomy: What principles guided the choice and separation of the four categories in Section 2? Could the classification be reorganized to improve clarity and avoid overlap?\n2. Feature specification: The main text lacks a detailed overview of OrbitZoo's configurable parameters and environment structure (e.g., thrust constraints, observation/action spaces, default models). Could this be summarized in a table or example use case?\n3. Challenge articulation (Section 3.2): The listed challenges often read as system properties rather than research obstacles. Could the authors reframe these to highlight RL-relevant aspects (e.g., uncertainty, scalability, delayed rewards)?\n4. Fidelity-performance trade-off: The choice of Orekit implies significant computational cost. Appendix C benchmarks this, but this critical trade-off is omitted from the main discussion. Could a summary be added to Section 4 or 5?\n5. Experiment scope vs. depth: The experiments aim to cover a broad range of missions but are presented succinctly, often without learning curves or ablation details. Are these meant to be illustrative use cases or benchmarkable evaluations? Could this section be reorganised?\n6. Federated learning: How is federated learning implemented in OrbitZoo, and what experimental configuration is used in Section 5.2? This concept appears abruptly and is not clearly contextualized.\n7. Multi-objective RL: Have the authors considered extending the environment or its reward structure to support multi-objective optimization? This is especially relevant in scenarios involving fuel efficiency, time, and risk trade-offs.\n8. Appendix content: Could selected content from Appendix D (e.g., D.2 comparison, D.6 validation) be promoted to the main paper? These elements are essential to support the framework’s claims of realism and generality.\n\nLIMITATIONS:\nThe manuscript does not very explicitly discuss limitations or potential risks. While the primary use case is scientific, autonomy in orbital environments raises questions around safety, robustness, and possible misuse. A brief discussion of intended scope, deployment assumptions, or responsible use (e.g., civilian vs. defense) would be appropriate. Scalability to large constellations or real-time applications is also not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}