Optimising ConOps of Water Electrolysis Propulsion systems using Deep Q-Learning
Creators: name:  SASCHA DENGLER 
name:   ADRIAN WAGMANN Y OTERO 
name:   CHIARA MANFLETTI 
Publisher: name: Proceedings of the Aerospace Europe Conference - EUCASS - CEAS - 2023
Year: 2023
Subjects: WATER ELECTROLYSIS PROPULSION
CONOPS
AI
REINFORCEMENT LEARNING
IN-SPACE PROPULSION

description: This work explores the use of deep reinforcement learning to derive and optimise the Concept of Operations (ConOps) for Water Electrolysis Propulsion (WEP) systems. WEP systems produce propellants on demand by decomposing liquid water into gaseous hydrogen and oxygen using an electrolyser. The added complexity of the propellant production must be considered for manoeuvre planning as well as for control and operation of the components themselves. An approach to optimising this ConOps using a Deep-Q-Network that controls a simulated spacecraft propelled by a WEP system is derived. The framework developed for this purpose is described and a Proof-of-Concept (PoC) on a reference mission scenario is conducted. 1. Motivation 1.1 Introduction to Water Electrolysis Propulsion In the pursuit for high-performance green propellants, Water Electrolysis Propulsion is one of the most promising technologies. A spacecraft filled with pure, liquid water guarantees easy handling during ground operations and potentially reduces cost compared to highly-toxic alternatives. Once in orbit, the water is decomposed into GH2 and GO2 by an electrolyser that is powered by solar cells. When being used in a chemical thruster, this propellant combination provides a high theoretical specific Impulse (Isp) of ca. 400 seconds and has the potential to outperform other storable chemical propulsion solutions. For certain mission profiles, it could be shown that WEP can be more mass-efficient than existing chemical propulsion solutions [1]. Water availability on other celestial bodies such as asteroids, the Moon, and planetary bodies, and the possibility of mining this resource and refilling spacecraft in-situ further increases the attractiveness of water-based propulsion solutions. These advantages however come at the cost of added complexity due to the on-board propellant production and intermediate storage. The time needed to produce a required amount of propellant must be considered and planned for and a suitable supply of electrical power needs to be ensured. This process should make optimal use of the individual components operational envelopes, as well as their interplay with each other and with other subsystems of the spacecraft, to achieve high systemic efficiency. Additionally, this more complex operation of the propulsion system should not fall back to the operator of the spacecraft. Both aspects could be an inhibitor to a broader use of the technology, despite the performance advantages mentioned. To tackle these challenges, an automated optimisation process for the Concept of Operation (ConOps) of WEP systems is needed. Such an optimiser should be capable of providing an ideal scheduling of production and manoeuvres as well as an optimal control approach within these processes. Ideally, a level of adaptability of the operations to changing environmental factors and self-diagnosis capabilities are present to enable autonomous functionalities at a later stage, further simplifying operations and improving usability. Reviewing possible solution approaches, reinforcement learning (RL) provides promising advances in optimal control problems in recent years. To pursue a ConOps optimiser based on RL with the aforementioned characteristics, a framework is implemented and described in this work. Making use of this framework, an initial Proof-of-Concept (PoC) on a simple application scenario is achieved and limitations found are discussed. Finally, an outlook on next steps and topics that emerged from this work is given.