Using Reinforcement Learning to Design and Control Free-Flying Space Robots
Creators: name: Jackson, Lucy Elaine
Publisher: name: University of Surrey
Year: 2023
Subjects: Reinforcement Learning
Free-Flying Space Robot
On-Orbit Assembly
Robotic Design

description: Free-flying space robots have the potential to revolutionise space exploration by facilitating a
description: range of on-orbit operations. Whilst there have been some successful demonstrations of space
description: robot technologies, the systems remain large with mission concepts limited to rendezvous with
description: co-operative targets. However, the recent surge in small satellite technologies is changing the
description: economics of space and downsizing a space robot is now viewed as a technologically feasible
description: aim. Even with these advances, the design and control of a free-flying space robot is very
description: challenging. Without a fixed base, the robot experiences large dynamic coupling effects where
description: every motion causes counter-motions in other parts of the robot. These are influenced most
description: heavily by the space robot’s size and mass distribution as well as the employed control technique.
description: As such, successful operation is heavily tied to both physical design and control. In fact, this
description: coupling underpins all robotic systems since control and hardware design are intrinsically linked.
description: This thesis therefore investigates the relationship between hardware and control, looking at
description: how they impact each other and how this relationship can be exploited to improve performance.
description: Although this work uses the On-Orbit Assembly (OOA) of a large aperture space telescope
description: using free-flying space robots as a mission concept, the methods developed have applicability
description: across the entire field of robotics.<br>
description: <br>
description: There exists no documented design technique for any space robot. In fact, the dimensions and
description: final design of successful missions are hard to find and never substantiated. This makes it hard
description: to understand how robotic design impacts performance. This thesis presents a transparent design
description: approach specific to free-flying space robots. A combined engineering approach is proposed,
description: considering the mechanics and dynamics of the integrated system in order to present a design
description: best suited for the OOA of a large aperture telescope. The feasibility of this space robot is
description: evaluated in simulation and in-depth results are presented. This analysis supports the hypothesis
description: that a small space robot is a viable solution for OOA and should be used as a starting point for
description: the design of future systems.<br>
description: <br>
description: As with the majority of robot design techniques, the initial hardware centered design approach in
description: this thesis treats the control as immutable. Instead, simultaneous optimisation of hardware and
description: control parameters is likely to improve overall performance. This is demonstrated by reasoning
description: over both control and design in a Reinforcement Learning (RL) pipeline. The proposed method
description: forms a complex differential path through a trajectory rollout, allowing a vast amount of
description: information that was previously lost in the ‘black-box’ environment to be used. This means
description: refinements can be made to both the morphology and control parameters simultaneously. The
description: result is an efficient and versatile approach to holistic robot design. Performance improvements
description: are seen with the space robot and a number of benchmark tasks compared to just performing
description: hardware design optimisation.<br>
description: <br>
description: While it is possible to modify robotic design in conjunction with control to improve performance,
description: in some instances the design maybe changed with no option to modify the control algorithm.
description: Data scarcity, brittle convergence and the gap between simulation &amp; real world environments
description: mean that most common RL approaches are subject to overfitting and fail to generalise to unseen
description: environments. This means the replacement of parts with nonidentical components or the failure
description: of sensors or joints will most likely render the control algorithm useless. Hardware-agnostic
description: policies would mitigate this by allowing a single network to operate in a variety of test domains,
description: where dynamics vary due to changes in robotic morphologies. This thesis utilises the idea
description: that learning to adapt a known and successful control policy is easier and more flexible than
description: jointly learning numerous policies for different morphologies. It presents the idea of Hardware
description: Agnostic RL. In this approach, two control polices are combined and varied embodiments are
description: sampled using a novel adversarial loss function. This self-regulates morphologies based on their
description: performance. The result is a final control policy that is robust to changes in the environment as
description: well as degradation and failure of the robot.<br>