Reinforcement learning-based motion planning in partially observable environments for complex tasks
Creators: name: Li, Junchao
Publisher: name: University of Iowa
Year: 2024
Subjects: deep learning
motion planning
partially observable Markov decision process
reinforcement learning
temporal logics
Robotics

description: Partially Observable Markov Decision Processes (POMDPs) have been used as mathematical models for sequential decision-making under uncertain and incomplete information. Since the state space is partially observable in a POMDP, the agent has to make a decision based on the integrated information over the past experiences of actions and observations. This dissertation aims to solve probabilistic motion planning problems in which the agent is assigned a complex task under a partially observable environment. We employ Linear Temporal Logic (LTL) to formulate the complex task and then convert it to a Limit-Deterministic Generalized Büchi Automaton (LDGBA). We formulate the problem as finding an optimal policy on the product of POMDP and LDGBA based on model-checking techniques. This dissertation also studies both model-based and model-free Reinforcement Learning (RL) algorithms as the approaches to solve such optimal policy.
description: Following the literature review in Chapter 1 and the introduction of RL in Chapter 2, Chapter 3 adopts two model-based RL approaches to approximate the value function of the belief state space in order to obtain the optimal policy on the corresponding product POMDP domain. Both approaches are model-based because the transition information of the underlying Markov Decision Process (MDP) is needed for the belief state update.
description: Inspired by the limitation of the model-based approaches employed in Chapter 3, a model-free approach is proposed in Chapter 4 by implementing Recurrent Neural Networks (RNNs) into Q-network architectures to directly process the agent's observation histories and task recognition without using the model knowledge. Two scenarios are considered, depending on whether the agent fully recognizes the LTL-induced automaton. The combination of observation histories with the automaton states is used as the input features to Q-networks as the agent previously acquired the automaton knowledge. Otherwise, the state labels with observations are utilized. Both settings give the agent a good sense of the problem goal for its decision-making.
description: The proposed model-free approach was then implemented to address the AI ethical decision-making in motion planning problems for complex tasks in partially observable environments. In Chapter 5, the developed approach is leveraged to model the ethical norms into `hard' and `soft' ethical constraints, where the formal logic language LTL is utilized to express the `hard' constraints, and a reward redesign method is applied to synthesize the `soft' ethical constraints. The innovated RNN-based Deep Q-learning approach is employed to solve such ethical decision-making problems. The simulation results showed its potential applicability to other scenarios and challenges in the field of ethical decision-making.
description: Chapter 6 provides a comprehensive summary of the proposed approaches for AI decision-making, with a particular focus on motion planning. Additionally, it provides a detailed comparative analysis, exploring the performance and limitations of these approaches relative to established models in the academic literature.