Question: Perhaps the conceptual novelty is limited?
Evidence: Deep reinforcement learning (RL) has achieved many successes with human-or superhuman-level performance across a wide range of complex domains. However, all these major breakthroughs focus on finding the bestperforming strategy by trial-and-error interactions with a single environment, which poses severe constraints for scenarios such as healthcare, autonomous driving and controlled-environment agriculture where safety is paramount. Moreover, these RL algorithms require tremendous explorations and training samples, and are also prone to over-fitting to the target task, resulting in poor generalization and robustness. To make RL truly practical in many real-world applications, a new paradigm with better safety, sample efficiency and generalization is in need.
Offline meta-RL, as a marriage between offline RL and meta-RL, has emerged as a promising candidate to address the aforementioned challenges. Like supervised learning, offline RL restricts the agent to solely learn from fixed and limited data, circumventing potentially risky explorations. Additionally, offline algorithms are by nature off-policy, which by reusing prior experience, have proven to achieve far better sample efficiency than on-policy counterparts.
Meta-RL, on the other hand, exploits the shared structure of a distribution of tasks and enables the agent to adapt to new tasks with minimal data. One popular approach is by learning a single universal policy conditioned on a latent task representation, known as context-based method. Alternatively, the shared skills can be learned with a meta-controller.
In this work we restrict our attention on context-based offline meta-RL (COMRL), an understudied framework with a few existing algorithms, for a set of tasks that differ in reward or transition dynamics. One major challenge associated with this scenario is termed Markov Decision Process (MDP) ambiguity, namely the task-conditioned policies spuriously correlate task identity with state-action pairs due to biased distribution of the fixed datasets. This phenomenon can be interpreted as a special form of memorization problem in classical meta-learning, where the value and policy functions overfit the training distributions without capturing causality from reward and transition functions, often leading to degenerate task representations and poor generalization. To alleviate such over-fitting, proposes a framework named FOCAL which decouples the learning of task inference from control by using self-supervised distance metric learning. However, they made a strong assumption on the existence of an injective map from each transistion tuple {s, a, s , r} to its task identity. Under extreme scenarios such as sparse reward, where a considerable portion of aggregated experience provides little information regarding task identity, efficient and robust learning of task representations is still challenging.
To address the aforementioned problem, in this paper we propose intra-task attention mechanism and inter-task contrastive learning objectives to achieve robust task inference. More specifically, for each task, we apply a batch-wise gated attention to recalibrate the weights of transition samples, and use sequence-wise self-attention to better capture the correlation within the transition (state, action, reward) dimensions. In addition, we implemented a matrix-form objective of the Momentum Contrast (MoCo) for task-level representation learning, by replacing its dictionary queue with a meta-batch sampled on-the-fly. We provide theoretical analyses showing that our objective serves as a better surrogate than naive contrastive loss for task inference and the proposed attention mechanism on top can also reduce the variance of task representation. Moreover, empirical evaluations demonstrate that the proposed design choices of attention and contrastive learning mechanisms not only boost the performance of task inference, but also significantly improve its robustness against sparse reward and distribution shift. We name our new method FOCAL++.
To make RL truly practical in many real-world applications, a new paradigm with better safety, sample efficiency and generalization is in need.
Answer: We have stressed and clarified our key contribution. Please check our General Response 1.We focus on improving the generalization and transfer learning ability of the current offline meta-RL methods, which is extremely important component of intelligence.