Before revision:  This value function is learned through off-policy reinforcement learning, using trajectory optimization as its behavior policy. Furthermore, we introduce "actor-guided" trajectory optimization to mitigate the actor-divergence issue in the proposed method. We benchmark our methods on continuous control tasks and demonstrate that it offers a significant improvement over the underlying model-based and model-free algorithms  .
After revision:  This value function is learned through off-policy reinforcement learning, offline reinforcement learning, and safe learning. We show that this method significantly improves the underlying model-based and model-free algorithms .