Before revision:  In this paper, we propose a counterfactual off-policy method to learn on a better synthesis of data. It takes advantage of a real responseto infer an alternative that was not takenusing a structural casual model. Learning on the counterfactual responses  helps to explore the high-reward area of the  response space.
After revision:  In this paper, we propose to explore potential responses by counterfactual reasoning. Given an observed response, the counterfactual reasoning model automatically infers the outcome of an alternative policy that could have been taken. The resulting counterfactual response synthesized in hindsight is of higher quality than the response synthesized from scratch. Training on the counterfactual responses helps to explore the high-reward area of the response space.