this can exhibit different and different instabilities one idea for trying to mitigate this is really the dumbest idea you can come up with is to assume that the Jacobian of Phi with respect to theta is identity and simply copy the gradient with this rectify to be the gradient with respect to theta and this actually works somewhat surprisingly well oddly enough on relatively simple problems although anecdotally we found it not to work well as you try to move towards more complex problems like imitation learning and reinforcement