The main "difficulty" is to compute the pre-multiplication matrices, because they're different for backward and forward warpings (that's because, in general, the input and output grids and their corresponding grid-to-space transformations are different). Let me think of a way of simplifying them, meanwhile I opened an issue on my dipy fork.