Arbitrary image resampling (a la `drizzle`) has been on the TODO list for matplotlib for a long time.  I'd love to find the time to do it (or find someone else's time to do it).  In the meantime, I think, as @leejjoon points out, our best bets are to approximate with affines or use pcolormesh if the transform is too non-linear.