ENH: make apply_inverse realtime friendly