Remove reference cycles in LinearOperators et.al to avoid memory
blows.

This commit removes reference cycles in rmatvec

These attributes cause a blowup in memory when called repeatedly.