Free device memory only when operator is destructed.