Improve rmi to delete only when necessary

Fixes #610