Fixes #1526: Only remove when close/cancel/destroy gets called