Fix bug in remove() where deferred was unset