Implement IdleTaskManager::remove