Make sure we remove on resume as well