Remove handle when it can't be swiped away