Correctly remove handler when destroy