Avoid calling purple_input_remove twice