Only remove the listeners we bound