Correctly handle the case when a user remove the handler from the pipeline but there are still bytes in the pipeline and messages to process