Fix a bug where freeInbound/OutboundBuffer() is not called when a handler is removed from a pipeline.