Fix a bug where the buffers are freed too early when a handler is removed or replaced