Unset ->source when removing a handler from the chain

Otherwise this reference may outlive the buffer and
cause the chain to get destroyed out of order.