IrcBufferModel: do not block IrcBuffer signals on destruction/removal

Be conservative and only disconnect signals to itself - let apps handle
for example destroyed() if they will.