Fix removal of r/w-notifiers if another notifier is still bound

I think this is a regression causd by c412687. That commit adds, a
wrapper for the callback provided by the user. The workaround supplied
by 43256c0 is not sufficient, and this commit aims to fix this.

A common situation to trigger this is the asyncio-supplied SSL
implementation.