Be more selective about how file descriptors get removed from Kombu's hub.

Given that file descriptor changes appear to get triggered on the Kombu side, these changes
may not make a material impact.  However, to make things more consistent with the
changes introduced in https://github.com/celery/kombu/pull/353, the changes have been
updated here.

This change would also help allow refactoring for remove_reader()/remove_writer() to
be smarter about how file descriptors get managed in the future (i.e. using a counter
instead of removes() to avoid possible race conditions with file descriptors being reused)