Repository: huggingface/diffusers

Issue:
logging.remove_handler() has a faulty assertion, doesn't allow registered handlers to be removed
### Describe the bug

in `utils/logging.py` there the function remove_handler seems to have a faulty assertion in it.

```py
def add_handler(handler: logging.Handler) -> None:
    """adds a handler to the HuggingFace Diffusers' root logger."""

    _configure_library_root_logger()

    assert handler is not None
    _get_library_root_logger().addHandler(handler)


def remove_handler(handler: logging.Handler) -> None:
    """removes given handler from the HuggingFace Diffusers' root logger."""

    _configure_library_root_logger()

    assert handler is not None and handler not in _get_library_root_logger().handlers # <<< THIS ONE
    _get_library_root_logger().removeHandler(handler)
```

That line seems to have a `not` that shouldn't be there. (the `not in` to be precise)

Normally I'd just do a PR to remove it, but as I'm not that familiar with the codebase, nor Python in general, and don't yet have a solid grasp of what these things actually do (I'm just playing around trying to familiarize myself), I decided to make an issue instead so people who actually know the codebase can do it.

### Reproduction

```py
from diffusers import logging
from logging import Handler

class TestHandler(Handler):
    def __init__(self):
        super().__init__()

    def emit(self):
        pass

handler = TestHandler()
logging.add_handler(handler)
logging.remove_handler(handler)
```

### Logs

```shell
Traceback (most recent call last):
  File ".\test.py", line 14, in <module>
    logging.remove_handler(handler)
  File "C:\Users\XXX\XXX\venv\lib\site-packages\diffusers\utils\logging.py", line 221, in remove_handler
    assert handler is not None and handler not in _get_library_root_logger().handlers
AssertionError
```


### System Info

- `diffusers` version: 0.3.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.9
- PyTorch version (GPU?): 1.12.1+cpu (False)
- Huggingface_hub version: 0.9.1
- Transformers version: 4.21.3
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: Don't understand the question, but seems irrelevant.


Discussion:
Hey @SimplyLinn,

I think you caught a bug here indeed! Just confirming with @LysandreJik (since this code is more or less copy-pasted from `transformers`). The `not in` should be changed to `in` just like @SimplyLinn says no? 
This was added by @philschmid in https://github.com/huggingface/transformers/pull/10633; I think the `not` should indeed be removed, but would like confirmation from Philipp before doing so
Cool thanks for the feedback! @SimplyLinn would you like to open a PR to fix it? :-) 
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/diffusers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Resolving patch:
diff --git a/src/diffusers/utils/logging.py b/src/diffusers/utils/logging.py
--- a/src/diffusers/utils/logging.py
+++ b/src/diffusers/utils/logging.py
@@ -213,7 +213,7 @@ def remove_handler(handler: logging.Handler) -> None:
 
     _configure_library_root_logger()
 
-    assert handler is not None and handler not in _get_library_root_logger().handlers
+    assert handler is not None and handler in _get_library_root_logger().handlers
     _get_library_root_logger().removeHandler(handler)
 
 


Test patch: