AttributeError: '_tqdm_cls' object has no attribute '_lock'
Opened by codingl2k1 (closed)

### Describe the bug

```python
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/site-packages/datasets/load.py", line 1034, in get_module
    data_files = DataFilesDict.from_patterns(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/site-packages/datasets/data_files.py", line 671, in from_patterns
    DataFilesList.from_patterns(
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/site-packages/datasets/data_files.py", line 586, in from_patterns
    origin_metadata = _get_origin_metadata(data_files, download_config=download_config)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/site-packages/datasets/data_files.py", line 502, in _get_origin_metadata
    return thread_map(
           ^^^^^^^^^^^
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 70, in thread_map
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 48, in _executor_map
    with ensure_lock(tqdm_class, lock_name=lock_name) as lk:
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/site-packages/tqdm/contrib/concurrent.py", line 25, in ensure_lock
    del tqdm_class._lock
        ^^^^^^^^^^^^^^^^
AttributeError: '_tqdm_cls' object has no attribute '_lock'
```

### Steps to reproduce the bug

Happens ocasionally.

### Expected behavior

I added a print in tqdm `ensure_lock()`, got a `ensure_lock <datasets.utils.logging._tqdm_cls object at 0x16dddead0> ` print.

According to the code in https://github.com/tqdm/tqdm/blob/master/tqdm/contrib/concurrent.py#L24

```python
@contextmanager
def ensure_lock(tqdm_class, lock_name=""):
    """get (create if necessary) and then restore `tqdm_class`'s lock"""
    print("ensure_lock", tqdm_class, lock_name)
    old_lock = getattr(tqdm_class, '_lock', None)  # don't create a new lock
    lock = old_lock or tqdm_class.get_lock()  # maybe create a new lock
    lock = getattr(lock, lock_name, lock)  # maybe subtype
    tqdm_class.set_lock(lock)
    yield lock
    if old_lock is None:
        del tqdm_class._lock  # <-- It tries to del the `_lock` attribute from tqdm_class.
    else:
        tqdm_class.set_lock(old_lock)
```

But, huggingface datasets `datasets.utils.logging._tqdm_cls` does not have the field `_lock`: https://github.com/huggingface/datasets/blob/main/src/datasets/utils/logging.py#L205

```python
class _tqdm_cls:
    def __call__(self, *args, disable=False, **kwargs):
        if _tqdm_active and not disable:
            return tqdm_lib.tqdm(*args, **kwargs)
        else:
            return EmptyTqdm(*args, **kwargs)

    def set_lock(self, *args, **kwargs):
        self._lock = None
        if _tqdm_active:
            return tqdm_lib.tqdm.set_lock(*args, **kwargs)

    def get_lock(self):
        if _tqdm_active:
            return tqdm_lib.tqdm.get_lock()
```



### Environment info

Python 3.11.4
tqdm '4.65.0'
datasets master

Comments:
Hi ! I opened https://github.com/huggingface/datasets/pull/6067 to add the missing `_lock`

We'll do a patch release soon, but feel free to install `datasets` from source in the meantime

I have tested the latest main, it does not work.

I add more logs to reproduce this issue, it looks like a multi threading bug:

```python
@contextmanager
def ensure_lock(tqdm_class, lock_name=""):
    """get (create if necessary) and then restore `tqdm_class`'s lock"""
    import os
    import threading
    print(os.getpid(), threading.get_ident(), "ensure_lock", tqdm_class, lock_name)
    old_lock = getattr(tqdm_class, '_lock', None)  # don't create a new lock
    lock = old_lock or tqdm_class.get_lock()  # maybe create a new lock
    lock = getattr(lock, lock_name, lock)  # maybe subtype
    tqdm_class.set_lock(lock)
    print(os.getpid(), threading.get_ident(), "set_lock")
    yield lock
    if old_lock is None:
        print(os.getpid(), threading.get_ident(), "del tqdm_class")
        del tqdm_class._lock
    else:
        tqdm_class.set_lock(old_lock)
```
output
```
64943 8424758784 ensure_lock <datasets.utils.logging._tqdm_cls object at 0x2aa7fb250> 
64943 8424758784 set_lock
64943 8424758784 del tqdm_class
64943 8424758784 ensure_lock <datasets.utils.logging._tqdm_cls object at 0x2aa7fb250> 
64943 8424758784 set_lock
64943 8424758784 del tqdm_class
64943 11638370304 ensure_lock <datasets.utils.logging._tqdm_cls object at 0x2aa7fb250> 
64943 11638370304 set_lock
64943 11568967680 ensure_lock <datasets.utils.logging._tqdm_cls object at 0x2aa7fb250> 
64943 11568967680 set_lock
64943 11638370304 del tqdm_class
64943 11638370304 ensure_lock <datasets.utils.logging._tqdm_cls object at 0x2aa7fb250> 
64943 11638370304 set_lock
64943 11638370304 del tqdm_class
64943 11568967680 del tqdm_class
```

Thread `11638370304` del the _lock from tqdm_class first, then thread `11568967680` del _lock failed.

Maybe it is a bug of tqdm? I think simply use `try ... except AttributeError ...` wraps `del tqdm_class._lock` should work.

Yes it looks like a bug on their end indeed, do you want to open a PR on tqdm ?

Let me see if I can find a workaround in the meantime

I opened https://github.com/huggingface/datasets/pull/6068 if you want to try it out

> I opened #6068 if you want to try it out

This fix works! Thanks.

Awesome ! closing this then :)
We'll do a patch release today or tomorrow