Repository: piskvorky/gensim

Issue:
Unnessary workaround in LDA tutorial
The documentation in the LDA tutorial mentions an already fixed issue (https://github.com/RaRe-Technologies/smart_open/issues/331).
What is now the proper way to read and untar the file on the fly?

https://github.com/RaRe-Technologies/gensim/blob/338ef330dea97c90c3180a9b570be9d0c9cef302/docs/src/auto_examples/tutorials/run_lda.py#L68-L91


Discussion:
You could try something like:

```python
with smart_open.open(fname, "r") as fin:
    with tarfile.open(fin) as fout:
        ...
```

But YMMV.
I tried the following:

``` python
    with smart_open.open('https://cs.nyu.edu/~roweis/data/nips12raw_str602.tgz', "r") as fin:
        with tarfile.open(fin, mode='r:gz') as tar:
            # Ignore directory entries, as well as files like README, etc.
            files = [
                m for m in tar.getmembers()
                if m.isfile() and re.search(r'nipstxt/nips\d+/\d+\.txt', m.name)
            ]
            for member in sorted(files, key=lambda x: x.name):
                member_bytes = tar.extractfile(member).read()
                yield member_bytes.decode('utf-8', errors='replace')
```
and received following error:
`TypeError: expected str, bytes or os.PathLike object, not StreamReader`
It seems like tarfile does not support this way of streaming or did I misunderstand you?
My example was incomplete. If you look at the reference for the [tarfile submodule], you'll find that the open function accepts a fileobj parameter. So, the working code would be:

```python
$ python
Python 3.8.7 (default, Feb  3 2021, 07:09:08)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import smart_open
>>> f = smart_open.open("https://cs.nyu.edu/~roweis/data/nips12raw_str602.tgz", "rb")
>>> import tarfile
>>> tf = tarfile.open(fileobj=f)
>>> tf.getmembers()[:5]
[<TarInfo 'nipstxt' at 0x111da1880>, <TarInfo 'nipstxt/nips00' at 0x111da17c0>, <TarInfo 'nipstxt/nips00/0387.txt' at 0x111da1700>, <TarInfo 'nipstxt/nips00/0001.txt' at 0x111da1640>, <TarInfo 'nipstxt/nips00/0009.txt' at 0x111da1a00>]
```

The other important point is to open the outer file object for reading in binary mode as opposed to text.

[tarfile submodule]: https://docs.python.org/3/library/tarfile.html
Thanks, I didn't know about the fileobj parameter.
Your suggested solution works but is slower than the proposed workaround (downloading the .tar archive):
``` python
def extract_documents(url='https://cs.nyu.edu/~roweis/data/nips12raw_str602.tgz'):
    with smart_open.open(url, "rb") as file:
        with tarfile.open(fileobj=file) as tar:
            # Ignore directory entries, as well as files like README, etc.
            files = [
                m for m in tar.getmembers()
                if m.isfile() and re.search(r'nipstxt/nips\d+/\d+\.txt', m.name)
            ]
            for member in sorted(files, key=lambda x: x.name):
                member_bytes = tar.extractfile(member).read()
                yield member_bytes.decode('utf-8', errors='replace')
```
On my local machine the workaround needs only:
CPU times: user 4.27 s, sys: 197 ms, total: 4.47 s
Wall time: 7.22 s

The filestream method is ~10x slower:
CPU times: user 8.81 s, sys: 940 ms, total: 9.75 s
Wall time: 1min 9s

I assume the sorting leads to this behavior. Is there any rationale behind sorting the extracted files?
LDA doesn't care about the order of documents :joy: 
Perhaps... can you try disabling the sorting and re-measure the time?

It's likely the sorting is there to make the processing order predictable for anybody eyeballing the intermediate result (e.g. log statements, etc). If it does not affect the final result, it isn't strictly necessary.

Resolving patch:
diff --git a/docs/src/auto_examples/tutorials/run_lda.py b/docs/src/auto_examples/tutorials/run_lda.py
--- a/docs/src/auto_examples/tutorials/run_lda.py
+++ b/docs/src/auto_examples/tutorials/run_lda.py
@@ -66,29 +66,12 @@
 import smart_open
 
 def extract_documents(url='https://cs.nyu.edu/~roweis/data/nips12raw_str602.tgz'):
-    fname = url.split('/')[-1]
-
-    # Download the file to local storage first.
-    # We can't read it on the fly because of
-    # https://github.com/RaRe-Technologies/smart_open/issues/331
-    if not os.path.isfile(fname):
-        with smart_open.open(url, "rb") as fin:
-            with smart_open.open(fname, 'wb') as fout:
-                while True:
-                    buf = fin.read(io.DEFAULT_BUFFER_SIZE)
-                    if not buf:
-                        break
-                    fout.write(buf)
-
-    with tarfile.open(fname, mode='r:gz') as tar:
-        # Ignore directory entries, as well as files like README, etc.
-        files = [
-            m for m in tar.getmembers()
-            if m.isfile() and re.search(r'nipstxt/nips\d+/\d+\.txt', m.name)
-        ]
-        for member in sorted(files, key=lambda x: x.name):
-            member_bytes = tar.extractfile(member).read()
-            yield member_bytes.decode('utf-8', errors='replace')
+    with smart_open.open(url, "rb") as file:
+        with tarfile.open(fileobj=file) as tar:
+            for member in tar.getmembers():
+                if member.isfile() and re.search(r'nipstxt/nips\d+/\d+\.txt', member.name):
+                    member_bytes = tar.extractfile(member).read()
+                    yield member_bytes.decode('utf-8', errors='replace')
 
 docs = list(extract_documents())
 
diff --git a/docs/src/gallery/tutorials/run_lda.py b/docs/src/gallery/tutorials/run_lda.py
--- a/docs/src/gallery/tutorials/run_lda.py
+++ b/docs/src/gallery/tutorials/run_lda.py
@@ -66,29 +66,12 @@
 import smart_open
 
 def extract_documents(url='https://cs.nyu.edu/~roweis/data/nips12raw_str602.tgz'):
-    fname = url.split('/')[-1]
-
-    # Download the file to local storage first.
-    # We can't read it on the fly because of
-    # https://github.com/RaRe-Technologies/smart_open/issues/331
-    if not os.path.isfile(fname):
-        with smart_open.open(url, "rb") as fin:
-            with smart_open.open(fname, 'wb') as fout:
-                while True:
-                    buf = fin.read(io.DEFAULT_BUFFER_SIZE)
-                    if not buf:
-                        break
-                    fout.write(buf)
-
-    with tarfile.open(fname, mode='r:gz') as tar:
-        # Ignore directory entries, as well as files like README, etc.
-        files = [
-            m for m in tar.getmembers()
-            if m.isfile() and re.search(r'nipstxt/nips\d+/\d+\.txt', m.name)
-        ]
-        for member in sorted(files, key=lambda x: x.name):
-            member_bytes = tar.extractfile(member).read()
-            yield member_bytes.decode('utf-8', errors='replace')
+    with smart_open.open(url, "rb") as file:
+        with tarfile.open(fileobj=file) as tar:
+            for member in tar.getmembers():
+                if member.isfile() and re.search(r'nipstxt/nips\d+/\d+\.txt', member.name):
+                    member_bytes = tar.extractfile(member).read()
+                    yield member_bytes.decode('utf-8', errors='replace')
 
 docs = list(extract_documents())
 


Test patch: