Repository: pypa/pip

Issue:
`pip list --outdated` fails on platforms without sem_open
**Environment**

* pip version: 20.1
* Python version: 3.8.2
* OS: Android Termux

<!-- Feel free to add more information about your environment here -->

**Description**
running `pip list -o` leads to an `ImportError` in `multiprocessing.dummy`:
"ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.”

not bothering with traceback, since it's easily traced to the use of `multiprocessing.dummy.Pool` in commands/list.py line 216.
allow failed import of Pool from multiprocessing
quick fix for #8161


Discussion:
The issue in the above log is https://bugs.python.org/issue3770
This import was introduced as a part of #7964. We probably should disable parallellisation when this happens. cc @CrafterKolyan 
Can we try and fix it the way Cpython detects it ?

https://github.com/python/cpython/blob/d9a43e20facdf4ad10186f820601c6580e1baa80/Lib/multiprocessing/synchronize.py#L24-L33
See the last comment on that bug report, it's been closed for over 10 years. If this *is* a Python issue, it would need to be raised as a new bug.

Following the Python docs, `multiprocessing.dummy` is a wrapper around `threading` which is always available in Python 3.7+ So it looks like the OP's version of Python is incomplete?

However, this does bring up the point that in Python older than 3.7, threading is an optional module, so pip should probably fall back to the old code when threading isn't available. In which case, we don't care about *why* multiprocessing.dummy doesn't work, and the OP's issue gets solved as well.

> Can we try and fix it the way Cpython detects it ?

I'd just say that if the multiprocessing.dummy import fails, we should fall back to the single-threaded code.
TBH, at this point, I feel like we should revert the original PR until we figure out the finer details of what we want parallelized code in pip to look like, what strategy we use to support platforms that don't have threading etc.

_Originally posted by @pradyunsg in https://github.com/pypa/pip/pull/8162#issuecomment-621079139_
I think something like this may be a fix:
```
    def fetch_latest_info_multithreaded(packages, latest_info_func):
        from multiprocessing.dummy import Pool

        # This is done for 2x speed up of requests to pypi.org
        # so that "real time" of this function
        # is almost equal to "user time"
        pool = Pool(DEFAULT_POOLSIZE)

        for dist in pool.imap_unordered(latest_info_func, packages):
            if dist is not None:
                yield dist

        pool.close()
        pool.join()

    def fetch_latest_info_singlethreaded(packages, latest_info_func):
        for dist in packages:
            dist = latest_info_func(dist)
            if dist is not None:
                yield dist

    def iter_packages_latest_infos(self, packages, options):
        with self._build_session(options) as session:
            finder = self._build_package_finder(options, session)

            def latest_info(dist):
                typ = 'unknown'
                all_candidates = finder.find_all_candidates(dist.key)
                if not options.pre:
                    # Remove prereleases
                    all_candidates = [candidate for candidate in all_candidates
                                      if not candidate.version.is_prerelease]

                evaluator = finder.make_candidate_evaluator(
                    project_name=dist.project_name,
                )
                best_candidate = evaluator.sort_best_candidate(all_candidates)
                if best_candidate is None:
                    return None

                remote_version = best_candidate.version
                if best_candidate.link.is_wheel:
                    typ = 'wheel'
                else:
                    typ = 'sdist'
                # This is dirty but makes the rest of the code much cleaner
                dist.latest_version = remote_version
                dist.latest_filetype = typ
                return dist

            try:
                fetch_latest_info_multithreaded(packages, latest_info)
            # See issue #8161
            except ImportError:
                fetch_latest_info_singlethreaded(packages, latest_info)
```

and also we should delete
```
from multiprocessing.dummy import Pool
```
from the beginning of the file
ok cool yeah, use that instead, it's much more pythonic
> I think something like this may be a fix:

`# See issue #8161` should instead include the actual issue link: `https://github.com/pypa/pip/issues/8161` in place of `#8161`

Hi there, a bit late for the party but I think we can abstract it a bit further, i.e. we can allow
```python
try:
    # Maybe adding a filter in this case
    return map_threaded(latest_info_func, packages)
except ImportError:
    return map(latest_info_func, packages)
```

I hope that the parallelization in `pip list` is the beginning of parallel code in `pip`, and the above might make it easier to implement it in other places.
While we're all discussing colors for the bikeshed (i.e. the various forms to write this) here's my take on how to paint the bikeshed :wink:

```diff
diff --git a/src/pip/_internal/commands/list.py b/src/pip/_internal/commands/list.py
index cf3be7eb..a0558079 100644
--- a/src/pip/_internal/commands/list.py
+++ b/src/pip/_internal/commands/list.py
@@ -5,7 +5,6 @@ from __future__ import absolute_import
 
 import json
 import logging
-from multiprocessing.dummy import Pool
 
 from pip._vendor import six
 from pip._vendor.requests.adapters import DEFAULT_POOLSIZE
@@ -24,9 +23,30 @@ from pip._internal.utils.misc import (
 )
 from pip._internal.utils.packaging import get_installer
 
+try:
+    from multiprocessing.dummy import Pool
+except ImportError:
+    Pool = None  # type: ignore
+
 logger = logging.getLogger(__name__)
 
 
+def map_threaded(function, sequence):
+    assert Pool is not None
+
+    # DEFAULT_POOLSIZE is the pool size used by requests.
+    with Pool(DEFAULT_POOLSIZE):
+        for retval in pool.imap_unordered(function, sequence):
+            yield retval
+
+
+def map_threaded_if_possible(function, sequence):
+    if Pool is None:
+        return map(function, sequence)
+
+    return map_threaded(function, sequence)
+
+
 class ListCommand(IndexGroupCommand):
     """
     List installed packages, including editables.
@@ -210,18 +230,10 @@ class ListCommand(IndexGroupCommand):
                 dist.latest_filetype = typ
                 return dist
 
-            # This is done for 2x speed up of requests to pypi.org
-            # so that "real time" of this function
-            # is almost equal to "user time"
-            pool = Pool(DEFAULT_POOLSIZE)
-
-            for dist in pool.imap_unordered(latest_info, packages):
+            for dist in map_threaded_if_possible(latest_info, packages):
                 if dist is not None:
                     yield dist
 
-            pool.close()
-            pool.join()
-
     def output_package_listing(self, packages, options):
         packages = sorted(
             packages,

```

Don't forget the `pool.close(); pool.join()` in `map_threaded`.

Also, can we confirm precisely where the `ImportError` is raised? #8161 said

> not bothering with traceback, since it's easily traced to the use of multiprocessing.dummy.Pool in commands/list.py line 216.

But we need the traceback to see what precise statement fails.

I'm concerned that as we don't have a test platform without multithreading, if we're not careful, we'll check the wrong place and won't catch our mistake.
https://github.com/termux/termux-packages/issues/5222#issuecomment-621056880 contains a traceback for this issue.
Sharing the same concern with @pfmoore, since after a glance at CPython, it seems that [the error is raised during initialization of multiprocessing objects](https://github.com/python/cpython/search?q=import+synchronize&unscoped_q=import+synchronize) (synchronize is the module raising the error at import stage).  If so, the ingredients to mix the paint can be prepared as follows
```python3
def map_threaded(function, iterable):  # I mean map_threaded_if_possible
    try:
        pool = Pool(DEFAULT_POOLSIZE)  # ugh no context manager for Python 2's pool
    except ImportError:
        return map(function, iterable)
    else:
        yield from pool.imap_unordered(function, iterable)
        pool.close()
        pool.join()
```

Mocking for testing of this in particular is not too difficult IMHO; however testing for thread-safety in general when we expand multithreading of networking code can be not as easy.

Edit: I'm typing way too slow again, thanks @deveshks for the pointer.
... and `yield from` isn't Python 2 compatible 🙁 

And for that matter, yielding from a try...except is very dodgy. Would `map_threaded` be a generator or not with this code?

I don't mean to pick on your particular comment, @McSinyx, but can I ask that we try to refrain from posting speculative bits of code here? I'm guilty of doing so myself (and for that matter, of a rather too quick approval of the original PR, although I think that original code would have worked fine, so maybe it wasn't so bad to have approved it!) But if we're going to debate the most correct and robust solution, it's getting *very* hard to keep track of what's good code and what seemed reasonable but turned out to not work.

My biggest concern here is that we don't have a platform to test proposed solutions on (other than getting @landfillbaby to check for us) so we need to be fairly conservative in how we address the issue.
> can I ask that we try to refrain from posting speculative bits of code here?

Guilty as charged and in agreement.

TBH, at this point, I feel like we should revert the original PR until we figure out the finer details of what we want parallelized code in pip to look like, what strategy we use to support platforms that don't have threading etc.

> My biggest concern here is that we don't have a platform to test proposed solutions on

If anyone has an android 7 or newer smartphone testing should be as easy as installing [termux](https://termux.com/) and then running 
```sh
apt update && apt install python
pip install --upgrade pip
```
And then apply a patch and test `pip list -o`. Messing around with patches might be annoying on a small phone screen though, setting up sshd and ssh'ing into termux would be more convenient.

Me, @landfillbaby and other termux-involved people can help testing in any case.
Sorry @pfmoore (and thanks for pointing out `yield from` in py2), my code sample was totally unnecessary and distracting (I think the only point there was to use the `else` clause), given we got the traceback now.  Agreeing with @pradyunsg, probably we would want to revert the multiprocessing code until having better analysis on supported platform.  Not because for termux in particular, but I'm wondering if there's any *other* platform failing short of it for *other* reasons, since threading and locking mechanisms are provided by lower level than what we can control.

That being said, to fix the linked issue, this PR should be sufficient, given[this comment in CPython codebase](https://github.com/python/cpython/blob/f22cc69b012f52882d434a5c44a004bc3aa5c33c/Lib/multiprocessing/synchronize.py#L24-L33).  bpo-3770 might never be fixed, but I think the comment can be used as a canonical reference to the raise of ImportError.  The module `synchronize`, AFAIK, is late-imported during initialization of objects as shown above.

Resolving patch:
diff --git a/src/pip/_internal/commands/list.py b/src/pip/_internal/commands/list.py
--- a/src/pip/_internal/commands/list.py
+++ b/src/pip/_internal/commands/list.py
@@ -5,10 +5,8 @@
 
 import json
 import logging
-from multiprocessing.dummy import Pool
 
 from pip._vendor import six
-from pip._vendor.requests.adapters import DEFAULT_POOLSIZE
 
 from pip._internal.cli import cmdoptions
 from pip._internal.cli.req_command import IndexGroupCommand
@@ -210,18 +208,10 @@ def latest_info(dist):
                 dist.latest_filetype = typ
                 return dist
 
-            # This is done for 2x speed up of requests to pypi.org
-            # so that "real time" of this function
-            # is almost equal to "user time"
-            pool = Pool(DEFAULT_POOLSIZE)
-
-            for dist in pool.imap_unordered(latest_info, packages):
+            for dist in map(latest_info, packages):
                 if dist is not None:
                     yield dist
 
-            pool.close()
-            pool.join()
-
     def output_package_listing(self, packages, options):
         packages = sorted(
             packages,


Test patch: