Repository: pypa/pip

Issue:
MovedModule errors on systems where they are n/a (like winreg on unix systems)
- If I install pip 1.5.2 (or any earlier version) my little script works:

```
diana$ pip install pip==1.5.2
diana$ python foo.py
works!
```
- If I install pip 1.5.3 or 1.5.4 my silly script stops working:

```
diana$ pip install pip==1.5.3
diana$ python foo.py
Traceback (most recent call last):
  File "foo.py", line 13, in <module>
    scan_module(pip)
  File "foo.py", line 11, in scan_module
    scan_module(item)
  File "foo.py", line 11, in scan_module
    scan_module(item)
  File "foo.py", line 11, in scan_module
    scan_module(item)
  File "foo.py", line 9, in scan_module
    item = getattr(module, name, None)
  File "/User/diana/env/foo/lib/python2.7/site-packages/pip/_vendor/six.py", line 86, in __get__
    result = self._resolve()
  File "/User/diana/env/foo/lib/python2.7/site-packages/pip/_vendor/six.py", line 105, in _resolve
    return _import_module(self.mod)
  File "/User/diana/env/foo/lib/python2.7/site-packages/pip/_vendor/six.py", line 76, in _import_module
    __import__(name)
ImportError: No module named _winreg
```
- My silly script (foo.py):

```
import pip
import types


def scan_module(module):
    modules.add(module)
    for name in dir(module):
        item = getattr(module, name, None)
        if isinstance(item, types.ModuleType) and item not in modules:
            scan_module(item)


modules = set()
scan_module(pip)
print 'works!'
```

A six.py regression?

PS. This code snippet was extracted from paver. My paver scripts import pip, and since paver scans all imported modules (paver/tasks.py), the paver commands stopped working when I upgraded to pip 1.5.3.

https://github.com/paver/paver



Discussion:
Yah, [seems to be](https://bitbucket.org/gutworth/six/issue/63/importing-six-breaks-pickling). Around [here](https://bitbucket.org/gutworth/six/src/320691c3ccadeb64bb8a368ef9d39cb2a9a44770/six.py?at=default#cl-93), apparently fixed for django's purposes, but obviously not everyone's.

edit: wanted to link to a six issue, not a sphinx issue >_>

I'm sure there's a good reason to not do the following... like not being able to differentiate between missing modules (winreg on unix systems), and modules that simply fail to import (for whatever reason). Thoughts @gutworth?

```
diff --git a/pip/_vendor/six.py b/pip/_vendor/six.py
index 7ec7f1b..34b7992 100644
--- a/pip/_vendor/six.py
+++ b/pip/_vendor/six.py
@@ -73,10 +73,12 @@ def _add_doc(func, doc):

 def _import_module(name):
     """Import module, returning the module after the last dot."""
-    __import__(name)
+    try:
+        __import__(name)
+    except ImportError:
+        return None
     return sys.modules[name]

-
 class _LazyDescr(object):

     def __init__(self, name):
@@ -84,6 +86,8 @@ class _LazyDescr(object):

     def __get__(self, obj, tp):
         result = self._resolve()
+        if result is None:
+            raise AttributeError
         setattr(obj, self.name, result) # Invokes __set__.
         # This is a bit ugly, but it avoids running this again.
         delattr(obj.__class__, self.name)
@@ -105,15 +109,9 @@ class MovedModule(_LazyDescr):
         return _import_module(self.mod)

     def __getattr__(self, attr):
-        # Hack around the Django autoreloader. The reloader tries to get
-        # __file__ or __name__ of every module in sys.modules. This doesn't work
-        # well if this MovedModule is for an module that is unavailable on this
-        # machine (like winreg on Unix systems). Thus, we pretend __file__ and
-        # __name__ don't exist if the module hasn't been loaded yet. See issues
-        # #51 and #53.
-        if attr in ("__file__", "__name__") and self.mod not in sys.modules:
-            raise AttributeError
         _module = self._resolve()
+        if _module is None:
+            raise AttributeError
         value = getattr(_module, attr)
         setattr(self, attr, value)
         return value
```

@dianaclarke That's not such a bad idea. Thanks! How does six 1.6.0 work for you?

@gutworth six 1.6.0 still doesn't work for me b/c it's the resolve call in `_LazyDescr.__get__` (not `MovedModule.__getattr__`) that raises an ImportError for things like winreg (in my case).

Ugg, I guess that's what I get for not scrolling to the right in your traceback. This is more of a borderline case because I think it's better for `six.moves.winreg` to raise an `ImportError` than an `AttributeError` if `winreg` is not in fact available. OTOH, it is an attribute access... Maybe 1.6.1 will do the trick?

@gutworth Yup! six 1.6.1 fixes it. Thanks, that was fun :)

```
diana$ python foo.py
works!
```

Is this an error that happened in normal use of pip, or only inside some larger use where you had to import it and scan modules?

My use case: We import `pip.req` to `parse_requirements` within one of our paver scripts (https://github.com/paver/paver). Paver scans all modules to get all tasks breaking things like: `paver help`.

For example, the following paver script breaks as of pip 1.5.3. 

```
from paver.easy import task
import pip


@task
def foo():
    print 'foo'
```


Resolving patch:
diff --git a/pip/_vendor/six.py b/pip/_vendor/six.py
--- a/pip/_vendor/six.py
+++ b/pip/_vendor/six.py
@@ -25,7 +25,7 @@
 import types
 
 __author__ = "Benjamin Peterson <benjamin@python.org>"
-__version__ = "1.5.2"
+__version__ = "1.6.1"
 
 
 # Useful for very coarse version differentiation.
@@ -83,7 +83,11 @@ def __init__(self, name):
         self.name = name
 
     def __get__(self, obj, tp):
-        result = self._resolve()
+        try:
+            result = self._resolve()
+        except ImportError:
+            # See the nice big comment in MovedModule.__getattr__.
+            raise AttributeError("%s could not be imported " % self.name)
         setattr(obj, self.name, result) # Invokes __set__.
         # This is a bit ugly, but it avoids running this again.
         delattr(obj.__class__, self.name)
@@ -105,15 +109,22 @@ def _resolve(self):
         return _import_module(self.mod)
 
     def __getattr__(self, attr):
-        # Hack around the Django autoreloader. The reloader tries to get
-        # __file__ or __name__ of every module in sys.modules. This doesn't work
-        # well if this MovedModule is for an module that is unavailable on this
-        # machine (like winreg on Unix systems). Thus, we pretend __file__ and
-        # __name__ don't exist if the module hasn't been loaded yet. See issues
-        # #51 and #53.
-        if attr in ("__file__", "__name__") and self.mod not in sys.modules:
-            raise AttributeError
-        _module = self._resolve()
+        # It turns out many Python frameworks like to traverse sys.modules and
+        # try to load various attributes. This causes problems if this is a
+        # platform-specific module on the wrong platform, like _winreg on
+        # Unixes. Therefore, we silently pretend unimportable modules do not
+        # have any attributes. See issues #51, #53, #56, and #63 for the full
+        # tales of woe.
+        #
+        # First, if possible, avoid loading the module just to look at __file__,
+        # __name__, or __path__.
+        if (attr in ("__file__", "__name__", "__path__") and
+            self.mod not in sys.modules):
+            raise AttributeError(attr)
+        try:
+            _module = self._resolve()
+        except ImportError:
+            raise AttributeError(attr)
         value = getattr(_module, attr)
         setattr(self, attr, value)
         return value
@@ -222,6 +233,7 @@ class _MovedItems(_LazyModule):
     MovedModule("urllib", __name__ + ".moves.urllib", __name__ + ".moves.urllib"),
     MovedModule("urllib_robotparser", "robotparser", "urllib.robotparser"),
     MovedModule("xmlrpc_client", "xmlrpclib", "xmlrpc.client"),
+    MovedModule("xmlrpc_server", "xmlrpclib", "xmlrpc.server"),
     MovedModule("winreg", "_winreg"),
 ]
 for attr in _moved_attributes:
@@ -241,6 +253,7 @@ class Module_six_moves_urllib_parse(_LazyModule):
 
 _urllib_parse_moved_attributes = [
     MovedAttribute("ParseResult", "urlparse", "urllib.parse"),
+    MovedAttribute("SplitResult", "urlparse", "urllib.parse"),
     MovedAttribute("parse_qs", "urlparse", "urllib.parse"),
     MovedAttribute("parse_qsl", "urlparse", "urllib.parse"),
     MovedAttribute("urldefrag", "urlparse", "urllib.parse"),
@@ -254,6 +267,7 @@ class Module_six_moves_urllib_parse(_LazyModule):
     MovedAttribute("unquote", "urllib", "urllib.parse"),
     MovedAttribute("unquote_plus", "urllib", "urllib.parse"),
     MovedAttribute("urlencode", "urllib", "urllib.parse"),
+    MovedAttribute("splitquery", "urllib", "urllib.parse"),
 ]
 for attr in _urllib_parse_moved_attributes:
     setattr(Module_six_moves_urllib_parse, attr.name, attr)


Test patch: