Repository: python/mypy

Issue:
typeshed not being found on windows user install
```powershell
PS D:\testing> git clone https://github.com/python/mypy.git
# no errors

PS D:\testing> cd .\mypy\

PS D:\testing\mypy> git submodule update --init typeshed
# worked, checked out '55dbb967ad006d5835590c53ab023774dc5a1341'

PS D:\testing\mypy> pip install --user --upgrade .
Processing D:\testing\mypy
Requirement already up-to-date: typed-ast<1.2.0,>=1.1.0 in c:\users\[redacted!]\appdata\roaming\python\python36\site-packages (from mypy===0.530-dev-2bd11f60f0a0332b4c9537983a6ac7da4416e9ca)
Installing collected packages: mypy
  Running setup.py install for mypy ... done
Successfully installed mypy-0.530-dev-2bd11f60f0a0332b4c9537983a6ac7da4416e9ca

PS D:\testing\mypy> mypy setup.py  # to test
Could not resolve typeshed subdirectories. If you are using MyPyfrom source, you need to run "git submodule --init update".Otherwise your MyPy install is broken.
```

The data directory (after searching for `typeshed`) appears to be `%AppData%\Python\lib\mypy`

full path: `C:\Users\[redacted!]\AppData\Roaming\Python\lib\mypy`

```powershell
PS D:\testing\mypy> ls $Env:AppData\Python\lib\mypy

typeshed
xml
```

using this directory leads to more expected results

```powershell
PS D:\testing\mypy> mypy --custom-typeshed-dir $Env:AppData\Python\lib\mypy\typeshed setup.py
setup.py:15: error: No library stub file for module 'setuptools'
setup.py:15: note: (Stub files are from https://github.com/python/typeshed)
setup.py:16: error: No library stub file for module 'setuptools.command.build_py'
setup.py:74: error: Need type annotation for variable
```


Discussion:
It would seem that mypy's search for typeshed needs to be augmented with a case for this type of configuration. Mypy itself is at `$HOME\appdata\roaming\python\python36\site-packages\mypy` and typeshed is at `$HOME\AppData\Python\lib\mypy\typeshed`. Could you take a debugger and step through the `default_data_dir()` function (or insert some print statements) to find out where it goes wrong? I'm guessing the reason might be that it's looking for a component of the path named `lib`, but in your case there isn't one.

How did your Python install get in that location? I don't think that's where the Windows installers downloaded from python.org put it (but I might be wrong).
I can debug this.

>How did your Python install get in that location? I don't think that's where the Windows installers downloaded from python.org put it (but I might be wrong).

For me the default 3.x install directory for the newer installers (3.5+) point to `%USERPROFILE%\AppData\Local\Programs\Python\PythonXY` instead of `C:\PythonXY`.
sorry for a slightly-too-late response, the fix doesn't seem to work for me :(

https://gist.github.com/Summertime/403ce90d067253669796326212becce0 probably-overkill install logs are here for the exact paths of everything (installing from the latest merge)

I can give a shot at trying to work out the issue in a couple days, however the problem might be obvious from the install logs already, I'm not too sure.
Hm okay. Based on the output of the log it seems that you are using a custom compiled Python. Is that true? Sorry that fix didn't work for you.
no custom compiling! I reinstalled python just to make sure, clicked the main 3.x.x download link on python.org, did [the following](https://imgur.com/a/jzyya), then ran `pip install . --user` in the same folder as before, same result!
Could you tell me what `where mypy` gives you now that you re-installed Python? Thanks!
I wonder if (at least on Windows) the search code should apply
os.path.normpath() before comparing?

On Sun, Sep 24, 2017 at 3:13 PM, Ethan Smith <notifications@github.com>
wrote:

> Could you tell me what where mypy gives you now that you re-installed
> Python? Thanks!
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/python/mypy/issues/3988#issuecomment-331743843>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ACwrMkCtUmPLkNqTdu6tANJwr6ik7HrBks5sltP0gaJpZM4PheqA>
> .
>



-- 
--Guido van Rossum (python.org/~guido)

Could you point out the comparison that you are thinking of? The part of the code in question is independent of the OS path separator to my knowledge. I'm thinking of here in particular https://github.com/python/mypy/blob/6c409b4ed370f425063b613ac0be31cc0151f4c5/mypy/build.py#L231
I wasn't thinking of `os.sep` but of different cases for `lib` or `python` (on the next line). In particular the OP has `Python` (`%AppData%\Python\lib\mypy` and `C:\Users\[redacted!]\AppData\Roaming\Python\lib\mypy`).
Confusingly, `%APPDATA%` expands to `%USERPROFILE%\AppData\Roaming` or `C:\Users\<username>\AppData\Roaming`, so essentially the two are the same.
I'm working on the hypothesis that `...\appdata\roaming\python\python36\site-packages` (as quoted by OP) might be using capitalization everywhere, i.e. `...\AppData\Roaming\Python\Python36\site-packages`. The only time this path is quoted by the OP it's clearly already lowercased by pip, so we have no hard evidence against this hypothesis. And if that's true, the algorithm (which seems to be [searching for](https://github.com/python/mypy/blob/6c409b4ed370f425063b613ac0be31cc0151f4c5/mypy/build.py#L232) the `python36` segment of the path) would fail since it's doing a case-sensitive comparison.

I have no idea what installer was used, so I don't want to exclude this hypothesis yet -- it seems to be using capitalization in earlier segments of the path (at least `...\AppData\Roaming\Python` are quoted by the OP). Also, I found this about `AppData\Roaming`: https://askleo.com/whats-the-appdata-roaming-folder/
> path of mypy after reinstall

    PS D:\[username]\testing\mypy> where.exe mypy
    C:\Users\[username]\AppData\Roaming\Python\Python36\Scripts\mypy.exe

I did make sure to wipe all site-package/script directories before reinstalling too, so not an artifact of the previous install

> paths and capitalization

As reported by powershell/windows (and as a result, probably the "official" capitalization that would show up:

    PS ...\AppData\Roaming\Python\Python36\Scripts> Resolve-Path .\mypy.exe | %{$_.Path}
    C:\Users\[Username]\AppData\Roaming\Python\Python36\Scripts\mypy.exe

    PS ...\AppData\Roaming\Python\Python36\Scripts> Resolve-Path .\mypy.exe | %{$_.Path}
    C:\Users\[Username]\AppData\Roaming\Python\Python36\Scripts\mypy.exe

    PS ...\AppData\Roaming\Python\Python36\Scripts> Resolve-Path ..\site-packages\mypy | %{$_.Path}
    C:\Users\[Username]\AppData\Roaming\Python\Python36\site-packages\mypy

    PS ...\AppData\Roaming\Python\Python36\Scripts> Resolve-Path ..\..\lib\mypy\typeshed | %{$_.Path}
    C:\Users\[Username]\AppData\Roaming\Python\lib\mypy\typeshed

> custom Installer

I'm working on seeing if this reproduces in a fresh windows 10 VM using only  
https://www.python.org/ftp/python/3.6.2/python-3.6.2.exe and  
https://github.com/git-for-windows/git/releases/download/v2.14.1.windows.1/Git-2.14.1-64-bit.exe  
Aha, I see an issue. In the screenshots of your installer, you are using Python 3.6 32 bits. It is being installed in `%USERPROFILE%\AppData\Local\...`. It seems that the `mypy.exe` is left over (it is installed in `%USERPROFILE%\AppData\Roaming`). 

However, I agree that the check for prefixes with `python` should probably be case insensitive. So I will make a PR about that.
By the way #3991 broke installing mypy on Linux Mint. So if #3991 doesn't really solve this problem then I propose to remove the ``or os.path.basename(lib).startswith('python')`` line.
Hm, that is concerning. Could you describe how it is broken?
...or make this check dependent on ``sys.platform == 'linux'``.
I see this when I install from git master:
```
tst22.py: error: Name '__builtins__' is not defined
tst22.py:1: error: No library stub file for standard library module 'typing'
tst22.py:1: note: (Stub files are from https://github.com/python/typeshed)
```
FWIW, removing the line I mentioned fixes the problem. But I have no idea why.
it goes into `appdata/roaming` when done with `--user`, if I did it without `--user` it would end up in the `appdata/local` spot

I did say I wiped the folder!

And it reproduces in a VM too!

```
# after installing python like I did in the screenshots before:
# Also using absolute paths since too lazy to update my path for pip's user install location
PS [blah]\mypy> C:\Users\[differentusername!]\AppData\Roaming\Python\Python36\Scripts\mypy.exe setup.py
Could not resolve typeshed subdirectories. If you are using mypy
from source, you need to run "git submodule --init update".
Otherwise your mypy install is broken.
Python executable is located at c:\users\[differentusername!]\appdata\local\programs\python\python36-32\python.exe.
Mypy located at C:\Users\[differentusername!]\AppData\Roaming\Python\Python36\site-packages
```

However I will note, running the module directly works:

```
PS [blah]\mypy> python -m mypy .\setup.py
setup.py:15: error: No library stub file for module 'setuptools'
setup.py:15: note: (Stub files are from https://github.com/python/typeshed)
setup.py:16: error: No library stub file for module 'setuptools.command.build_py'
setup.py:74: error: Need type annotation for variable
```

(but for some reason my main install doesn't mirror that, dunno!) (I guess I'll consider my main install damaged for now since it repros differently)
It looks like installation on Mac is also broken now. I see:
```
tst22.py: error: Name '__builtins__' is not defined
tst22.py:1: error: No library stub file for standard library module 'typing'
tst22.py:1: note: (Stub files are from https://github.com/python/typeshed)
```
Should we just roll back whatever broke things recently?

On Sep 25, 2017 6:58 AM, "Ivan Levkivskyi" <notifications@github.com> wrote:

It looks like installation on Mac is also broken now. I see:

tst22.py: error: Name '__builtins__' is not defined
tst22.py:1: error: No library stub file for standard library module 'typing'
tst22.py:1: note: (Stub files are from https://github.com/python/typeshed)

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
<https://github.com/python/mypy/issues/3988#issuecomment-331889104>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ACwrMpNq95PXa1706JJE3P_zHYh7zUUKks5sl7E7gaJpZM4PheqA>
.


Resolving patch:
diff --git a/mypy/build.py b/mypy/build.py
--- a/mypy/build.py
+++ b/mypy/build.py
@@ -17,6 +17,7 @@
 import json
 import os.path
 import re
+import site
 import sys
 import time
 from os.path import dirname, basename
@@ -213,6 +214,12 @@ def default_data_dir(bin_dir: Optional[str]) -> str:
       bin_dir: directory containing the mypy script
     """
     if not bin_dir:
+        if os.name == 'nt':
+            prefixes = [os.path.join(sys.prefix, 'Lib'), os.path.join(site.getuserbase(), 'lib')]
+            for parent in prefixes:
+                    data_dir = os.path.join(parent, 'mypy')
+                    if os.path.exists(data_dir):
+                        return data_dir
         mypy_package = os.path.dirname(__file__)
         parent = os.path.dirname(mypy_package)
         if (os.path.basename(parent) == 'site-packages' or
@@ -223,13 +230,10 @@ def default_data_dir(bin_dir: Optional[str]) -> str:
             # or .../blah/lib64/python3.N/dist-packages/mypy/build.py (Gentoo)
             # or .../blah/lib/site-packages/mypy/build.py (Windows)
             # blah may be a virtualenv or /usr/local.  We want .../blah/lib/mypy.
-            # On Windows, if the install is .../python/PythonXY/site-packages, we want
-            # .../python/lib/mypy
             lib = parent
             for i in range(2):
                 lib = os.path.dirname(lib)
-                if os.path.basename(lib) in ('lib', 'lib32', 'lib64') \
-                        or os.path.basename(lib).startswith('python'):
+                if os.path.basename(lib) in ('lib', 'lib32', 'lib64'):
                     return os.path.join(os.path.dirname(lib), 'lib/mypy')
         subdir = os.path.join(parent, 'lib', 'mypy')
         if os.path.isdir(subdir):


Test patch: