Repository: ycm-core/ycmd

Issue:
make cmake use env python
CustomPythonCmakeArgs forces CMake to pickup the environment python libraries and include but the interpreter is still picked using PythonInterp which picks up the system default instead of the environment python. This cause issues with vitualenv.

<!-- Reviewable:start -->

[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/valloric/ycmd/196)

<!-- Reviewable:end -->

Ycmd build system assumes that the current python on OSX is the same to compile ycmd for.
I tried to install ycmd on macosx and it took me a while to actually figure out that:
1) you need python3 (both for Sublime Text3 and for the example)

2) the build system tries to build the library from the output of python-config --prefix. So even if I was running `python3 build.py` it would still try to use the default python configuration (that is 2.7 on my machine).

it would be useful if the build.py uses the current python executable to determine the prefix. 



Discussion:
Hm, this has the potential to break everything. The whole multiple-versions-of-python-on-one-machine thing is the single most frequent reason for failed YCM installations.

So we need to be _really_ sure this is doing the right thing. I'd appreciated it if various people could try out this change; the more OS's/distros we cover, the better.

@vheon @micbou @puremourning 

The `CustomPythonCmakeArgs` function is only called on OS X so Linux and Windows are not affected by this change.

happy to give this a spin as i have 2 macs - 1 which the install always works and 1 which i always have do battle with homebrew/python/etc.

However, @amchoukir could you elaborate on _exactly_ what problem this is solving? Does it relate to an issue on the tracker? Are there some obvious test steps we should go through?

Ta,
Ben

Thanks for having a look and giving it a try. 

@puremourning, @Valloric 

`CustomPythonCmakeArgs` uses `python-config` to figure out where python is installed in the current environment. Unfortunately `virtualenv` does not copy `python-config` when setting up a new environment, I will take this issue separately to the `virtualenv` github repo. Copying `python-config` to my `virtualenv` takes care of the `PythonLibs`.

But there is still an issue with `PythonInterp`  as can be seen from the install log below:

```
-- Found PythonLibs: /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.dylib (found suitable version "2.7.10", minimum required is "2.6")
-- Found PythonInterp: /usr/bin/python2.6 (found suitable version "2.6.9", minimum required is "2.6")
```

`PythonInterp` picks up the system default instead of the interpreter in my `virtualenv`. The fix I proposed take care of the issue in a similar way as `PythonLibs`.

My setup is as follows:

OS X 10.10.3 (Maverick)
Python 2.7.10 installed via homebrew
Macvim 7.4 installed via homebrew

The test steps I went through are:
- Make sure the same version of the lib and interpreter are picked during the install
- Check that you don't get vim crashing upon startup.

To my understanding `virtualenv` is a tool for python available not only on OSX, but the fix you're proposing is implemented in a function that is called only on OSX, so is the fix not needed on other platforms?

@vheon yes it might definitely be needed. I have access to linux servers on which I could try it out, if someone could have a look at windows. 

> @vheon yes it might definitely be needed.

@amchoukir Then this PR should be modified to run on linux as well right?

I think we should hold out on this change while I get the Travis OSX builds working. I've had to do some work to streamline the build which I think obsolete this change. 

> On 19 Aug 2015, at 18:46, Andrea Cedraro notifications@github.com wrote:
> 
> @vheon yes it might definitely be needed.
> 
> Then this PR should be modified to run on linux as well right?
> 
> —
> Reply to this email directly or view it on GitHub.

@puremourning Sure, np.

Here are the results of my tests on Linux

**Scenario 1: System default sanity check**

-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.6", minimum required is "2.6")
-- Found PythonInterp: /home/vagrant/.virtualenv/ycmd/bin/python2 (found suitable version "2.7.6", minimum required is "2.6")

**Scenario 2: Vanilla virtualenv with 2.7.10**

-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.6", minimum required is "2.6")
-- Found PythonInterp: /home/vagrant/.virtualenv/ycmd2.10/bin/python2 (found suitable version "2.7.10", minimum required is "2.6")

Wrong library picked up. It picked up the  system default instead of the 2.7.10.

**Scenario 3: virtualenv with 2.7.10 and python-config copied over**

-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.6", minimum required is "2.6")
-- Found PythonInterp: /home/vagrant/.virtualenv/ycmd2.10/bin/python2 (found suitable version "2.7.10", minimum required is "2.6")

Wrong library picked up. It picked up the  system default instead of the 2.7.10. So here python-config does not fix the issue. 

Here are the specifics of python-config

(ycmd2.10)vagrant@vvv:~/ycmd$ python-config --prefix
/usr/local/python/2.10
(ycmd2.10)vagrant@vvv:~/ycmd$ which python-config
/home/vagrant/.virtualenv/ycmd2.10/bin/python-config

Most likely this is due to the difference in paths between OS X and Linux

**Next Steps**

I will dig into build.py to check what happens.

> @amchoukir Unfortunately virtualenv does not copy python-config when setting up a new environment, I will take this issue separately to the virtualenv github repo. Copying python-config to my virtualenv takes care of the PythonLibs.

Yes, this is annoying. I found the same problem when making the ycmd build/test run on Travis OS X.  Did you raise this with virtualenv folks? If so could we get a reference?

Thanks,
Ben

@puremourning yes Paul opened the following issue:
https://github.com/pypa/virtualenv/pull/783

Hello Everyone, I am back from vacation and I will have a look at why the build does not work on virtualenv for Linux.

Meanwhile the fix to get the python-config copied in virtualenv has been merged:

https://github.com/pypa/virtualenv/pull/798



Resolving patch:
diff --git a/build.py b/build.py
--- a/build.py
+++ b/build.py
@@ -7,12 +7,17 @@
 from __future__ import division
 from __future__ import absolute_import
 
+from shutil import rmtree
+from tempfile import mkdtemp
+import errno
+import re
+import multiprocessing
 import os
-import subprocess
 import os.path as p
-import sys
+import platform
 import shlex
-import errno
+import subprocess
+import sys
 
 PY_MAJOR, PY_MINOR = sys.version_info[ 0 : 2 ]
 if not ( ( PY_MAJOR == 2 and PY_MINOR >= 6 ) or
@@ -33,11 +38,22 @@
 
 sys.path.insert( 1, p.abspath( p.join( DIR_OF_THIRD_PARTY, 'argparse' ) ) )
 
-from tempfile import mkdtemp
-from shutil import rmtree
-import platform
 import argparse
-import multiprocessing
+
+NO_DYNAMIC_PYTHON_ERROR = (
+  'ERROR: found static Python library ({library}) but a dynamic one is '
+  'required. You must use a Python compiled with the {flag} flag. '
+  'If using pyenv, you need to run the command:\n'
+  '  export PYTHON_CONFIGURE_OPTS="{flag}"\n'
+  'before installing a Python version.' )
+NO_PYTHON_LIBRARY_ERROR = 'ERROR: unable to find an appropriate Python library.'
+
+LIBRARY_LDCONFIG_REGEX = re.compile(
+  '(?P<library>\S+) \(.*\) => (?P<path>\S+)' )
+
+
+def OnLinux():
+  return platform.system() == 'Linux'
 
 
 def OnMac():
@@ -120,82 +136,125 @@ def CheckOutput( *popen_args, **kwargs ):
   return output
 
 
+def GetPythonNameOnUnix():
+  python_name = 'python' + str( PY_MAJOR ) + '.' + str( PY_MINOR )
+  # Python 3 has an 'm' suffix on Unix platforms, for instance libpython3.3m.so.
+  if PY_MAJOR == 3:
+    python_name += 'm'
+  return python_name
+
+
+def GetStandardPythonLocationsOnUnix( prefix, name ):
+  return ( '{0}/lib/lib{1}'.format( prefix, name ),
+           '{0}/include/{1}'.format( prefix, name ) )
+
+
+def FindPythonLibrariesOnLinux():
+  python_name = GetPythonNameOnUnix()
+  python_library_root, python_include = GetStandardPythonLocationsOnUnix(
+    sys.exec_prefix, python_name )
+
+  python_library = python_library_root + '.so'
+  if p.isfile( python_library ):
+    return python_library, python_include
+
+  python_library = python_library_root + '.a'
+  if p.isfile( python_library ):
+    sys.exit( NO_DYNAMIC_PYTHON_ERROR.format( library = python_library,
+                                              flag = '--enable-shared' ) )
+
+  # On some distributions (Ubuntu for instance), the Python system library is
+  # not installed in its default path: /usr/lib. We use the ldconfig tool to
+  # find it.
+  python_library = 'lib' + python_name + '.so'
+  ldconfig_output = CheckOutput( [ 'ldconfig', '-p' ] ).strip().decode( 'utf8' )
+  for line in ldconfig_output.splitlines():
+    match = LIBRARY_LDCONFIG_REGEX.search( line )
+    if match and match.group( 'library' ) == python_library:
+      return match.group( 'path' ), python_include
+
+  sys.exit( NO_PYTHON_LIBRARY_ERROR )
+
+
+def FindPythonLibrariesOnMac():
+  python_prefix = sys.exec_prefix
+
+  python_library = p.join( python_prefix, 'Python' )
+  if p.isfile( python_library ):
+    return python_library, p.join( python_prefix, 'Headers' )
+
+  python_name = GetPythonNameOnUnix()
+  python_library_root, python_include = GetStandardPythonLocationsOnUnix(
+    python_prefix, python_name )
+
+  # On MacOS, ycmd does not work with statically linked python library.
+  # It typically manifests with the following error when there is a
+  # self-compiled python without --enable-framework (or, technically
+  # --enable-shared):
+  #
+  #   Fatal Python error: PyThreadState_Get: no current thread
+  #
+  # The most likely explanation for this is that both the ycm_core.so and the
+  # python binary include copies of libpython.a (or whatever included
+  # objects). When the python interpreter starts it initializes only the
+  # globals within its copy, so when ycm_core.so's copy starts executing, it
+  # points at its own copy which is uninitialized.
+  #
+  # Some platforms' dynamic linkers (ld.so) are able to resolve this when
+  # loading shared libraries at runtime[citation needed], but OSX seemingly
+  # cannot.
+  #
+  # So we do 2 things special on OS X:
+  #  - look for a .dylib first
+  #  - if we find a .a, raise an error.
+  python_library = python_library_root + '.dylib'
+  if p.isfile( python_library ):
+    return python_library, python_include
+
+  python_library = python_library_root + '.a'
+  if p.isfile( python_library ):
+    sys.exit( NO_DYNAMIC_PYTHON_ERROR.format( library = python_library,
+                                              flag = '--enable-framework' ) )
+
+  sys.exit( NO_PYTHON_LIBRARY_ERROR )
+
+
+def FindPythonLibrariesOnWindows():
+  python_prefix = sys.exec_prefix
+  python_name = 'python' + str( PY_MAJOR ) + str( PY_MINOR )
+
+  python_library = p.join( python_prefix, 'libs', python_name + '.lib' )
+  if p.isfile( python_library ):
+    return python_library, p.join( python_prefix, 'include' )
+
+  sys.exit( NO_PYTHON_LIBRARY_ERROR )
+
+
+def FindPythonLibraries():
+  if OnLinux():
+    return FindPythonLibrariesOnLinux()
+
+  if OnMac():
+    return FindPythonLibrariesOnMac()
+
+  if OnWindows():
+    return FindPythonLibrariesOnWindows()
+
+  sys.exit( 'ERROR: your platform is not supported by this script. Follow the '
+            'Full Installation Guide instructions in the documentation.' )
+
+
 def CustomPythonCmakeArgs():
   # The CMake 'FindPythonLibs' Module does not work properly.
   # So we are forced to do its job for it.
+  print( 'Searching Python {major}.{minor} libraries...'.format(
+    major = PY_MAJOR, minor = PY_MINOR ) )
 
-  print( 'Searching for python libraries...' )
-
-  python_prefix = CheckOutput( [
-    'python-config',
-    '--prefix'
-  ] ).strip().decode( 'utf8' )
-
-  if p.isfile( p.join( python_prefix, '/Python' ) ):
-    python_library = p.join( python_prefix, '/Python' )
-    python_include = p.join( python_prefix, '/Headers' )
-    print( 'Using OSX-style libs from {0}'.format( python_prefix ) )
-  else:
-    major_minor = CheckOutput( [
-      'python',
-      '-c',
-      'import sys;i=sys.version_info;print( "%d.%d" % (i[0], i[1]) )'
-    ] ).strip().decode( 'utf8' )
-    which_python = 'python' + major_minor
-
-    # Python 3 has an 'm' suffix, for instance libpython3.3m.a
-    if major_minor.startswith( '3' ):
-      which_python += 'm'
-
-    lib_python = '{0}/lib/lib{1}'.format( python_prefix, which_python ).strip()
-
-    print( 'Searching for python with prefix: {0} and lib {1}:'.format(
-      python_prefix, which_python ) )
-
-    # On MacOS, ycmd does not work with statically linked python library.
-    # It typically manifests with the following error when there is a
-    # self-compiled python without --enable-framework (or, technically
-    # --enable-shared):
-    #
-    #   Fatal Python error: PyThreadState_Get: no current thread
-    #
-    # The most likely explanation for this is that both the ycm_core.so and the
-    # python binary include copies of libpython.a (or whatever included
-    # objects). When the python interpreter starts it initializes only the
-    # globals within its copy, so when ycm_core.so's copy starts executing, it
-    # points at its own copy which is uninitialized.
-    #
-    # Some platforms' dynamic linkers (ld.so) are able to resolve this when
-    # loading shared libraries at runtime[citation needed], but OSX seemingly
-    # cannot.
-    #
-    # So we do 2 things special on OS X:
-    #  - look for a .dylib first
-    #  - if we find a .a, raise an error.
-
-    if p.isfile( '{0}.dylib'.format( lib_python ) ):
-      python_library = '{0}.dylib'.format( lib_python )
-    elif p.isfile( '/usr/lib/lib{0}.dylib'.format( which_python ) ):
-      # For no clear reason, python2.6 only exists in /usr/lib on OS X and
-      # not in the python prefix location
-      python_library = '/usr/lib/lib{0}.dylib'.format( which_python )
-    elif p.isfile( '{0}.a'.format( lib_python ) ):
-      if OnMac():
-        sys.exit( 'ERROR: You must use a python compiled with '
-                  '--enable-shared or --enable-framework (and thus a {0}.dylib '
-                  'library) on OS X'.format( lib_python ) )
-
-      python_library = '{0}.a'.format( lib_python )
-    # This check is for CYGWIN
-    elif p.isfile( '{0}.dll.a'.format( lib_python ) ):
-      python_library = '{0}.dll.a'.format( lib_python )
-    else:
-      sys.exit( 'ERROR: Unable to find an appropriate python library' )
+  python_library, python_include = FindPythonLibraries()
 
-    python_include = '{0}/include/{1}'.format( python_prefix, which_python )
+  print( 'Found Python library: {0}'.format( python_library ) )
+  print( 'Found Python headers folder: {0}'.format( python_include ) )
 
-  print( 'Using PYTHON_LIBRARY={0} PYTHON_INCLUDE_DIR={1}'.format(
-      python_library, python_include ) )
   return [
     '-DPYTHON_LIBRARY={0}'.format( python_library ),
     '-DPYTHON_INCLUDE_DIR={0}'.format( python_include )
@@ -322,8 +381,7 @@ def BuildYcmdLib( args ):
 
   try:
     full_cmake_args = [ '-G', GetGenerator( args ) ]
-    if OnMac():
-      full_cmake_args.extend( CustomPythonCmakeArgs() )
+    full_cmake_args.extend( CustomPythonCmakeArgs() )
     full_cmake_args.extend( GetCmakeArgs( args ) )
     full_cmake_args.append( p.join( DIR_OF_THIS_SCRIPT, 'cpp' ) )
 


Test patch: