Repository: conan-io/conan

Issue:
Global short_paths
We are trying to standardize Keysight on conan and one of the big problems we are having in DevOps is that of filename too long for MS builds.  While we all recognize this to be a Microsoft problem that will never be fixed by them it is not practical for us to have to re-package 3rd party (or in-house) conan packages by adding short_paths=True to every package in our repo.  Is there is a global override I am unaware of that forces all packages to using short paths or is this something that may be added in the next release please?  We build across a number of targets with a large number of packages and it would become a full time project just linting conanfiles for short_paths.  Hoping for an option in conan.config if possible.



Discussion:
Hi @groovyd 

The truth is that Msft has already fixed it, if you are in Windows 10, using py36, you can enable the py36 "long paths" api, and it works. I guess you are constrained to develop for older platforms?

The global application of ``short_paths`` has not been implemented because using short_paths is not the nicest/more elegant solution. It puts the shorted paths in the C:/ folder, or you need to explicitly set some CONAN_USER_HOME_SHORT env var.

Also, I don't fully understand why adding ``short_paths=True`` to all recipes is such a big deal. For your own recipes, it is a trivial change, isn't it? If it is because you are building the packages directly from recipes existing in conan-center or other bintray repos, I wouldn't fully discard myself the idea of forking the recipes under my own control. I know it is not the most convenient approach, but also owning the full origin of packages is more robust and you are already prepared if want to improve the packages in some other way. In the same way I prefer to host the binaries I am using in my own Artifactory, I like to build them, even for third-parties from forks under my control.

However, given that python is so dynamic, I am thinking: Have you tried to directly change the global ``short_paths`` default? Something like this in your consuming recipe:

```python
from conans import ConanFile

ConanFile.short_paths=True

class MyPkg(ConanFile):
    requires = "...."
```

As long as this recipe is the first one that is parsed, it might work. What do you think? Thanks!
Hi James and thanks for the quick response.  The truth is we leverage both custom and existing conan packages from many offices all over the globe and other open source repos.  I am requesting this feature since it has already proven to be very difficult to curate and correct and fork all of these packages from all of these places.  Someone in the organization is always uploading something new that is not using short_paths and it always breaks the builds and causes a big stir.  Ideally if there were a global option in config we would never have to confront this issue.  We have already enabled the long file support in the recent toolchains and it is true we have to support older toolchains too but it appears that enabling support does not always solve the problem.  MSBuild will still complain often.  We have also tried the changing of the CONAN_USER_HOME_SHORT and shortened our devops vm’s build paths as much as possible to the point it is sometimes difficult to even figure out what is going on when it fails.

Additionally there seems to be a path length issue with vcvars() which I guess is an unrelated issue to short_paths but still causes issues for us but that only seems to cause a problem in VS2017 builds and not VS2015 as often.

I understand is globalizing an override for all packages to enable short_paths is difficult to implement but it would be greatly appreciated.  In DevOps we try to set some standards but at the end of the day when working across hundreds of teams and thousands of developers around the globe sometimes the no-op solution is the best.

> On Sep 20, 2018, at 3:58 PM, James <notifications@github.com> wrote:
> 
> Hi @groovyd
> 
> The truth is that Msft has already fixed it, if you are in Windows 10, using py36, you can enable the py36 "long paths" api, and it works. I guess you are constrained to develop for older platforms?
> 
> The global application of short_paths has not been implemented because using short_paths is not the nicest/more elegant solution. It puts the shorted paths in the C:/ folder, or you need to explicitly set some CONAN_USER_HOME_SHORT env var.
> 
> Also, I don't fully understand why adding short_paths=True to all recipes is such a big deal. For your own recipes, it is a trivial change, isn't it? If it is because you are building the packages directly from recipes existing in conan-center or other bintray repos, I wouldn't fully discard myself the idea of forking the recipes under my own control. I know it is not the most convenient approach, but also owning the full origin of packages is more robust and you are already prepared if want to improve the packages in some other way. In the same way I prefer to host the binaries I am using in my own Artifactory, I like to build them, even for third-parties from forks under my control.
> 
> However, given that python is so dynamic, I am thinking: Have you tried to directly change the global short_paths default? Something like this in your consuming recipe:
> 
> from conans import
>  ConanFile
> 
> ConanFile.short_paths
> =True
> 
> 
> 
> class MyPkg(ConanFile
> ):
>     requires 
> = "...."
> As long as this recipe is the first one that is parsed, it might work. What do you think? Thanks!
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
> 


..an additional example is for instance just today a few developers uploaded sub minor upgrades to some 3rd party packages without modifying them to short_paths and a dozen of so projects that were floating with their version requirements around minor versions of the package imported them and it botched up all the builds for quite some time until the dev was tracked down to remove or re-commit those packages.  It effected a number of teams depending on those packages but not focused on them in their day to day routines.

> On Sep 20, 2018, at 3:58 PM, James <notifications@github.com> wrote:
> 
> Hi @groovyd
> 
> The truth is that Msft has already fixed it, if you are in Windows 10, using py36, you can enable the py36 "long paths" api, and it works. I guess you are constrained to develop for older platforms?
> 
> The global application of short_paths has not been implemented because using short_paths is not the nicest/more elegant solution. It puts the shorted paths in the C:/ folder, or you need to explicitly set some CONAN_USER_HOME_SHORT env var.
> 
> Also, I don't fully understand why adding short_paths=True to all recipes is such a big deal. For your own recipes, it is a trivial change, isn't it? If it is because you are building the packages directly from recipes existing in conan-center or other bintray repos, I wouldn't fully discard myself the idea of forking the recipes under my own control. I know it is not the most convenient approach, but also owning the full origin of packages is more robust and you are already prepared if want to improve the packages in some other way. In the same way I prefer to host the binaries I am using in my own Artifactory, I like to build them, even for third-parties from forks under my control.
> 
> However, given that python is so dynamic, I am thinking: Have you tried to directly change the global short_paths default? Something like this in your consuming recipe:
> 
> from conans import
>  ConanFile
> 
> ConanFile.short_paths
> =True
> 
> 
> 
> class MyPkg(ConanFile
> ):
>     requires 
> = "...."
> As long as this recipe is the first one that is parsed, it might work. What do you think? Thanks!
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
> 


Hi @groovyd 

No problem, thanks for the feedback. We will try to add it, though next 1.8 release is already very packed. Labelling as good first issue, maybe some contributor could help, otherwise it might need to wait until 1.9.
Great thanks!  This will really solve a number of issues for us and we will be looking forward to it.

> On Sep 24, 2018, at 9:11 AM, James <notifications@github.com> wrote:
> 
> Hi @groovyd
> 
> No problem, thanks for the feedback. We will try to add it, though next 1.8 release is already very packed. Labelling as good first issue, maybe some contributor could help, otherwise it might need to wait until 1.9.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
> 



Resolving patch:
diff --git a/conans/client/conf/__init__.py b/conans/client/conf/__init__.py
--- a/conans/client/conf/__init__.py
+++ b/conans/client/conf/__init__.py
@@ -99,6 +99,7 @@
 # pylintrc = path/to/pylintrc_file    # environment CONAN_PYLINTRC
 # cache_no_locks = True               # Disable locking mechanism of local cache
 # user_home_short = your_path         # environment CONAN_USER_HOME_SHORT
+# use_always_short_paths = False      # environment CONAN_USE_ALWAYS_SHORT_PATHS
 # skip_vs_projects_upgrade = False    # environment CONAN_SKIP_VS_PROJECTS_UPGRADE
 # non_interactive = False             # environment CONAN_NON_INTERACTIVE
 
@@ -175,6 +176,7 @@ def env_vars(self):
                "CONAN_CPU_COUNT": self._env_c("general.cpu_count", "CONAN_CPU_COUNT", None),
                "CONAN_READ_ONLY_CACHE": self._env_c("general.read_only_cache", "CONAN_READ_ONLY_CACHE", None),
                "CONAN_USER_HOME_SHORT": self._env_c("general.user_home_short", "CONAN_USER_HOME_SHORT", None),
+               "CONAN_USE_ALWAYS_SHORT_PATHS": self._env_c("general.use_always_short_paths", "CONAN_USE_ALWAYS_SHORT_PATHS", None),
                "CONAN_VERBOSE_TRACEBACK": self._env_c("general.verbose_traceback", "CONAN_VERBOSE_TRACEBACK", None),
                # http://www.vtk.org/Wiki/CMake_Cross_Compiling
                "CONAN_CMAKE_GENERATOR": self._env_c("general.cmake_generator", "CONAN_CMAKE_GENERATOR", None),
diff --git a/conans/util/windows.py b/conans/util/windows.py
--- a/conans/util/windows.py
+++ b/conans/util/windows.py
@@ -1,8 +1,9 @@
 import os
 import subprocess
+import tempfile
 
 from conans.util.files import load, mkdir, save, rmdir
-import tempfile
+from conans.util.env_reader import get_env
 
 
 CONAN_LINK = ".conan_link"
@@ -37,6 +38,9 @@ def path_shortener(path, short_paths):
     True: Always shorten the path, create link if not existing
     None: Use shorten path only if already exists, not create
     """
+    use_always_short_paths = get_env("CONAN_USE_ALWAYS_SHORT_PATHS", False)
+    short_paths = use_always_short_paths or short_paths
+
     if short_paths is False or os.getenv("CONAN_USER_HOME_SHORT") == "None":
         return path
     link = os.path.join(path, CONAN_LINK)


Test patch:
diff --git a/conans/test/functional/paths_test.py b/conans/test/functional/paths_test.py
--- a/conans/test/functional/paths_test.py
+++ b/conans/test/functional/paths_test.py
@@ -1,10 +1,18 @@
 import os
 import platform
 import unittest
+import tempfile
+import shutil
+from parameterized import parameterized
+from mock import mock
+
 from conans.paths import (BUILD_FOLDER, PACKAGES_FOLDER, EXPORT_FOLDER, conan_expand_user,
                           SimplePaths)
 from conans.model.ref import ConanFileReference, PackageReference
 from conans.test.utils.test_files import temp_folder
+from conans.client.tools.env import environment_append
+from conans.client.tools.files import save
+from conans.util.windows import path_shortener
 
 
 class PathsTest(unittest.TestCase):
@@ -37,3 +45,43 @@ def basic_test(self):
         self.assertEqual(paths.package(package_ref),
                          os.path.join(paths.store, expected_base, PACKAGES_FOLDER,
                                       "456fa678eae68"))
+
+
+@mock.patch("subprocess.check_output", return_value=None)
+class PathShortenerTest(unittest.TestCase):
+
+    def run(self, *args, **kwargs):
+        self.home_short = tempfile.mkdtemp(suffix='_home_short')
+        self.home = tempfile.mkdtemp(suffix="_home")
+        try:
+            with environment_append({"CONAN_USER_HOME_SHORT": self.home_short,
+                                     "USERNAME": "jgsogo"}):
+                super(PathShortenerTest, self).run(*args, **kwargs)
+        finally:
+            shutil.rmtree(self.home_short, ignore_errors=True)
+            shutil.rmtree(self.home, ignore_errors=True)
+
+    @parameterized.expand([(False,), (True,)])
+    def test_default(self, _, short_paths):
+        p = tempfile.mkdtemp(dir=self.home)
+        r = path_shortener(path=p, short_paths=short_paths)
+
+        self.assertEqual(self.home_short in r, short_paths)
+        self.assertEqual(self.home in r, not short_paths)
+
+    @parameterized.expand([(False,), (True,)])
+    def test_with_env_variable(self, _, short_paths):
+        with environment_append({'CONAN_USE_ALWAYS_SHORT_PATHS': "True"}):
+            p = tempfile.mkdtemp(dir=self.home)
+            r = path_shortener(path=p, short_paths=short_paths)
+
+            self.assertEqual(self.home_short in r, True)
+            self.assertEqual(self.home in r, False)
+
+        with environment_append({'CONAN_USE_ALWAYS_SHORT_PATHS': "False"}):
+            p = tempfile.mkdtemp(dir=self.home)
+            r = path_shortener(path=p, short_paths=short_paths)
+
+            self.assertEqual(self.home_short in r, short_paths)
+            self.assertEqual(self.home in r, not short_paths)
+