Repository: conan-io/conan

Issue:
tools.vcvars() and system PATH variable overflow
Windows OSes have limitations on the maximum system PATH variable size: https://software.intel.com/en-us/articles/limitation-to-the-length-of-the-system-path-variable

If you exceed the maximum size you'll get an empty PATH and everything stops working (I've just stumbled on that)

The way `tools.vcvars()` updates the environment, effectively doubling PATH size makes this even more likely to happen.

I would propose to filter out non-unique elements while updating the environment:
```python
def unique_list(lst): # preserving the order
    used = set()
    return [x for x in lst if x not in used and (used.add(x) or True)]

@contextmanager
def environment_append(env_vars):
    """
    :param env_vars: List of simple environment vars. {name: value, name2: value2} => e.g.: MYVAR=1
                     The values can also be lists of appendable environment vars. {name: [value, value2]}
                      => e.g. PATH=/path/1:/path/2
    :return: None
    """
    for name, value in env_vars.items():
        if isinstance(value, list):
            old = os.environ.get(name)
            if old:
                value += old.split(os.pathsep)
            env_vars[name] = os.pathsep.join(unique_list(value))
    if env_vars:
```
(this way it works for me) and maybe issue a warning if PATH becomes too big.








Discussion:
Hi! Which Conan version are you using? If I remember correctly in 1.7 we slightly changed the behavior of the `vcvars` so it returns only the paths changed in the PATH variable (also in some other env var). So the information shouldn't be duplicated. 

Nevertheless, I agree the `environment_append` code could be improved to check not repeated elements when they are lists.
> Hi! Which Conan version are you using? 

1.7.3. This was the first time I hit the problem on the same OS environment.

> If I remember correctly in 1.7 we slightly changed the behavior of the `vcvars` so it returns only the paths changed in the PATH variable (also in some other env var). So the information shouldn't be duplicated.

I'm afraid it is. My initial PATH size is about 2800 chars and if I remember my debugging data correctly it''s almost tripled after `tools.vcvars()` I can debug it again and provide more detail if you like.

> I can debug it again and provide more detail if you like.

Yes please, it would be useful to know the PATH values before and after the `tools.vcvars()` call (or any other affecting)
OK, here are my data. The following fragment of `conanfile.py`

```python
    def build(self, parallel="False"):
        print("before: size=%d, PATH=\n%s" % (len(os.environ["PATH"]), os.environ["PATH"].replace(';',"\n")))
        print("============================================================")
        with tools.vcvars(self.settings):
            print("after: size=%d, PATH=\n%s" % (len(os.environ["PATH"]), os.environ["PATH"].replace(';',"\n")))
```
yelds
```
before: size=2298, PATH=
D:/.conan\qp4auzo6\1\bin
C:\Users\dbely\.conan\data\cxr\1.0.0\dbely\stable\package\b2fcbc60b3bee75023a4f7934cadb723eec550bd\bin
C:\Users\dbely\.conan\data\jwasm\2.12\dbely\testing\package\60b04a80ac4dd5c23f1acffacc679e9902e636e5\bin
C:\Users\dbely\.conan\data\pccts\1.33MR33\dbely\stable\package\74ffb11f954ecf2f1211a6ccd197c5997da03768\bin
C:\Users\dbely\.opam\4.02.3+local+vc15+x86\bin
D:/.conan\p4g57n7p\1\x86\vc15\bin
C:\Users\dbely\.conan\data\cmake_installer\3.10.0\conan\stable\package\456f15897172eef340fcbac8a70811f2beb26a93\bin
C:\Users\dbely\.conan\data\opam_bin\1.2.2\dbely\testing\package\9372ebca25e5d8c584c6593f32288cc03608963c\bin
D:/.conan\51k__t5e\1\bin
C:\Program Files\Far Manager\ConEmu\Scripts
C:\Program Files\Far Manager
C:\Program Files\Far Manager\ConEmu
C:\Perl64\site\bin
C:\Perl64\bin
C:\Python37\Scripts\
C:\Python37\
C:\Python36\
C:\Python36\Scripts\
C:\Python27\
C:\Python27\Scripts
C:\Program Files (x86)\Common Files\Oracle\Java\javapath
C:\ProgramData\Oracle\Java\javapath
C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler
C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\ia32\compiler
C:\Windows\system32
C:\Windows
C:\Windows\System32\Wbem
C:\Windows\System32\WindowsPowerShell\v1.0\
C:\ProgramData\chocolatey\bin
C:\Program Files\Microsoft SQL Server\110\Tools\Binn\
C:\Program Files (x86)\Subversion\bin
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\Program Files\nodejs\
C:\Program Files\JAI\SDK\bin
C:\Program Files\JAI\SDK\bin\Win32_i86
C:\Program Files\JAI\SDK\GenICam\bin\Win64_x64
C:\Program Files\JAI\SDK\GenICam\bin\Win32_i86
C:\Program Files\LLVM\bin
C:\Program Files (x86)\vim\vim80
C:\Program Files\Calibre2\
C:\Program Files\Java\jdk1.8.0_162\bin
C:\Program Files\Java\jdk1.8.0_172\bin
C:\Program Files\Git\cmd
C:\Program Files\Microsoft VS Code\bin
C:\Program Files\Java\jdk1.8.0_181\bin
C:\Program Files\TortoiseSVN\bin
C:\Program Files (x86)\GitExtensions\
C:\Tcl\bin
C:\ProgramData\chocolatey\lib\msys2
C:\tools\msys64
C:\Users\dbely\AppData\Roaming\npm
C:\Program Files (x86)\Microsoft VS Code Insiders\bin
C:\Users\dbely\AppData\Local\Microsoft\WindowsApps
C:\Program Files\Microsoft VS Code\bin
C:\local\runtime
C:\local\Dependencies
============================================================
after: size=6241, PATH=
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.15.26726\bin\HostX64\x86
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.15.26726\bin\HostX64\x64
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\VC\VCPackages
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\TestWindow
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\bin\Roslyn
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Team Tools\Performance Tools
C:\Program Files (x86)\Microsoft Visual Studio\Shared\Common\VSPerfCollectionTools\
C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\x64\
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\FSharp\
C:\Program Files (x86)\Windows Kits\10\bin\10.0.17134.0\x64
C:\Program Files (x86)\Windows Kits\10\bin\x64
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\\MSBuild\15.0\bin
C:\Windows\Microsoft.NET\Framework64\v4.0.30319
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\Tools\
D:/.conan\qp4auzo6\1\bin
C:\Users\dbely\.conan\data\cxr\1.0.0\dbely\stable\package\b2fcbc60b3bee75023a4f7934cadb723eec550bd\bin
C:\Users\dbely\.conan\data\jwasm\2.12\dbely\testing\package\60b04a80ac4dd5c23f1acffacc679e9902e636e5\bin
C:\Users\dbely\.conan\data\pccts\1.33MR33\dbely\stable\package\74ffb11f954ecf2f1211a6ccd197c5997da03768\bin
C:\Users\dbely\.opam\4.02.3+local+vc15+x86\bin
D:/.conan\p4g57n7p\1\x86\vc15\bin
C:\Users\dbely\.conan\data\cmake_installer\3.10.0\conan\stable\package\456f15897172eef340fcbac8a70811f2beb26a93\bin
C:\Users\dbely\.conan\data\opam_bin\1.2.2\dbely\testing\package\9372ebca25e5d8c584c6593f32288cc03608963c\bin
D:/.conan\51k__t5e\1\bin
C:\Program Files\Far Manager\ConEmu\Scripts
C:\Program Files\Far Manager
C:\Program Files\Far Manager\ConEmu
C:\Perl64\site\bin
C:\Perl64\bin
C:\Python37\Scripts\
C:\Python37\
C:\Python36\
C:\Python36\Scripts\
C:\Python27\
C:\Python27\Scripts
C:\Program Files (x86)\Common Files\Oracle\Java\javapath
C:\ProgramData\Oracle\Java\javapath
C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler
C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\ia32\compiler
C:\Windows\system32
C:\Windows
C:\Windows\System32\Wbem
C:\Windows\System32\WindowsPowerShell\v1.0\
C:\ProgramData\chocolatey\bin
C:\Program Files\Microsoft SQL Server\110\Tools\Binn\
C:\Program Files (x86)\Subversion\bin
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\Program Files\nodejs\
C:\Program Files\JAI\SDK\bin
C:\Program Files\JAI\SDK\bin\Win32_i86
C:\Program Files\JAI\SDK\GenICam\bin\Win64_x64
C:\Program Files\JAI\SDK\GenICam\bin\Win32_i86
C:\Program Files\LLVM\bin
C:\Program Files (x86)\vim\vim80
C:\Program Files\Calibre2\
C:\Program Files\Java\jdk1.8.0_162\bin
C:\Program Files\Java\jdk1.8.0_172\bin
C:\Program Files\Git\cmd
C:\Program Files\Microsoft VS Code\bin
C:\Program Files\Java\jdk1.8.0_181\bin
C:\Program Files\TortoiseSVN\bin
C:\Program Files (x86)\GitExtensions\
C:\Tcl\bin
C:\ProgramData\chocolatey\lib\msys2
C:\tools\msys64
C:\Users\dbely\AppData\Roaming\npm
C:\Program Files (x86)\Microsoft VS Code Insiders\bin
C:\Users\dbely\AppData\Local\Microsoft\WindowsApps
C:\Program Files\Microsoft VS Code\bin
C:\local\runtime
C:\local\Dependencies
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja
D:/.conan\qp4auzo6\1\bin
C:\Users\dbely\.conan\data\cxr\1.0.0\dbely\stable\package\b2fcbc60b3bee75023a4f7934cadb723eec550bd\bin
C:\Users\dbely\.conan\data\jwasm\2.12\dbely\testing\package\60b04a80ac4dd5c23f1acffacc679e9902e636e5\bin
C:\Users\dbely\.conan\data\pccts\1.33MR33\dbely\stable\package\74ffb11f954ecf2f1211a6ccd197c5997da03768\bin
C:\Users\dbely\.opam\4.02.3+local+vc15+x86\bin
D:/.conan\p4g57n7p\1\x86\vc15\bin
C:\Users\dbely\.conan\data\cmake_installer\3.10.0\conan\stable\package\456f15897172eef340fcbac8a70811f2beb26a93\bin
C:\Users\dbely\.conan\data\opam_bin\1.2.2\dbely\testing\package\9372ebca25e5d8c584c6593f32288cc03608963c\bin
D:/.conan\51k__t5e\1\bin
C:\Program Files\Far Manager\ConEmu\Scripts
C:\Program Files\Far Manager
C:\Program Files\Far Manager\ConEmu
C:\Perl64\site\bin
C:\Perl64\bin
C:\Python37\Scripts\
C:\Python37\
C:\Python36\
C:\Python36\Scripts\
C:\Python27\
C:\Python27\Scripts
C:\Program Files (x86)\Common Files\Oracle\Java\javapath
C:\ProgramData\Oracle\Java\javapath
C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler
C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\ia32\compiler
C:\Windows\system32
C:\Windows
C:\Windows\System32\Wbem
C:\Windows\System32\WindowsPowerShell\v1.0\
C:\ProgramData\chocolatey\bin
C:\Program Files\Microsoft SQL Server\110\Tools\Binn\
C:\Program Files (x86)\Subversion\bin
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\Program Files\nodejs\
C:\Program Files\JAI\SDK\bin
C:\Program Files\JAI\SDK\bin\Win32_i86
C:\Program Files\JAI\SDK\GenICam\bin\Win64_x64
C:\Program Files\JAI\SDK\GenICam\bin\Win32_i86
C:\Program Files\LLVM\bin
C:\Program Files (x86)\vim\vim80
C:\Program Files\Calibre2\
C:\Program Files\Java\jdk1.8.0_162\bin
C:\Program Files\Java\jdk1.8.0_172\bin
C:\Program Files\Git\cmd
C:\Program Files\Microsoft VS Code\bin
C:\Program Files\Java\jdk1.8.0_181\bin
C:\Program Files\TortoiseSVN\bin
C:\Program Files (x86)\GitExtensions\
C:\Tcl\bin
C:\ProgramData\chocolatey\lib\msys2
C:\tools\msys64
C:\Users\dbely\AppData\Roaming\npm
C:\Program Files (x86)\Microsoft VS Code Insiders\bin
C:\Users\dbely\AppData\Local\Microsoft\WindowsApps
C:\Program Files\Microsoft VS Code\bin
C:\local\runtime
C:\local\Dependencies
```

Note duplicate entries, e.g. `C:\Program Files\JAI\SDK\bin`

For the refence, `vcvars64.bat` gives me
```
size=3349, PATH=
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.15.26726\bin\HostX64\x64
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\VC\VCPackages
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\TestWindow
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\bin\Roslyn
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Team Tools\Performance Tools\x64
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Team Tools\Performance Tools
C:\Program Files (x86)\Microsoft Visual Studio\Shared\Common\VSPerfCollectionTools\\x64
C:\Program Files (x86)\Microsoft Visual Studio\Shared\Common\VSPerfCollectionTools\
C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\x64\
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\FSharp\
C:\Program Files (x86)\Windows Kits\10\bin\10.0.17134.0\x64
C:\Program Files (x86)\Windows Kits\10\bin\x64
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\\MSBuild\15.0\bin
C:\Windows\Microsoft.NET\Framework64\v4.0.30319
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\Tools\
C:\Program Files\Far Manager\ConEmu\Scripts
C:\Program Files\Far Manager
C:\Program Files\Far Manager\ConEmu
C:\Perl64\site\bin
C:\Perl64\bin
C:\Python37\Scripts\
C:\Python37\
C:\Python36\
C:\Python36\Scripts\
C:\Python27\
C:\Python27\Scripts
C:\Program Files (x86)\Common Files\Oracle\Java\javapath
C:\ProgramData\Oracle\Java\javapath
C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\intel64\compiler
C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\ia32\compiler
C:\Windows\system32
C:\Windows
C:\Windows\System32\Wbem
C:\Windows\System32\WindowsPowerShell\v1.0\
C:\ProgramData\chocolatey\bin
C:\Program Files\Microsoft SQL Server\110\Tools\Binn\
C:\Program Files (x86)\Subversion\bin
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\Program Files\nodejs\
C:\Program Files\JAI\SDK\bin
C:\Program Files\JAI\SDK\bin\Win32_i86
C:\Program Files\JAI\SDK\GenICam\bin\Win64_x64
C:\Program Files\JAI\SDK\GenICam\bin\Win32_i86
C:\Program Files\LLVM\bin
C:\Program Files (x86)\vim\vim80
C:\Program Files\Calibre2\
C:\Program Files\Java\jdk1.8.0_162\bin
C:\Program Files\Java\jdk1.8.0_172\bin
C:\Program Files\Git\cmd
C:\Program Files\Microsoft VS Code\bin
C:\Program Files\Java\jdk1.8.0_181\bin
C:\Program Files\TortoiseSVN\bin
C:\Program Files (x86)\GitExtensions\
C:\Tcl\bin
C:\ProgramData\chocolatey\lib\msys2
C:\tools\msys64
C:\Users\dbely\AppData\Roaming\npm
C:\Program Files (x86)\Microsoft VS Code Insiders\bin
C:\Users\dbely\AppData\Local\Microsoft\WindowsApps
C:\Program Files\Microsoft VS Code\bin
C:\local\runtime
C:\local\Dependencies
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja
```

Thanks, looks like a bug
it seems like I am running into the same issue with libvpx package
Ok, I'm taking a look for 1.8

Resolving patch:
diff --git a/conans/client/tools/win.py b/conans/client/tools/win.py
--- a/conans/client/tools/win.py
+++ b/conans/client/tools/win.py
@@ -306,7 +306,7 @@ def vcvars_command(settings, arch=None, compiler_version=None, force=False, vcva
 
 def vcvars_dict(settings, arch=None, compiler_version=None, force=False, filter_known_paths=False,
                 vcvars_ver=None, winsdk_version=None, only_diff=True):
-    known_path_lists = ("INCLUDE", "LIB", "LIBPATH", "PATH")
+    known_path_lists = ("include", "lib", "libpath", "path")
     cmd = vcvars_command(settings, arch=arch,
                          compiler_version=compiler_version, force=force,
                          vcvars_ver=vcvars_ver, winsdk_version=winsdk_version) + " && echo __BEGINS__ && set"
@@ -324,14 +324,17 @@ def vcvars_dict(settings, arch=None, compiler_version=None, force=False, filter_
             continue
         try:
             name_var, value = line.split("=", 1)
-            new_value = value.split(os.pathsep) if name_var in known_path_lists else value
+            new_value = value.split(os.pathsep) if name_var.lower() in known_path_lists else value
             # Return only new vars & changed ones, but only with the changed elements if the var is
             # a list
             if only_diff:
                 old_value = os.environ.get(name_var)
-                # The new value ends with separator and the old value, is a list, get only the
-                # new elements
-                if old_value and value.endswith(os.pathsep + old_value):
+                if name_var.lower() == "path":
+                    old_values_lower = [v.lower() for v in old_value.split(os.pathsep)]
+                    # Clean all repeated entries, not append if the element was already there
+                    new_env[name_var] = [v for v in new_value if v.lower() not in old_values_lower]
+                elif old_value and value.endswith(os.pathsep + old_value):
+                    # The new value ends with separator and the old value, is a list, get only the new elements
                     new_env[name_var] = value[:-(len(old_value) + 1)].split(os.pathsep)
                 elif value != old_value:
                     # Only if the vcvars changed something, we return the variable,
@@ -355,7 +358,6 @@ def relevant_path(path):
 
     return new_env
 
-
 @contextmanager
 def vcvars(*args, **kwargs):
     if platform.system() == "Windows":


Test patch:
diff --git a/conans/test/util/tools_test.py b/conans/test/util/tools_test.py
--- a/conans/test/util/tools_test.py
+++ b/conans/test/util/tools_test.py
@@ -710,6 +710,31 @@ def vcvars_echo_test(self):
             self.assertIn("Conan:vcvars already set", str(output))
             self.assertIn("VS140COMNTOOLS=", str(output))
 
+    @unittest.skipUnless(platform.system() == "Windows", "Requires Windows")
+    def vcvars_env_not_duplicated_path_test(self):
+        """vcvars is not looking at the current values of the env vars, with PATH it is a problem because you
+        can already have set some of the vars and accumulate unnecessary entries."""
+        settings = Settings.loads(default_settings_yml)
+        settings.os = "Windows"
+        settings.compiler = "Visual Studio"
+        settings.compiler.version = "15"
+        settings.arch = "x86"
+        settings.arch_build = "x86_64"
+
+        # Set the env with a PATH containing the vcvars paths
+        tmp = tools.vcvars_dict(settings, only_diff=False)
+        tmp = {key.lower(): value for key, value in tmp.items()}
+        with tools.environment_append({"path": tmp["path"]}):
+            previous_path = os.environ["PATH"].split(";")
+            # Duplicate the path, inside the tools.vcvars shouldn't have repeated entries in PATH
+            with tools.vcvars(settings):
+                path = os.environ["PATH"].split(";")
+                values_count = {value: path.count(value) for value in path}
+                for value, counter in values_count.items():
+                    if value and counter > 1 and previous_path.count(value) != counter:
+                        # If the entry was already repeated before calling "tools.vcvars" we keep it
+                        self.fail("The key '%s' has been repeated" % value)
+
     @unittest.skipUnless(platform.system() == "Windows", "Requires Windows")
     def vcvars_amd64_32_cross_building_support_test(self):
         # amd64_x86 crossbuilder