Repository: conan-io/conan

Issue:
Cannot build using Conan and Visual Studio 2017 in Japanese Environment
Tests are with conan 1.4.4, Python 2.7.10, Windows 7

When trying to build a package in the local cache
 (conan install MyPackage/1.0.0@user/channel --build=missing)
I get the following error

        vcvars = tools.vcvars_command(self.settings)
        ValueError: Invalid \escape: line 12 column 54 (char 597)

This occurred after installing Visual Studio 2017.

The environment is Japanese Windows 7, but set to English environment. This sets the UI to English but many system strings still appear in Japanese.

Tracing through conan source shows the problem to be in the line
    return json.loads(vswhere_out)
of vswhere in win.py.

The output of vswhere.exe for VS2017 contains the line
 "description": "生産性向上と、さまざまな規模のチーム間の調整のための Microsoft DevOps ソリューション",
(This is line 12 of the vswhere.exe output)

Adding the code 

    output = '\n'.join(line for line in output.split('\n') if 'description' not in line)

to strip out this line from the vshwere.exe output before the line

    vswhere_out = decode_text(output).strip()

in vswhere in win.py allows the build to proceed normally.

So it seems the Japanese message in the json from vswhere is causing the loading of the json to fail.

When running vshwere.exe directly, the output is in utf-8 but this bug
 https://github.com/Microsoft/vswhere/issues/146
suggests that sometimes the output is not in utf-8. 









Discussion:
This is the local output of vswhere
[where.txt](https://github.com/conan-io/conan/files/2093321/where.txt)

Have you tried using python 3.X instead?

The fact is that python 2 support for unicode is a bit complicated, to say something. We have been fixing some things, but still python 3 is strongly recommended. Not only for this reason, but the deprecation of python 2 has already started, as some of conan dependencies have started to be python 3 only.

Please try python 3 if possible. Thanks!

The output of vswhere depends on the currently active console code page when redirecting to a file or writing to a pipe. Microsoft/vswhere#146 is about adding a new switch to force UTF8 regardless of the active code page, so once I release from master (2.5) I recommend you rely on that version by redistributing. I'll try to get it into the next Visual Studi 2017 update, but by redistributing it or otherwise taking a dependency on the version release here from GH, you can be sure it supports the parameters you require.
I tried with Python 3.6.5  and it still has the same problem. The error changes to 

    JSONDecodeError: Invalid \escape: line 12 column 54 (char 597)


You need to use the private build (a release is currently building) and pass the `-utf8` option. I didn't change the default encoding because it would be a breaking change.
As @dibben-jsol suggested, we should take a look at the Conan code and see if we can "decode" or "ignore" or "strip" the offending characters from the json. The "description" field is not relevant for us.

Thanks @heaths for your collaboration 😃 , defininitely a `utf8` could help but probably is complex for Conan to know when the new parameter can be applied or not to `vswhere`.





Resolving patch:
diff --git a/conans/client/tools/win.py b/conans/client/tools/win.py
--- a/conans/client/tools/win.py
+++ b/conans/client/tools/win.py
@@ -5,6 +5,7 @@
 import deprecation
 
 import subprocess
+
 from contextlib import contextmanager
 
 from conans.client.tools.env import environment_append
@@ -136,8 +137,6 @@ def vswhere(all_=False, prerelease=False, products=None, requires=None, version=
 
     program_files = os.environ.get("ProgramFiles(x86)", os.environ.get("ProgramFiles"))
 
-    vswhere_path = ""
-
     if program_files:
         vswhere_path = os.path.join(program_files, "Microsoft Visual Studio", "Installer",
                                     "vswhere.exe")
@@ -186,11 +185,16 @@ def vswhere(all_=False, prerelease=False, products=None, requires=None, version=
 
     try:
         output = subprocess.check_output(arguments)
-        vswhere_out = decode_text(output).strip()
+        output = decode_text(output).strip()
+        # Ignore the "description" field, that even decoded contains non valid charsets for json
+        # (ignored ones)
+        output = "\n".join([line for line in output.splitlines()
+                            if not line.strip().startswith('"description"')])
+
     except (ValueError, subprocess.CalledProcessError, UnicodeDecodeError) as e:
         raise ConanException("vswhere error: %s" % str(e))
 
-    return json.loads(vswhere_out)
+    return json.loads(output)
 
 
 def vs_comntools(compiler_version):


Test patch:
diff --git a/conans/test/util/tools_test.py b/conans/test/util/tools_test.py
--- a/conans/test/util/tools_test.py
+++ b/conans/test/util/tools_test.py
@@ -6,6 +6,9 @@
 import unittest
 
 from collections import namedtuple
+
+import six
+from mock.mock import patch, mock_open
 from six import StringIO
 
 from conans.client.client_cache import CONAN_CONF
@@ -14,7 +17,7 @@
 from conans.client.conan_api import ConanAPIV1
 from conans.client.conf import default_settings_yml, default_client_conf
 from conans.client.output import ConanOutput
-from conans.client.tools.win import vcvars_dict
+from conans.client.tools.win import vcvars_dict, vswhere
 from conans.client.tools.scm import Git
 
 from conans.errors import ConanException, NotFoundException
@@ -516,9 +519,8 @@ def system_package_tool_installed_test(self):
         # This package hopefully doesn't exist
         self.assertFalse(spt._tool.installed("oidfjgesiouhrgioeurhgielurhgaeiorhgioearhgoaeirhg"))
 
+    @unittest.skipUnless(platform.system() == "Windows", "Requires vswhere")
     def msvc_build_command_test(self):
-        if platform.system() != "Windows":
-            return
         settings = Settings.loads(default_settings_yml)
         settings.os = "Windows"
         settings.compiler = "Visual Studio"
@@ -535,12 +537,77 @@ def msvc_build_command_test(self):
         with self.assertRaisesRegexp(ConanException, "Cannot build_sln_command"):
             tools.msvc_build_command(settings, "project.sln")
 
-        # succesful definition via settings
+        # successful definition via settings
         settings.build_type = "Debug"
         cmd = tools.msvc_build_command(settings, "project.sln")
         self.assertIn('msbuild project.sln /p:Configuration=Debug /p:Platform="x86"', cmd)
         self.assertIn('vcvarsall.bat', cmd)
 
+    @unittest.skipUnless(platform.system() == "Windows", "Requires vswhere")
+    def vswhere_description_strip_test(self):
+        myoutput = """
+[
+  {
+    "instanceId": "17609d7c",
+    "installDate": "2018-06-11T02:15:04Z",
+    "installationName": "VisualStudio/15.7.3+27703.2026",
+    "installationPath": "",
+    "installationVersion": "15.7.27703.2026",
+    "productId": "Microsoft.VisualStudio.Product.Enterprise",
+    "productPath": "",
+    "isPrerelease": false,
+    "displayName": "Visual Studio Enterprise 2017",
+    "description": "生産性向上と、さまざまな規模のチーム間の調整のための Microsoft DevOps ソリューション",
+    "channelId": "VisualStudio.15.Release",
+    "channelUri": "https://aka.ms/vs/15/release/channel",
+    "enginePath": "",
+    "releaseNotes": "https://go.microsoft.com/fwlink/?LinkId=660692#15.7.3",
+    "thirdPartyNotices": "https://go.microsoft.com/fwlink/?LinkId=660708",
+    "updateDate": "2018-06-11T02:15:04.7009868Z",
+    "catalog": {
+      "buildBranch": "d15.7",
+      "buildVersion": "15.7.27703.2026",
+      "id": "VisualStudio/15.7.3+27703.2026",
+      "localBuild": "build-lab",
+      "manifestName": "VisualStudio",
+      "manifestType": "installer",
+      "productDisplayVersion": "15.7.3",
+      "productLine": "Dev15",
+      "productLineVersion": "2017",
+      "productMilestone": "RTW",
+      "productMilestoneIsPreRelease": "False",
+      "productName": "Visual Studio",
+      "productPatchVersion": "3",
+      "productPreReleaseMilestoneSuffix": "1.0",
+      "productRelease": "RTW",
+      "productSemanticVersion": "15.7.3+27703.2026",
+      "requiredEngineVersion": "1.16.1187.57215"
+    },
+    "properties": {
+      "campaignId": "",
+      "canceled": "0",
+      "channelManifestId": "VisualStudio.15.Release/15.7.3+27703.2026",
+      "nickname": "",
+      "setupEngineFilePath": ""
+    }
+  },
+  {
+    "instanceId": "VisualStudio.12.0",
+    "installationPath": "",
+    "installationVersion": "12.0"
+  }
+]
+
+"""
+        if six.PY3:
+            # In python3 the output from subprocess.check_output are bytes, not str
+            myoutput = myoutput.encode()
+        myrunner = mock_open()
+        myrunner.check_output = lambda x: myoutput
+        with patch('conans.client.tools.win.subprocess', myrunner):
+            json = vswhere()
+            self.assertNotIn("descripton", json)
+
     def vcvars_echo_test(self):
         if platform.system() != "Windows":
             return