Repository: easybuilders/easybuild-framework

Issue:
Quoting issues when using '--job' with GC3Pie
EasyBuild environment variables seem to be converted to command-line options when using `--job` with GC3Pie.  However, quoting seems to be broken in some places.  Below is a list of the issues I ran into (but there may be more, of course):
- Using
  
  ```
  EASYBUILD_TEST_REPORT_ENV_FILTER="^SSH|GPG|USER|HOSTNAME|UID|KDE|XDG|.*(COOKIE|SESSION).*"
  ```
  
  leads to
  
  ```
  --test-report-env-filter=^SSH|GPG|USER|HOSTNAME|UID|KDE|XDG|.*(COOKIE|SESSION).*
  ```
  
  Here, the parens cause havoc:
  
  ```
  /bin/sh: 1: Syntax error: word unexpected (expecting ")")
  ```
- Using
  
  ```
  EASYBUILD_SUFFIX_MODULES_PATH=
  ```
  
  to get rid of the `all` default setting leads to
  
  ```
  --suffix-modules-path=\\\"\\\"
  ```
  
  which in the end creates a directory called `""` in the modules path.  This lets the sanity check fail (`loading fake module failed`).



Discussion:
Initial tests indicate that both issues disappear if `shell_quote` (see https://github.com/hpcugent/vsc-base/blob/master/lib/vsc/utils/missing.py#L282) gets fixed. Below is a poor man's approach of addressing it, but I guess there are much better ways to express this in Python ;-)  I'm also not sure whether I caught all shell special characters that need escaping (or whether I escaped too much).

```
def shell_quote(x):
    """Add quotes so it can be passed to shell"""
    result = str(x)
    result = result.replace("\\", "\\\\")
    result = result.replace("\"", "\\\"")
    result = result.replace("\'", "\\\'")
    result = result.replace(" ", "\\ ")
    result = result.replace("(", "\\(")
    result = result.replace(")", "\\)")
    result = result.replace("[", "\\[")
    result = result.replace("]", "\\]")
    result = result.replace("|", "\\|")
    result = result.replace("<", "\\<")
    result = result.replace(">", "\\>")
    result = result.replace("&", "\\&")
    result = result.replace("*", "\\*")
    result = result.replace("?", "\\?")
    return result
```

This is apparently a known issue, see https://github.com/hpcugent/vsc-base/issues/152.

@geimer: an attempt to fix this was made in https://github.com/hpcugent/vsc-base/issues/151, but it didn't make it in... 

The gist of it was to define `shell_quote` as follows, using a negative lookbehind assertion (see also https://docs.python.org/2/library/re.html):

``` python
def shell_quote(token):
    return "'%s'" % re.sub(r"(?<!\\)'", r"\'", str(token))
```

Can you check whether that fixes the problem you're having too? If so, I can make a new attempt at fixing it.

@boegel: Yep, this unreadable piece of code fixes my issues as well ;-)
I guess the "unset vs. empty string" thing has to be handled in `generate_cmd_line` (https://github.com/hpcugent/vsc-base/blob/master/lib/vsc/utils/generaloption.py#L1550)?  Or maybe it is already and I just missed it...

@geimer so both errors occur when using `--job`? it has not so much to do with the shell_quote (i'm not sure how idempotent shell_quote should be, need to think about it), but there is a bug in the generation of the commandline in generaloption i think, but there is something a lot better that can be done to handle this imho.

@stdweird Yes, both happen with `--job` and GC3Pie.  I don't have access to a PBS system, so I can't verify that it's job backend affected as well.  There may be other ways to address this, sure, but `shell_quote` doesn't do what the name implies, i.e., that characters are properly quoted to be used in a shell command.

+1 on the latter: `shell_quote` doesn't do what it's supposed to do (and worse, it uses an undocumented function from subprocess...)

Using `--try-toolchain` is also very broken (probably same reason). I get this (using `--job`):

```
--try-toolchain="['\"[\'foss\'', '']" PLUMED-2.2.0-foss-2015b.eb '2015b']
```

@wpoely86: The issues may be related, but there is apparently a difference.  I'm locally using the proposed fix from https://github.com/hpcugent/vsc-base/pull/151 to get GC3Pie to work with my setup, but today also ran into problems with `--try-toolchain` and strange quoting.  Need to investigate...

Any chance that https://github.com/hpcugent/vsc-base/pull/151 gets revisited?  This damn issue is hitting me every time I upgrade to a new EasyBuild version, which is really annoying...

@wpoely86: The `--try-toolchain` thing is a different problem.  But once it is resolved, the quoting may become an issue again ;-)

But first things first:  It seems that the `generate_cmd_line` method in `generaloption.py` of vsc-base does not handle lists correctly (whatever that means).  Or it is triggered with improper arguments when composing the command line for the job script.  The branch taken for `--try-toolchain` is actually https://github.com/hpcugent/vsc-base/blob/master/lib/vsc/utils/generaloption.py#L1695, where `opt_value` is some sort of Python list object.  This seems to get expanded to a string like `["'[\\'GCC\\'", " \\'4.9.3-2.25\\']'"]`, which is then subject to quoting.  This can only end up in a mess...

@wpoely86: Please reopen https://github.com/hpcugent/easybuild-framework/issues/1475.  Using `--try-toolchain` together with `--job` is a different issue.  It can be fixed by filtering out the `--try-toolchain` option like it is done for `--robot`, see https://github.com/hpcugent/easybuild-framework/blob/master/easybuild/tools/parallelbuild.py#L130. This is valid as the temporary easyconfig is already generated by the `eb` "driver" instance, i.e., the `eb` instance running within the batch job doesn't have to do the same thing again. One can probably also filter out some more `--try-*` options...

I'd like to give this issue another ping...

What about monkey patching `shell_quote` in EB until there is a proper fix in vsc-base?  Of course we have to make sure that this doesn't break anything else.


Resolving patch:
diff --git a/easybuild/tools/options.py b/easybuild/tools/options.py
--- a/easybuild/tools/options.py
+++ b/easybuild/tools/options.py
@@ -40,7 +40,10 @@
 import shutil
 import sys
 import tempfile
+import vsc.utils.generaloption
 from distutils.version import LooseVersion
+from vsc.utils import fancylogger
+from vsc.utils.generaloption import GeneralOption
 
 import easybuild.tools.environment as env
 from easybuild.framework.easyblock import MODULE_ONLY_STEPS, SOURCE_STEP, EasyBlock
@@ -78,8 +81,6 @@
 from easybuild.tools.toolchain.utilities import search_toolchain
 from easybuild.tools.repository.repository import avail_repositories
 from easybuild.tools.version import this_is_easybuild
-from vsc.utils import fancylogger
-from vsc.utils.generaloption import GeneralOption
 
 try:
     from humanfriendly.terminal import terminal_supports_colors
@@ -92,6 +93,21 @@ def terminal_supports_colors(stream):
             # in case of errors do not bother and just return the safe default
             return False
 
+# monkey patch shell_quote in vsc.utils.generaloption, used by generate_cmd_line,
+# to fix known issue, cfr. https://github.com/hpcugent/vsc-base/issues/152;
+# inspired by https://github.com/hpcugent/vsc-base/pull/151
+# this fixes https://github.com/hpcugent/easybuild-framework/issues/1438
+# proper fix would be to implement a serialiser for command line options
+def eb_shell_quote(token):
+    """
+    Wrap provided token in single quotes (to escape space and characters with special meaning in a shell),
+    so it can be used in a shell command. This results in token that is not expanded/interpolated by the shell.
+    """
+    # escape any non-escaped single quotes, and wrap entire token in single quotes
+    return "'%s'" % re.sub(r"(?<!\\)'", r"\'", str(token))
+
+vsc.utils.generaloption.shell_quote = eb_shell_quote
+
 
 CONFIG_ENV_VAR_PREFIX = 'EASYBUILD'
 
@@ -228,7 +244,7 @@ def basic_options(self):
                       "and skipping check for OS dependencies", None, 'store_true', False, 'f'),
             'job': ("Submit the build as a job", None, 'store_true', False),
             'logtostdout': ("Redirect main log to stdout", None, 'store_true', False, 'l'),
-            'only-blocks': ("Only build listed blocks", None, 'extend', None, 'b', {'metavar': 'BLOCKS'}),
+            'only-blocks': ("Only build listed blocks", 'strlist', 'extend', None, 'b', {'metavar': 'BLOCKS'}),
             'rebuild': ("Rebuild software, even if module already exists (don't skip OS dependencies checks)",
                         None, 'store_true', False),
             'robot': ("Enable dependency resolution, using easyconfigs in specified paths",
@@ -260,13 +276,13 @@ def software_options(self):
                        "for example: versionprefix=foo or patches=one.patch,two.patch)"),
                       None, 'append', None, {'metavar': 'VAR=VALUE[,VALUE]'}),
             'software': ("Search and build software with given name and version",
-                         None, 'extend', None, {'metavar': 'NAME,VERSION'}),
+                         'strlist', 'extend', None, {'metavar': 'NAME,VERSION'}),
             'software-name': ("Search and build software with given name",
                               None, 'store', None, {'metavar': 'NAME'}),
             'software-version': ("Search and build software with given version",
                                  None, 'store', None, {'metavar': 'VERSION'}),
             'toolchain': ("Search and build with given toolchain (name and version)",
-                          None, 'extend', None, {'metavar': 'NAME,VERSION'}),
+                          'strlist', 'extend', None, {'metavar': 'NAME,VERSION'}),
             'toolchain-name': ("Search and build with given toolchain name",
                                None, 'store', None, {'metavar': 'NAME'}),
             'toolchain-version': ("Search and build with given toolchain version",
@@ -391,7 +407,7 @@ def config_options(self):
                               sorted(avail_module_generators().keys())),
             'moduleclasses': (("Extend supported module classes "
                                "(For more info on the default classes, use --show-default-moduleclasses)"),
-                              None, 'extend', [x[0] for x in DEFAULT_MODULECLASSES]),
+                              'strlist', 'extend', [x[0] for x in DEFAULT_MODULECLASSES]),
             'modules-footer': ("Path to file containing footer to be added to all generated module files",
                                None, 'store_or_None', None, {'metavar': "PATH"}),
             'modules-header': ("Path to file containing header to be added to all generated module files",


Test patch:
diff --git a/test/framework/options.py b/test/framework/options.py
--- a/test/framework/options.py
+++ b/test/framework/options.py
@@ -307,13 +307,13 @@ def check_args(job_args, passed_args=None):
 
         # options passed are reordered, so order here matters to make tests pass
         check_args(['--debug'])
-        # --try-* options are not passed down
         check_args(['--debug', '--stop=configure', '--try-software-name=foo'],
-                   passed_args=['--debug', '--stop=configure'])
-        check_args(['--debug', '--robot-paths=/tmp/foo:/tmp/bar'])
+                   passed_args=['--debug', "--stop='configure'"])
+        check_args(['--debug', '--robot-paths=/tmp/foo:/tmp/bar'],
+                   passed_args=['--debug', "--robot-paths='/tmp/foo:/tmp/bar'"])
         # --robot has preference over --robot-paths, --robot is not passed down
         check_args(['--debug', '--robot-paths=/tmp/foo', '--robot=/tmp/bar'],
-                   passed_args=['--debug', '--robot-paths=/tmp/bar:/tmp/foo'])
+                   passed_args=['--debug', "--robot-paths='/tmp/bar:/tmp/foo'"])
 
     # 'zzz' prefix in the test name is intentional to make this test run last,
     # since it fiddles with the logging infrastructure which may break things
@@ -1612,7 +1612,7 @@ def toy(extra_args=None):
         self.assertFalse(res, "No match for %s in %s" % (test_var_secret_regex.pattern, test_report_txt))
         self.assertTrue(test_var_public_regex.search(test_report_txt))
         # make sure that used filter is reported correctly in test report
-        filter_arg_regex = re.compile(filter_arg.replace('*', '\*'))
+        filter_arg_regex = re.compile(r"--test-report-env-filter='.\*_SECRET_ENV_VAR_FOR_EASYBUILD'")
         tup = (filter_arg_regex.pattern, test_report_txt)
         self.assertTrue(filter_arg_regex.search(test_report_txt), "%s in %s" % tup)
 
@@ -1813,12 +1813,31 @@ def generate_cmd_line(ebopts):
         self.assertEqual(generate_cmd_line(ebopts), ['--force'])
 
         ebopts = EasyBuildOptions(go_args=['--search=bar', '--search', 'foobar'], envvar_prefix='EASYBUILD')
-        self.assertEqual(generate_cmd_line(ebopts), ['--search=foobar'])
+        self.assertEqual(generate_cmd_line(ebopts), ["--search='foobar'"])
 
         os.environ['EASYBUILD_DEBUG'] = '1'
         ebopts = EasyBuildOptions(go_args=['--force'], envvar_prefix='EASYBUILD')
         self.assertEqual(generate_cmd_line(ebopts), ['--debug', '--force'])
 
+        args = [
+            # install path with a single quote in it, iieeeuuuwww
+            "--installpath=/this/is/a/weird'prefix",
+            '--test-report-env-filter=(COOKIE|SESSION)',
+            '--suffix-modules-path=',
+            '--try-toolchain=foss,2015b',
+            '--logfile-format=easybuild,eb-%(name)s.log',
+        ]
+        expected = [
+            '--debug',
+            "--installpath='/this/is/a/weird\\'prefix'",
+            "--logfile-format='easybuild,eb-%(name)s.log'",
+            "--suffix-modules-path=''",
+            "--test-report-env-filter='(COOKIE|SESSION)'",
+            "--try-toolchain='foss,2015b'",
+        ]
+        ebopts = EasyBuildOptions(go_args=args, envvar_prefix='EASYBUILD')
+        self.assertEqual(generate_cmd_line(ebopts), expected)
+
     def test_include_easyblocks(self):
         """Test --include-easyblocks."""
         orig_local_sys_path = sys.path[:]