Repository: horovod/horovod

Issue:
A bug "/bin/sh: -c: line 0: syntax error near unexpected token `('"
when I run "horovodrun -np 4 -H localhost:4 python3 train_mfh_coatt_glove.py "
then a bug "/bin/sh: -c: line 0: syntax error near unexpected token `('" shows




Discussion:
@Tangjiandd, thanks for being one of the early adopters of `horovodrun`!

Could you re-run your command with `--verbose` flag and paste the output? It will help us debug the problem.
> @Tangjiandd, thanks for being one of the early adopters of horovodrun!
> Could you re-run your command with --verbose flag and paste the output? It will help us debug the problem.

thanks for your replying
I have tried to use --verbose flag, but it just ouputs the bug "bash: syntax error near unexpected token `('".
whether it may be caused by the version of bash.
Interesting.  I'm wondering where is the `(` coming from.

Did you add `--verbose` flag before the program name, like this?
```
root@6e7730eb4a05:/examples# horovodrun -np 4 -H localhost:4 --verbose python pytorch_mnist.py
Filtering local host names.
mpirun --allow-run-as-root --tag-output -np 4 -H localhost:4 -bind-to none -map-by slot -mca pml ob1 -mca btl ^openib   -x NCCL_DEBUG=INFO  -x NVIDIA_DRIVER_CAPABILITIES -x MXNET_URL -x PYTHON_VERSION -x NVIDIA_VISIBLE_DEVICES -x PATH -x CUDA_VERSION -x SHLVL -x CUDNN_VERSION -x LS_COLORS -x _HOROVOD_SECRET_KEY -x _ -x CUDA_PKG_VERSION -x HOME -x NVIDIA_REQUIRE_CUDA -x HOSTNAME -x TERM -x LIBRARY_PATH -x TENSORFLOW_VERSION -x NCCL_VERSION -x PWD -x PYTORCH_VERSION -x LD_LIBRARY_PATH python pytorch_mnist.py
...
```

What does the output of `horovodrun --version -np 1` look like?
> Interesting.  I'm wondering where is the ( coming from.
> Did you add --verbose flag before the program name, like this?
> root@6e7730eb4a05:/examples# horovodrun -np 4 -H localhost:4 --verbose python pytorch_mnist.py
> Filtering local host names.
> mpirun --allow-run-as-root --tag-output -np 4 -H localhost:4 -bind-to none -map-by slot -mca pml ob1 -mca btl ^openib   -x NCCL_DEBUG=INFO  -x NVIDIA_DRIVER_CAPABILITIES -x MXNET_URL -x PYTHON_VERSION -x NVIDIA_VISIBLE_DEVICES -x PATH -x CUDA_VERSION -x SHLVL -x CUDNN_VERSION -x LS_COLORS -x _HOROVOD_SECRET_KEY -x _ -x CUDA_PKG_VERSION -x HOME -x NVIDIA_REQUIRE_CUDA -x HOSTNAME -x TERM -x LIBRARY_PATH -x TENSORFLOW_VERSION -x NCCL_VERSION -x PWD -x PYTORCH_VERSION -x LD_LIBRARY_PATH python pytorch_mnist.py
> ...
> 
> What does the output of horovodrun --version -np 1 look like?

I am also confused by the origin of  '('.
when I run "horovodrun -np 4 -H localhost:4 --verbose python3 train_mfh_coatt_glove.py", the output is '/bin/sh: -c:行0: 未预期的符号 `(' 附近有语法错误'.
The output of "horovodrun --version -np 1" is "0.16.1"
Is `Filtering local host names.` present in the output with the `--verbose` flag?
> Is Filtering local host names. present in the output with the --verbose flag?

Yes, it ouputs the 'filtering local host names' followed by ''/bin/sh: -c:行0: 未预期的符号 `(' 附近有语法错误'.'
Is there anything else between `Filtering local host names.` and `/bin/sh: -c:行0: 未预期的符号 `(' 附近有语法错误'`?
> Is there anything else between Filtering local host names. and /bin/sh: -c:行0: 未预期的符号 (' 附近有语法错误'`?

"mpirun --allow-run-as-root --tag-output -np 4 -H localhost:4 -bind-to none -map-by slot -mca pml ob1 -mca btl ^openib   -x NCCL_DEBUG=INFO  -x XDG_SESSION_ID -x HOSTNAME -x SHELL -x TERM -x HISTSIZE -x SSH_CLIENT -x QTDIR -x QTINC -x SSH_TTY -x QT_GRAPHICSSYSTEM_CHECKED -x USER -x CUDA_HOME -x LD_LIBRARY_PATH -x LS_COLORS -x TERMCAP -x PATH -x MAIL -x STY -x PWD -x JAVA_HOME -x LANG -x MODULEPATH -x KDEDIRS -x LOADEDMODULES -x HISTCONTROL -x LD_LIBRARY_PATH_PRESERVED -x HOME -x SHLVL -x JDK_HOME -x LOGNAME -x QTLIB -x WINDOW -x SSH_CONNECTION -x MODULESHOME -x LESSOPEN -x XDG_RUNTIME_DIR -x QT_PLUGIN_PATH -x BASH_FUNC_module() -x OLDPWD -x _ -x _HOROVOD_SECRET_KEY python3 train_mfh_coatt_glove.py
"
Thanks!  This seems to be related to https://bugzilla.redhat.com/show_bug.cgi?id=1147453.  What's the output of your `bash --version`?
> Thanks!  This seems to be related to https://bugzilla.redhat.com/show_bug.cgi?id=1147453.  What's the output of your bash --version?

Thank you very much!
bash --version outputs "GNU bash， 版本 4.2.46(1)-release (x86_64-redhat-linux-gnu)"
Added PR #949 with a potential fix.  Could you try it by reinstalling horovod with `[flags] pip install --no-cache-dir git+https://github.com/horovod/horovod@ignore_bash_func_module`?
> Added PR #949 with a potential fix.  Could you try it by reinstalling horovod with [flags] pip install --no-cache-dir git+https://github.com/horovod/horovod@ignore_bash_func_module?

thanks for your detailed reply.
I am trying to reinstall horovod by the script above.
> Added PR #949 with a potential fix.  Could you try it by reinstalling horovod with [flags] pip install --no-cache-dir git+https://github.com/horovod/horovod@ignore_bash_func_module?

the version of "ignore_bash_func_moudle" has satisfied. but the error still exits.
![image](https://user-images.githubusercontent.com/18570393/54898232-671f1880-4f06-11e9-82f9-81a242dc3e13.png)
@Tangjiandd, can you `pip uninstall -y horovod` first?
> @Tangjiandd, can you pip uninstall -y horovod first?

Thank you very much!
The problem has been solved by pip uninstall horovod and then pip install --no-cache-dir git+https://github.com/horovod/horovod@ignore_bash_func_module
The issue still has not been resolved for me from the master branch:

`syntax error near unexpected token `('`

```
(powerai.1.6) [benchmarks]$ horovodrun -np 4     python scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py         --model resnet101         --batch_size 64         --variable_update horovod
/bin/sh: -c: line 0: syntax error near unexpected token `('
/bin/sh: -c: line 0: `mpirun --allow-run-as-root --tag-output -np 4  -bind-to none -map-by slot -mca pml ob1 -mca btl ^openib   -x NCCL_DEBUG=INFO  -x AS -x LDFLAGS -x AR -x CUDA_PATH -x MANPATH -x XDG_SESSION_ID -x GCC_NM -x _ModuleTable003_ -x HOSTNAME -x HRTC_PATH -x HOST -x TERM -x SHELL -x NM -x HISTSIZE -x CPPFLAGS -x LMOD_DEFAULT_MODULEPATH -x LMOD_SYSTEM_DEFAULT_MODULES -x MODULEPATH_ROOT -x SSH_CLIENT -x LIBRARY_PATH -x CONDA_SHLVL -x LMOD_PACKAGE_PATH -x CONDA_PROMPT_MODIFIER -x TF_INCLUDE_DIR -x SIZE -x LMOD_PKG -x LSF_SERVERDIR -x OLDPWD -x LMOD_VERSION -x SSH_TTY -x CAFFE_BIN -x LSF_LIBDIR -x OPAL_PREFIX -x http_proxy -x USER -x LMOD_sys -x LD_LIBRARY_PATH -x LS_COLORS -x CONDA_BACKUP_HOST -x CONDA_EXE -x HRTC_NB_CORES -x CPATH -x _ModuleTable004_ -x POWERAI_caffe_CAFFE_BIN -x CXXFLAGS -x STRINGS -x LMOD_IGNORE_CACHE -x CPP -x LMOD_PREPEND_BLOCK -x CXXFILT -x HRTC_USE_SHARE -x BASELINE -x _ModuleTable001_ -x MODULE_VERSION -x MAIL -x PATH -x CPUTYPE -x DEBUG_CXXFLAGS -x LD -x CONDA_PREFIX -x LMOD_SETTARG_CMD -x PWD -x STRIP -x _LMFILES_ -x POWERAI_pytorch_OMP_NUM_THREADS -x ELFEDIT -x GCC_RANLIB -x LMOD_PAGER -x LANG -x MODULEPATH -x LOADEDMODULES -x _ModuleTable_Sz_ -x LMOD_SYSTEM_NAME -x _ModuleTable005_ -x LMOD_CMD -x https_proxy -x LMOD_AVAIL_STYLE -x HRTC_LD_LIBRARY_PATH -x LSF_BINDIR -x CXX -x HISTCONTROL -x OBJCOPY -x HRTC_DEFAULT_JOB_SCHEDULER -x SHLVL -x HOME -x TF_LIBRARY_DIR -x DEBUG_CPPFLAGS -x CFLAGS -x _ModuleTable002_ -x BINARY_TYPE_HPC -x _CONDA_PYTHON_SYSCONFIGDATA_NAME -x GCC -x BASH_ENV -x MPI_ROOT -x ADDR2LINE -x CONDA_PYTHON_EXE -x LESS -x HRTC_SYSTEM_NAME -x LMOD_arch -x LOGNAME -x VIADEV_ENABLE_AFFINITY -x SSH_CONNECTION -x MODULESHOME -x OMP_NUM_THREADS -x CONDA_DEFAULT_ENV -x LESSOPEN -x DEBUG_CFLAGS -x RANLIB -x __Init_Default_Modules -x HRTC_APPLI_DIR -x LMOD_FULL_SETTARG_SUPPORT -x CC -x XDG_RUNTIME_DIR -x READELF -x LMOD_DIR -x LSF_ENVDIR -x GCC_AR -x OBJDUMP -x LOCATION -x GPROF -x LMOD_COLORIZE -x POWERAI_caffe_OMP_NUM_THREADS -x GXX -x LMOD_INSTALLATION_DIR -x BASH_FUNC_ml() -x _ -x _HOROVOD_SECRET_KEY python scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py --model resnet101 --batch_size 64 --variable_update horovod'
```
bash version:
```
bash --version
GNU bash, version 4.2.46(2)-release (powerpc64le-redhat-linux-gnu)
```

Resolving patch:
diff --git a/horovod/run/common/util/env_constants.py b/horovod/run/common/util/env.py
similarity index 77%
rename from horovod/run/common/util/env_constants.py
rename to horovod/run/common/util/env.py
--- a/horovod/run/common/util/env_constants.py
+++ b/horovod/run/common/util/env.py
@@ -13,4 +13,12 @@
 # limitations under the License.
 # ==============================================================================
 
-IGNORE_LIST = {'BASH_FUNC_module()'}
+import re
+
+
+# List of regular expressions to ignore environment variables by.
+IGNORE_REGEXES = {'BASH_FUNC_.*\(\)'}
+
+
+def is_exportable(v):
+    return not any(re.match(r, v) for r in IGNORE_REGEXES)
diff --git a/horovod/run/run.py b/horovod/run/run.py
--- a/horovod/run/run.py
+++ b/horovod/run/run.py
@@ -17,6 +17,7 @@
 import argparse
 import hashlib
 import os
+import re
 import sys
 import traceback
 import six
@@ -26,7 +27,7 @@
     from pipes import quote
 import horovod
 
-from horovod.run.common.util import codec, env_constants, safe_shell_exec, timeout, secret
+from horovod.run.common.util import codec, env as env_util, safe_shell_exec, timeout, secret
 from horovod.run.driver import driver_service
 from horovod.run.task import task_service
 from horovod.run.util import cache, threads, network
@@ -469,7 +470,7 @@ def run():
                     nccl_socket_intf_arg=nccl_socket_intf_arg,
                     ssh_port_arg=ssh_port_arg,
                     env=' '.join('-x %s' % key for key in env.keys()
-                                 if key not in env_constants.IGNORE_LIST),
+                                 if env_util.is_exportable(key)),
                     command=' '.join(quote(par) for par in args.command))
     )
 
diff --git a/horovod/spark/__init__.py b/horovod/spark/__init__.py
--- a/horovod/spark/__init__.py
+++ b/horovod/spark/__init__.py
@@ -15,12 +15,13 @@
 
 import os
 import pyspark
+import re
 from six.moves import queue
 import sys
 import threading
 
 from horovod.spark.task import task_service
-from horovod.run.common.util import codec, env_constants, safe_shell_exec, timeout, \
+from horovod.run.common.util import codec, env as env_util, safe_shell_exec, timeout, \
     host_hash, secret
 from horovod.spark.driver import driver_service, job_id
 
@@ -172,8 +173,7 @@ def run(fn, args=(), kwargs={}, num_proc=None, start_timeout=None, env=None, std
                     hosts=','.join('%s:%d' % (host_hash, len(driver.task_host_hash_indices()[host_hash]))
                                    for host_hash in host_hashes),
                     common_intfs=','.join(common_intfs),
-                    env=' '.join('-x %s' % key for key in env.keys()
-                                 if key not in env_constants.IGNORE_LIST),
+                    env=' '.join('-x %s' % key for key in env.keys() if env_util.is_exportable(key)),
                     python=sys.executable,
                     encoded_driver_addresses=codec.dumps_base64(driver.addresses())))
         if verbose >= 2:


Test patch: