internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: NVIDIA/NVFlare

Issue:
[BUG] Single quotes in organization name are not escaped properly 
An Org Admin registered with an organization name "Ben's Site". Unfortunately the single quote / apostrophe is not properly escaped and when I startup the client with /var/tmp/cloud/startup/start.sh I get this 

```
PYTHONPATH is /local/custom:
/var/tmp/cloud/startup/sub_start.sh: line 82: unexpected EOF while looking for matching `"'
/var/tmp/cloud/startup/sub_start.sh: line 88: syntax error: unexpected end of file
```

The org name is not escaped and changing "Ben's" to "Ben\\'s" did the trick

```
grep "Ben's" sub_start.sh
((python3 -u -m nvflare.private.fed.app.client.client_train -m $DIR/.. -s fed_client.json --set secure_train=true uid=AWS-T4 org=Ben's site config_folder=config 2>&1 & echo $! >&3 ) 3>$DIR/../pid.fl )
```



Discussion:
Thanks for raising this issue, @IsaacYangSLA can you check if we can support this scenario? thanks
Thanks for reporting the issue.  The issue might be related to how to allow special characters in different places.  As a quick fix, we can escape that in shell scripts.

Resolving patch:
diff --git a/nvflare/dashboard/application/blob.py b/nvflare/dashboard/application/blob.py
--- a/nvflare/dashboard/application/blob.py
+++ b/nvflare/dashboard/application/blob.py
@@ -180,6 +180,11 @@ def gen_client(key, id):
         "type": "client",
         "cln_uid": f"uid={entity.name}",
     }
+    for k in ["client_name", "org_name", "cln_uid"]:
+        value = replacement_dict[k]
+        escaped_value = value.replace("'", "\\'")
+        replacement_dict[k] = escaped_value
+
     if project.ha_mode:
         overseer_agent = {"path": "nvflare.ha.overseer_agent.HttpOverseerAgent"}
         overseer_agent["args"] = {


Test patch:
Repository: Parsl/parsl

Issue:
Do not raise `NotImplemented`
In a few places we raise `NotImplemented`, which itself raises a `SyntaxError`; this should be `NotImplementedError`.


Discussion:

Resolving patch:
diff --git a/parsl/app/app.py b/parsl/app/app.py
--- a/parsl/app/app.py
+++ b/parsl/app/app.py
@@ -60,7 +60,7 @@ def __init__(self, func, executor, walltime=60, sites='all', cache=False, exec_t
     def __call__(self, *args, **kwargs):
         ''' The __call__ function must be implemented in the subclasses
         '''
-        raise NotImplemented
+        raise NotImplementedError
 
 
 def app_wrapper(func):
diff --git a/parsl/executors/threads.py b/parsl/executors/threads.py
--- a/parsl/executors/threads.py
+++ b/parsl/executors/threads.py
@@ -66,7 +66,7 @@ def scale_out(self, workers=1):
              NotImplemented exception
         '''
 
-        raise NotImplemented
+        raise NotImplementedError
 
     def scale_in(self, workers=1):
         ''' Scale in the number of active workers by 1
@@ -76,7 +76,7 @@ def scale_in(self, workers=1):
              NotImplemented exception
         '''
 
-        raise NotImplemented
+        raise NotImplementedError
 
     def shutdown(self, block=False):
         ''' Shutdown the ThreadPool


Test patch:
Repository: Parsl/parsl

Issue:
Do not import `parsl` before requirements are setup
```
[annawoodard@midway001 parsl]$ python setup.py install
Traceback (most recent call last):
  File "setup.py", line 2, in <module>
    from parsl.version import VERSION
  File "/home/annawoodard/parsl/parsl/__init__.py", line 35, in <module>
    from parsl.executors.ipp import IPyParallelExecutor
  File "/home/annawoodard/parsl/parsl/executors/ipp.py", line 4, in <module>
    from ipyparallel import Client
ModuleNotFoundError: No module named 'ipyparallel'
```

Setuptools is supposed to take care of dependencies for us, but importing parsl in `setup.py` breaks that (because we require the dependencies by importing the parsl version from `version.py` before they can be installed). We should avoid this.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,5 +1,7 @@
 from setuptools import setup, find_packages
-from parsl.version import VERSION
+
+with open('parsl/version.py') as f:
+    exec(f.read())
 
 with open('requirements.txt') as f:
     install_requires = f.readlines()


Test patch:
Repository: Parsl/parsl

Issue:
Add missing `scaling_enabled` method for `TurbineExecutor`
Currently `tests/test_swift.py` fails:
```
======================================================================
ERROR: test_swift.test_except
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/annawoodard/.conda/envs/parsl_py36/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/annawoodard/parsl/parsl/validation/test_swift.py", line 38, in test_except
    tex = TurbineExecutor()
TypeError: Can't instantiate abstract class TurbineExecutor with abstract methods scaling_enabled
-------------------- >> begin captured stdout << ---------------------
Start

--------------------- >> end captured stdout << ----------------------
```



Discussion:

Resolving patch:
diff --git a/parsl/executors/swift_t.py b/parsl/executors/swift_t.py
--- a/parsl/executors/swift_t.py
+++ b/parsl/executors/swift_t.py
@@ -302,6 +302,7 @@ def __init__(self, swift_attribs=None, config=None, **kwargs):
         self.worker.start()
         logger.debug("Created worker : %s", self.worker)
         self.tasks = {}
+        self._scaling_enabled = False
 
     def submit(self, func, *args, **kwargs):
         ''' Submits work to the the outgoing_q, an external process listens on this
@@ -337,6 +338,10 @@ def submit(self, func, *args, **kwargs):
         # Return the future
         return self.tasks[task_id]
 
+    @property
+    def scaling_enabled(self):
+        return self._scaling_enabled
+
     def scale_out(self, workers=1):
         ''' Scales out the number of active workers by 1
         This method is notImplemented for threads and will raise the error if called.


Test patch:
Repository: Parsl/parsl

Issue:
PathLike behavior for File
With the recent changes to the DataFuture behavior and the File based model that we are moving to in general, we want to have File behave as closely to old behavior as possible so that there's least amount of code breakage. This is one such case:

Earlier we could do this since the result of a DataFuture was a string
> f = open(DataFuture1.result(), 'r') 

Now the result of a DataFuture is a File, which can't simply be passed to open:
> f = open(File, 'r') <-- Fails 

This can be fixed by making File follow the [os.PathLike](https://docs.python.org/3/library/os.html#os.PathLike) interface 


Discussion:

Resolving patch:
diff --git a/parsl/data_provider/files.py b/parsl/data_provider/files.py
--- a/parsl/data_provider/files.py
+++ b/parsl/data_provider/files.py
@@ -15,7 +15,7 @@
 logger = logging.getLogger(__name__)
 
 
-class File(object):
+class File(str):
     """The Parsl File Class.
 
     This is planned to be a very simple class that simply
@@ -59,6 +59,9 @@ def __str__(self):
     def __repr__(self):
         return self.__str__()
 
+    def __fspath__(self):
+        return self.filepath
+
     @property
     def filepath(self):
         """Returns the resolved filepath on the side where it is called from.


Test patch:
diff --git a/parsl/tests/test_data/test_file.py b/parsl/tests/test_data/test_file.py
--- a/parsl/tests/test_data/test_file.py
+++ b/parsl/tests/test_data/test_file.py
@@ -18,6 +18,16 @@ def test_files():
             test['path'], x.path)
 
 
+def test_open():
+    with open('test-open.txt', 'w') as tfile:
+        tfile.write('Hello')
+
+    pfile = File('test-open.txt')
+
+    with open(pfile, 'r') as opfile:
+        assert (opfile.readlines()[0] == 'Hello')
+
+
 if __name__ == '__main__':
 
     test_files()
Repository: Parsl/parsl

Issue:
Allow `DataFuture` to be initialized with a `str` file object
[Here](https://github.com/Parsl/parsl/blob/master/parsl/app/futures.py#L77) we check if `file_obj` is `str`. Now that `File` is subclassed from `str`, this will always evaluate as `True`.


Discussion:

Resolving patch:
diff --git a/parsl/app/futures.py b/parsl/app/futures.py
--- a/parsl/app/futures.py
+++ b/parsl/app/futures.py
@@ -74,7 +74,7 @@ def __init__(self, fut, file_obj, parent=None, tid=None):
         """
         super().__init__()
         self._tid = tid
-        if isinstance(file_obj, str):
+        if isinstance(file_obj, str) and not isinstance(file_obj, File):
             self.file_obj = File(file_obj)
         else:
             self.file_obj = file_obj


Test patch:
Repository: Parsl/parsl

Issue:
User tracking to report parsl version
User tracking as described in #34 tracks a range of script attributes and anonymous user information.
It would help to track the version of Parsl and python. This would help us understand:
1. Rate of upgrade to newer releases
2. Python version in use, and support requirements
3. Support requirements for older parsl releases.



Discussion:

Resolving patch:
diff --git a/parsl/dataflow/usage_tracking/usage.py b/parsl/dataflow/usage_tracking/usage.py
--- a/parsl/dataflow/usage_tracking/usage.py
+++ b/parsl/dataflow/usage_tracking/usage.py
@@ -6,8 +6,10 @@
 import json
 import logging
 import socket
+import sys
 
 from parsl.dataflow.states import States
+from parsl.version import VERSION as PARSL_VERSION
 
 logger = logging.getLogger(__name__)
 
@@ -48,6 +50,10 @@ def __init__(self, dfk, ip='52.3.111.203', port=50077,
         self.dfk = dfk
         self.config = self.dfk.config
         self.uuid = str(uuid.uuid4())
+        self.parsl_version = PARSL_VERSION
+        self.python_version = "{}.{}.{}".format(sys.version_info.major,
+                                                sys.version_info.minor,
+                                                sys.version_info.micro)
         self.test_mode, self.tracking_enabled = self.check_tracking_enabled()
         logger.debug("Tracking status: {}".format(self.tracking_enabled))
         logger.debug("Testing mode   : {}".format(self.test_mode))
@@ -93,6 +99,8 @@ def construct_start_message(self):
                    'uname': hashed_username,
                    'hname': hashed_hostname,
                    'test': self.test_mode,
+                   'parsl_v': self.parsl_version,
+                   'python_v': self.python_version,
                    'start': time.time()}
 
         return json.dumps(message)


Test patch:
Repository: Parsl/parsl

Issue:
Describe code state more precisely in the logging
We currently log the Parsl version. For debugging, it would be convenient to log the most recent commit hash and whether or not there are any additional changes.


Discussion:

Resolving patch:
diff --git a/parsl/dataflow/dflow.py b/parsl/dataflow/dflow.py
--- a/parsl/dataflow/dflow.py
+++ b/parsl/dataflow/dflow.py
@@ -20,6 +20,7 @@
 from parsl.dataflow.config_defaults import update_config
 from parsl.data_provider.data_manager import DataManager
 from parsl.execution_provider.provider_factory import ExecProviderFactory as EPF
+from parsl.utils import get_version
 
 # from parsl.dataflow.start_controller import Controller
 # Exceptions
@@ -72,7 +73,7 @@ def __init__(self, config=None, executors=None, lazyErrors=True, appCache=True,
         parsl.set_file_logger("{}/parsl.log".format(self.rundir),
                               level=logging.DEBUG)
 
-        logger.info("Parsl version: {}".format(parsl.__version__))
+        logger.info("Parsl version: {}".format(get_version()))
         logger.info("Libsubmit version: {}".format(libsubmit.__version__))
 
         # Update config with defaults
diff --git a/parsl/utils.py b/parsl/utils.py
new file mode 100644
--- /dev/null
+++ b/parsl/utils.py
@@ -0,0 +1,26 @@
+import logging
+import os
+import shlex
+import subprocess
+
+import parsl
+from parsl.version import VERSION
+
+logger = logging.getLogger(__name__)
+
+
+def get_version():
+    version = parsl.__version__
+    work_tree = os.path.dirname(os.path.dirname(__file__))
+    git_dir = os.path.join(work_tree, '.git')
+    env = {'GIT_WORK_TREE': work_tree, 'GIT_DIR': git_dir}
+    try:
+        cmd = shlex.split('git rev-parse --short HEAD')
+        head = subprocess.check_output(cmd, env=env).strip().decode('utf-8')
+        diff = subprocess.check_output(shlex.split('git diff HEAD'), env=env)
+        status = 'dirty' if diff else 'clean'
+        version = '{v}-{head}-{status}'.format(v=VERSION, head=head, status=status)
+    except Exception as e:
+        logger.exception("Unable to determine code state")
+
+    return version


Test patch:
Repository: Parsl/parsl

Issue:
`globals` should be an optional config field
Running over a config without `globals`, I see:
```
    def make_rundir(config=None, path=None):
        """When a path has not been specified, make the run directory.

        Creates a rundir with the following hierarchy:
            ./runinfo <- Home of all run directories
              |----000
              |----001 <- Directories for each run
              | ....
              |----NNN

        Kwargs:
            - path (str): String path to a specific run dir
                   Default : None.
        """
        try:
            if not path:
                path = None
>           elif config["globals"].get('runDir', None):
E           KeyError: 'globals'

../dataflow/rundirs.py:25: KeyError
```


Discussion:

Resolving patch:
diff --git a/parsl/dataflow/rundirs.py b/parsl/dataflow/rundirs.py
--- a/parsl/dataflow/rundirs.py
+++ b/parsl/dataflow/rundirs.py
@@ -22,7 +22,7 @@ def make_rundir(config=None, path=None):
     try:
         if not path:
             path = None
-        elif config["globals"].get('runDir', None):
+        elif config.get("globals", {}).get('runDir'):
             path = config["globals"]['runDir']
 
         if not path:


Test patch:
Repository: Parsl/parsl

Issue:
Automatically convert notebooks to rst
@yadudoc had a good idea, we should automatically convert the tutorial notebooks that appear in the docs to rst. There is a Sphinx [extension](https://nbsphinx.readthedocs.io/en/0.3.1/) for this.


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -19,6 +19,7 @@
 #
 import os
 import sys
+import requests
 sys.path.insert(0, os.path.abspath('../'))
 sys.path.insert(0, os.path.abspath('../parsl/libsubmit'))
 import parsl
@@ -32,12 +33,18 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+    'nbsphinx',
     'sphinx.ext.autodoc',
     'sphinx.ext.autosummary',
     'sphinx.ext.intersphinx',
     'sphinx.ext.linkcode'
 ]
 
+url = 'https://raw.githubusercontent.com/Parsl/parsl-tutorial/master/parsl-introduction.ipynb'
+r = requests.get(url)
+with open(os.path.join(os.path.dirname(__file__), 'parsl-introduction.ipynb'), 'wb') as f:
+    f.write(r.content)
+
 def linkcode_resolve(domain, info):
     if domain != 'py':
         return None


Test patch:
Repository: Parsl/parsl

Issue:
fatal: Not a git repository: '/homes/vvikraman/anaconda3/lib/python3.6/site-packages/.git
Hi 
When I try to run parsl I am getting the following issue:

fatal: Not a git repository: '/homes/vvikraman/anaconda3/lib/python3.6/site-packages/.git

Is it a real issue?

I am using python3 and jupyter but run parsl in a terminal. 
Issue in parsl.log
I tried to run a simple script given in the parsl documentation 

```
import parsl
from parsl import *
import time

workers = ThreadPoolExecutor(max_workers=4)
dfk = DataFlowKernel(executors=[workers])
print(1)
@App('python', dfk)
def hello ():
    import time
    time.sleep(5)
    return 'Hello World!'
print(2)
app_future = hello()
print ('Done: %s' % app_future.done())
print ('Result: %s' % app_future.result())
print ('Done: %s' % app_future.done())
```
However, in the parsl.log shows this issue

2018-06-07 21:45:37 parsl.utils:24 [ERROR]  Unable to determine code state
Traceback (most recent call last):
  File "/homes/vvikraman/anaconda3/lib/python3.6/site-packages/parsl/utils.py", line 19, in get_version
    head = subprocess.check_output(cmd, env=env).strip().decode('utf-8')
  File "/homes/vvikraman/anaconda3/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/homes/vvikraman/anaconda3/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'rev-parse', '--short', 'HEAD']' returned non-zero exit status 128.



Discussion:
Parsl tries to check and report the git version in case the user is trying to use parsl from source. This is non-fatal and should not affect workflow execution in any way.
We'll have this cleaned up in the next release. 


Resolving patch:
diff --git a/parsl/utils.py b/parsl/utils.py
--- a/parsl/utils.py
+++ b/parsl/utils.py
@@ -25,7 +25,7 @@ def get_version():
         status = 'dirty' if diff else 'clean'
         version = '{v}-{head}-{status}'.format(v=VERSION, head=head, status=status)
     except Exception as e:
-        logger.exception("Unable to determine code state")
+        pass
 
     return version
 


Test patch:
Repository: Parsl/parsl

Issue:
Duplicate log lines when using IPP
The following code fragment emits log lines to STDERR with the default format style even though no logging is requested. I suspect that `ipyparallel` is doing something dirty.

    from parsl import DataFlowKernel
    from parsl.configs.local import localIPP as config

    dfk = DataFlowKernel(config=config)
    dfk.cleanup()

The above code with the minor change of using threads will not emit the log lines.


    from parsl import DataFlowKernel
    from parsl.configs.local import localThreads as config

    dfk = DataFlowKernel(config=config)
    dfk.cleanup()


Please help test by running this with the latest parsl code. 


Discussion:
It looks like whenever the IPP client is used, a `StreamHandler` is added to the `RootLogger`. This leads to duplicate log lines. I'm adding some code to identify the StreamHandler and mute it by setting the logLevel to ERROR.

Resolving patch:
diff --git a/parsl/dataflow/strategy.py b/parsl/dataflow/strategy.py
--- a/parsl/dataflow/strategy.py
+++ b/parsl/dataflow/strategy.py
@@ -116,6 +116,8 @@ def __init__(self, dfk):
         self.strategies = {None: self._strategy_noop, 'simple': self._strategy_simple}
 
         self.strategize = self.strategies[self.config.strategy]
+        self.logger_flag = False
+        self.prior_loghandlers = set(logging.getLogger().handlers)
 
         logger.debug("Scaling strategy: {0}".format(self.config.strategy))
 
@@ -129,6 +131,20 @@ def _strategy_noop(self, tasks, *args, kind=None, **kwargs):
             - kind (Not used)
         """
 
+    def unset_logging(self):
+        """ Mute newly added handlers to the root level, right after calling executor.status
+        """
+        if self.logger_flag is True:
+            return
+
+        root_logger = logging.getLogger()
+
+        for hndlr in root_logger.handlers:
+            if hndlr not in self.prior_loghandlers:
+                hndlr.setLevel(logging.ERROR)
+
+        self.logger_flag = True
+
     def _strategy_simple(self, tasks, *args, kind=None, **kwargs):
         """Peek at the DFK and the executors specified.
 
@@ -143,9 +159,6 @@ def _strategy_simple(self, tasks, *args, kind=None, **kwargs):
         KWargs:
             - kind (Not used)
         """
-        # Add logic to check executors
-        # for task in tasks :
-        #    if self.dfk.tasks[task]:
 
         for label, executor in self.dfk.executors.items():
             if not executor.scaling_enabled:
@@ -155,6 +168,7 @@ def _strategy_simple(self, tasks, *args, kind=None, **kwargs):
             active_tasks = executor.executor.outstanding
 
             status = executor.status()
+            self.unset_logging()
 
             # FIXME we need to handle case where provider does not define these
             # FIXME probably more of this logic should be moved to the provider


Test patch:
Repository: Parsl/parsl

Issue:
Add OS to usage stats collection



Discussion:

Resolving patch:
diff --git a/parsl/dataflow/usage_tracking/usage.py b/parsl/dataflow/usage_tracking/usage.py
--- a/parsl/dataflow/usage_tracking/usage.py
+++ b/parsl/dataflow/usage_tracking/usage.py
@@ -7,6 +7,7 @@
 import logging
 import socket
 import sys
+import platform
 import multiprocessing as mp
 
 from parsl.dataflow.states import States
@@ -168,6 +169,8 @@ def construct_start_message(self):
                    'test': self.test_mode,
                    'parsl_v': self.parsl_version,
                    'python_v': self.python_version,
+                   'os': platform.system(),
+                   'os_v': platform.release(),
                    'start': time.time()}
 
         return json.dumps(message)


Test patch:
Repository: Parsl/parsl

Issue:
parsl installed from pip tries to determine its version using git
Every time I run parsl I get:

```
kacperk@dxl1: /dpool/kacperk/arxiv $ python scraper_parsl.py
fatal: Not a git repository: '/home/kacperk/.local/lib/python3.6/site-packages/.git'
```

and in logs:

```
2018-07-15 12:54:06 parsl.utils:24 [ERROR]  Unable to determine code state
Traceback (most recent call last):
  File "/home/kacperk/.local/lib/python3.6/site-packages/parsl/utils.py", line 19, in get_version
    head = subprocess.check_output(cmd, env=env).strip().decode('utf-8')
  File "/home/kacperk/miniconda3/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/home/kacperk/miniconda3/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'rev-parse', '--short', 'HEAD']' returned non-zero exit status 128.
```


Discussion:

Resolving patch:
diff --git a/parsl/utils.py b/parsl/utils.py
--- a/parsl/utils.py
+++ b/parsl/utils.py
@@ -17,15 +17,16 @@ def get_version():
     version = parsl.__version__
     work_tree = os.path.dirname(os.path.dirname(__file__))
     git_dir = os.path.join(work_tree, '.git')
-    env = {'GIT_WORK_TREE': work_tree, 'GIT_DIR': git_dir}
-    try:
-        cmd = shlex.split('git rev-parse --short HEAD')
-        head = subprocess.check_output(cmd, env=env).strip().decode('utf-8')
-        diff = subprocess.check_output(shlex.split('git diff HEAD'), env=env)
-        status = 'dirty' if diff else 'clean'
-        version = '{v}-{head}-{status}'.format(v=VERSION, head=head, status=status)
-    except Exception as e:
-        pass
+    if os.path.exists(git_dir):
+        env = {'GIT_WORK_TREE': work_tree, 'GIT_DIR': git_dir}
+        try:
+            cmd = shlex.split('git rev-parse --short HEAD')
+            head = subprocess.check_output(cmd, env=env).strip().decode('utf-8')
+            diff = subprocess.check_output(shlex.split('git diff HEAD'), env=env)
+            status = 'dirty' if diff else 'clean'
+            version = '{v}-{head}-{status}'.format(v=VERSION, head=head, status=status)
+        except Exception as e:
+            pass
 
     return version
 


Test patch: