anon-iclr-submission/benchname-bug-localization huggingface dataset:
Repository: ericsson/codechecker
Issue: "unsupported operand type(s)" when using a skipfile

Using version 6.5.1
I am trying to run CodeChecker on the Clangd code base and filter-out the llvm and clang headers.

/foo/llvm/build$ CodeChecker parse -i SKIPFILE ./reports
[LOW] atomic_base.h:188:20: Value stored to '__b' during its initialization is never read [deadcode.DeadStores]
      memory_order __b = __m & __memory_order_mask;
                   ^

[LOW] atomic_base.h:199:20: Value stored to '__b' during its initialization is never read [deadcode.DeadStores]
      memory_order __b = __m & __memory_order_mask;
                   ^

Traceback (most recent call last):
  File "/foo/CodeChecker/cc_bin/CodeChecker.py", line 93, in main
    args.func(args)
  File "/foo/CodeChecker/lib/python2.7/libcodechecker/libhandlers/parse.py", line 156, in __handle
    main(args)
  File "/foo/CodeChecker/lib/python2.7/libcodechecker/libhandlers/parse.py", line 302, in main
    'print_steps' in args)
  File "/foo/CodeChecker/lib/python2.7/libcodechecker/libhandlers/parse.py", line 204, in parse
    report_stats = rh.write(files, reports, analyzed_source_file)
  File "/foo/CodeChecker/lib/python2.7/libcodechecker/analyze/plist_parser.py", line 379, in write
    LOG.debug(report + ' is skipped (in ' + f_path + ")")
TypeError: unsupported operand type(s) for +: 'Report' and 'str'
/foo/llvm/build$ cat SKIPFILE 
-/foo/llvm/include/*
-/foo/llvm/tools/clang/include/*

Changed files:
libcodechecker/analyze/plist_parser.py

diff --git a/libcodechecker/analyze/plist_parser.py b/libcodechecker/analyze/plist_parser.py
index b79fe1fa..4b54f220 100644
--- a/libcodechecker/analyze/plist_parser.py
+++ b/libcodechecker/analyze/plist_parser.py
@@ -376,7 +376,8 @@ class PlistToPlaintextFormatter(object):
             f_path = files[events[-1]['location']['file']]
             if self.skiplist_handler and \\
                     self.skiplist_handler.should_skip(f_path):
-                LOG.debug(report + ' is skipped (in ' + f_path + ")")
+                LOG.debug("Skipped report in '%s'", f_path)
+                LOG.debug(report)
                 continue
             hash_value = report.main['issue_hash_content_of_line_in_context']
             bug = {'hash_value': hash_value,
Repository: ericsson/codechecker
Issue: profile name conflict with checker or checker group name

If I try to list the `portability` checkers I get this error, we should think about the profile naming or handle this differently. The same applies for the new `security` profile.
```
CodeChecker checkers --profile portability
[ERROR 2019-04-10 14:58] - Profile name 'portability' conflicts with a checker(-group) name.
[ERROR 2019-04-10 14:58] - Please choose another profile name in '....config/config.json and rebuild.
```

Changed files:
analyzer/codechecker_analyzer/analyzers/config_handler.py

diff --git a/analyzer/codechecker_analyzer/analyzers/config_handler.py b/analyzer/codechecker_analyzer/analyzers/config_handler.py
index 3aa5f093..86cc112b 100644
--- a/analyzer/codechecker_analyzer/analyzers/config_handler.py
+++ b/analyzer/codechecker_analyzer/analyzers/config_handler.py
@@ -168,12 +168,8 @@ class AnalyzerConfigHandler(object):
                         sys.exit(1)
 
                     if profile_name in reserved_names:
-                        LOG.error("Profile name '%s' conflicts with a "
-                                  "checker(-group) name.", profile_name)
-                        LOG.error("Please choose another profile name in "
-                                  "'%s'/config/config.json and rebuild.",
-                                  package_root)
-                        sys.exit(1)
+                        LOG.warning("Profile name '%s' conflicts with a "
+                                    "checker(-group) name.", profile_name)
 
                     profile_checkers = (name for name, profile_list
                                         in checker_config.items()
Repository: ericsson/codechecker
Issue: Analyze accumulate zombie processes on Ctrl+C

If I start jobs using multiple cores, and ctrl-c the job, seems over time doing this a few times starts to accumulate zombie processes.

Changed files:
analyzer/codechecker_analyzer/analysis_manager.py

diff --git a/analyzer/codechecker_analyzer/analysis_manager.py b/analyzer/codechecker_analyzer/analysis_manager.py
index 41edb34d..7ad29087 100644
--- a/analyzer/codechecker_analyzer/analysis_manager.py
+++ b/analyzer/codechecker_analyzer/analysis_manager.py
@@ -729,11 +729,15 @@ def start_workers(actions_map, actions, analyzer_config_map,
     def signal_handler(signum, frame):
         try:
             pool.terminate()
+            pool.join()
             manager.shutdown()
+        except Exception as e:
+            LOG.error("Failed to clean up after the pool!:\\n")
+            LOG.error(e)
+            raise
         finally:
             sys.exit(128 + signum)
 
-    signal.signal(signal.SIGINT, signal_handler)
     actions, skipped_actions = skip_cpp(actions, skip_handlers)
     # Start checking parallel.
     checked_var = multiprocessing.Value('i', 1)
@@ -742,6 +746,7 @@ def start_workers(actions_map, actions, analyzer_config_map,
                                 initializer=init_worker,
                                 initargs=(checked_var,
                                           actions_num))
+    signal.signal(signal.SIGINT, signal_handler)
 
     # If the analysis has failed, we help debugging.
     failed_dir = os.path.join(output_path, "failed")
Repository: ericsson/codechecker
Issue: Unexpected error code for unrecognized arguments

If providing unrecognized arguments to CodeChecker (tried with check and analyze) the exit code is 2 ("At least one report emitted by an analyzer") instead of the expected 1 ("CodeChecker error"). Maybe this does also affect other CodeChecker commands and maybe also other types of errors?


CodeChecker analyzer version:
```
---------------------------------------------------------------
Kind                 | Version                                 
---------------------------------------------------------------
Base package version | 6.17.0                                  
Package build date   | 2021-08-23T13:08                        
Git commit ID (hash) | e3b63963ae0c0cc32983c20017cdac7e0f05956a
Git tag information  | 6.17                                    
---------------------------------------------------------------
```


Repro:
```
$ CodeChecker check -b "ninja" bogus
usage: CodeChecker [-h] {check} ...
CodeChecker: error: unrecognized arguments: bogus
$ echo $?
2
```

This was run on fairly up-to-date Arch Linux. 



Changed files:
analyzer/tests/functional/cmdline/test_cmdline.py, codechecker_common/cli.py

diff --git a/analyzer/tests/functional/cmdline/test_cmdline.py b/analyzer/tests/functional/cmdline/test_cmdline.py
index efa03bf8..eb8e6cbe 100644
--- a/analyzer/tests/functional/cmdline/test_cmdline.py
+++ b/analyzer/tests/functional/cmdline/test_cmdline.py
@@ -84,6 +84,12 @@ class TestCmdline(unittest.TestCase):
         parse_help = [env.codechecker_cmd(), 'parse', '--help']
         self.assertEqual(0, run_cmd(parse_help)[0])
 
+    def test_invalid_subcommand(self):
+        """ Call CodeChecker with and invalid subcommand. """
+
+        dummy_cmd = [env.codechecker_cmd(), "dummy"]
+        self.assertEqual(1, run_cmd(dummy_cmd)[0])
+
     def test_checkers(self):
         """ Listing available checkers. """
 
diff --git a/codechecker_common/cli.py b/codechecker_common/cli.py
index 0895f25e..94e74a42 100755
--- a/codechecker_common/cli.py
+++ b/codechecker_common/cli.py
@@ -19,6 +19,18 @@ import signal
 import sys
 
 
+class ArgumentParser(argparse.ArgumentParser):
+    def error(self, message):
+        """
+        By default argparse will exit with error code 2 in case of error, but
+        we are using this error code for different purposes. For this reason
+        we will override this function, so we will exit with error code 1 in
+        such cases too.
+        """
+        self.print_usage(sys.stderr)
+        self.exit(1, f"{self.prog}: error: {message}\\n")
+
+
 def add_subcommand(subparsers, sub_cmd, cmd_module_path, lib_dir_path):
     """
     Load the subcommand module and then add the subcommand to the available
@@ -106,7 +118,7 @@ def main():
     signal.signal(signal.SIGTERM, signal_handler)
 
     try:
-        parser = argparse.ArgumentParser(
+        parser = ArgumentParser(
             prog="CodeChecker",
             formatter_class=argparse.RawDescriptionHelpFormatter,
             description="""Run the CodeChecker sourcecode analyzer framework.
Repository: ericsson/codechecker
Issue: CodeChecker check stops with "ValueError: need more than 2 values to unpack"

```
[INFO] - clang-tidy analyzed Unified_cpp_gfx_src0.cpp successfully.
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 396, in _handle_results
    cache[job]._set(i, obj)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 599, in _set
    self._callback(self._value)
  File "/data/jenkins/codechecker_package/CodeChecker/cc_lib/python2.7/codechecker_lib/analysis_manager.py", line 35, in worker_result_handler
    for res, skipped, analyzer_type in results:
ValueError: need more than 2 values to unpack
```

And the analysis freezes 


Changed files:
codechecker_lib/analysis_manager.py

diff --git a/codechecker_lib/analysis_manager.py b/codechecker_lib/analysis_manager.py
index a1f5f671..44f13837 100644
--- a/codechecker_lib/analysis_manager.py
+++ b/codechecker_lib/analysis_manager.py
@@ -67,10 +67,10 @@ def check(check_data):
     args, action, context, analyzer_config_map, skp_handler, \\
         report_output_dir, use_db = check_data
 
+    skipped = False
     try:
         # if one analysis fails the check fails
         return_codes = 0
-        skipped = False
         for source in action.sources:
 
             # if there is no skiplist handler there was no skip list file
@@ -135,7 +135,7 @@ def check(check_data):
     except Exception as e:
         LOG.debug_analyzer(str(e))
         traceback.print_exc(file=sys.stdout)
-        return (1, action.analyzer_type)
+        return (1, skipped, action.analyzer_type)
 
 def start_workers(args, actions, context, analyzer_config_map, skp_handler):
     """
Repository: ericsson/codechecker
Issue: exception is thrown while parsing compilation json

[ERROR] [17:04] - Running check failed. list index out of range
Traceback (most recent call last):
  File "/afs/sunrise.ericsson.se/se/app/vbuild/SLED11-x86_64/codechecker/6.2.0/lib/python2.7/libcodechecker/libhandlers/check.py", line 454, in main
    analyze_module.main(analyze_args)
  File "/afs/sunrise.ericsson.se/se/app/vbuild/SLED11-x86_64/codechecker/6.2.0/lib/python2.7/libcodechecker/libhandlers/analyze.py", line 393, in main
    actions += log_parser.parse_log(log_file, parseLogOptions)
  File "/afs/sunrise.ericsson.se/se/app/vbuild/SLED11-x86_64/codechecker/6.2.0/lib/python2.7/libcodechecker/analyze/log_parser.py", line 294, in parse_log
    actions = parse_compile_commands_json(logfile, parseLogOptions)
  File "/afs/sunrise.ericsson.se/se/app/vbuild/SLED11-x86_64/codechecker/6.2.0/lib/python2.7/libcodechecker/analyze/log_parser.py", line 259, in parse_compile_commands_json
    get_compiler_target(parseLogOptions, results.compiler)
  File "/afs/sunrise.ericsson.se/se/app/vbuild/SLED11-x86_64/codechecker/6.2.0/lib/python2.7/libcodechecker/analyze/log_parser.py", line 169, in get_compiler_target
    return parse_compiler_target(err)
  File "/afs/sunrise.ericsson.se/se/app/vbuild/SLED11-x86_64/codechecker/6.2.0/lib/python2.7/libcodechecker/analyze/log_parser.py", line 92, in parse_compiler_target
    if line[0] == target_label:
IndexError: list index out of range


Changed files:
libcodechecker/analyze/log_parser.py

diff --git a/libcodechecker/analyze/log_parser.py b/libcodechecker/analyze/log_parser.py
index 06a395ca..ef769e99 100644
--- a/libcodechecker/analyze/log_parser.py
+++ b/libcodechecker/analyze/log_parser.py
@@ -89,7 +89,7 @@ def parse_compiler_target(lines):
 
     for line in lines.splitlines(True):
         line = line.strip().split()
-        if line[0] == target_label:
+        if len(line) > 1 and line[0] == target_label:
             target = line[1]
 
     return target
Repository: ericsson/codechecker
Issue: clang 5.0 unknown arguments error

Clang 5.0 throws a lot of unknown error messages when running the following command: 
`CodeChecker analyze test.json -o ./reports`

```bash
[ERROR] [2018-01-16 08:42] - Analyzing 'test1.c' with clangsa failed.
[ERROR] [2018-01-16 08:42] - clang-5.0: error: unknown argument: '-msahf'
clang-5.0: error: unknown argument: '-mno-abm'

[ERROR] [2018-01-16 08:42] - Analyzing 'test2.c' with clangsa failed.
[ERROR] [2018-01-16 08:42] - clang-5.0: error: unknown argument: '-msahf'
clang-5.0: error: unknown argument: '-mno-abm'

[ERROR] [2018-01-16 08:42] - Analyzing 'test3.c' with clangsa failed.
[ERROR] [2018-01-16 08:42] - clang-5.0: error: unknown argument: '-msahf'
clang-5.0: error: unknown argument: '-mno-abm'

[ERROR] [2018-01-16 08:42] - Analyzing 'test4.c' with clangsa failed.
[ERROR] [2018-01-16 08:42] - clang-5.0: error: unknown argument: '-fr'
clang-5.0: error: unknown argument: '-fs'
clang-5.0: error: unknown argument: '-ffp'
clang-5.0: error: unknown argument: '-fno-jump-table'
clang-5.0: error: unknown argument: '-finline-local-initialisers'
clang-5.0: error: unknown argument: '-fno-jump-table'

[ERROR] [2018-01-16 08:42] - Analyzing 'test5.c' with clangsa failed.
[ERROR] [2018-01-16 08:42] - clang-5.0: error: unknown argument: '-fr'
clang-5.0: error: unknown argument: '-fs'
clang-5.0: error: unknown argument: '-ffp'
clang-5.0: error: unknown argument: '-fno-jump-table'
clang-5.0: error: unknown argument: '-finline-local-initialisers'
clang-5.0: error: unknown argument: '-fno-jump-table'
```

Changed files:
libcodechecker/log/option_parser.py

diff --git a/libcodechecker/log/option_parser.py b/libcodechecker/log/option_parser.py
index d1452bfb..0c06bcb0 100644
--- a/libcodechecker/log/option_parser.py
+++ b/libcodechecker/log/option_parser.py
@@ -153,34 +153,41 @@ UNKNOWN_OPTIONS_MAP_REGEX = {
     '^-fcse-follow-jumps': 0,
     '^-fcse-skip-blocks': 0,
     '^-ffixed-r2': 0,
+    '^-ffp$': 0,
     '^-fgcse-lm': 0,
     '^-fhoist-adjacent-loads': 0,
     '^-findirect-inlining': 0,
     '^-finline-limit.*': 0,
+    '^-finline-local-initialisers': 0,
     '^-fipa-sra': 0,
     '^-fno-aggressive-loop-optimizations': 0,
     '^-fno-delete-null-pointer-checks': 0,
+    '^-fno-jump-table': 0,
     '^-fno-strength-reduce': 0,
     '^-fno-toplevel-reorder': 0,
     '^-fno-unit-at-a-time': 0,
     '^-fno-var-tracking-assignments': 0,
     '^-fpartial-inlining': 0,
     '^-fpeephole2': 0,
+    '^-fr$': 0,
     '^-fregmove': 0,
     '^-frename-registers': 0,
     '^-freorder-functions': 0,
     '^-frerun-cse-after-loop': 0,
+    '^-fs$': 0,
     '^-fsched-spec': 0,
     '^-fthread-jumps': 0,
     '^-ftree-pre': 0,
     '^-ftree-switch-conversion': 0,
     '^-ftree-tail-merge': 0,
+    '^-m(no-)?abm.*$': 0,
     '^-m(no-)?sdata.*$': 0,
     '^-m(no-)?spe.*': 0,
     '^-m(no-)?string$': 0,
     '^-maccumulate-outgoing-args': 0,
     '^-mfix-cortex-m3-ldrd$': 0,
     '^-mmultiple$': 0,
+    '^-msahf$': 0,
     '^-mthumb-interwork$': 0,
     '^-mupdate$': 0,
Repository: pennylaneai/pennylane
Issue: scipy import bug

#### Issue description
When calling `pennylane.ops.cv.Rotation._heisenberg_rep([phi])` if one does not explicitly import the block_diag function via `from scipy.linalg import block_diag` there is an error.

```pennylane/pennylane/ops/cv.py", line 109, in _rotation
    return sp.linalg.block_diag(1, temp) # pylint: disable=no-member
AttributeError: module 'scipy' has no attribute 'linalg'
````

This is not a bug in scipy as it is a design choice not to import top level components such as `linalg` as they are not sub-modules but directories. Read this [discussion](https://groups.google.com/a/continuum.io/forum/#!topic/anaconda/mMiv9QslB4E).

I encountered this when I wrote an independent test for the ops #99 

#### System information

* **Operating system:**
  MacOS 10.14.3 

* **PennyLane version:**
  master

* **Python version:**
Python 3.7.2 (default, Dec 29 2018, 00:00:04)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin


* **NumPy and SciPy versions:**
  NumPy 1.15.0
  Scipy 1.2.1

* **Installation method:**

  Installed using  `python setup.py develop`on latest master.


Changed files:
pennylane/ops/cv.py

diff --git a/pennylane/ops/cv.py b/pennylane/ops/cv.py
index a50cb090a..c0f54fadc 100644
--- a/pennylane/ops/cv.py
+++ b/pennylane/ops/cv.py
@@ -78,7 +78,7 @@ Code details
 ~~~~~~~~~~~~
 """
 import numpy as np
-import scipy as sp
+from scipy.linalg import block_diag
 
 from pennylane.operation import CVOperation
 
@@ -106,7 +106,7 @@ def _rotation(phi, bare=False):
     temp = np.array([[c, -s], [s, c]])
     if bare:
         return temp
-    return sp.linalg.block_diag(1, temp) # pylint: disable=no-member
+    return block_diag(1, temp) # pylint: disable=no-member
 
 
 class Rotation(CVOperation):
Repository: googlecloudplatform/perfkitbenchmarker
Issue: Help doesn't render with FlagValuesProxy.

Example:

```
[:~/git/PerfKitBenchmarker] [perfkit] release-0.23.0+* 1 ± python pkb.py --benchmarks redis_ycsb  --machine_type n1-standard-4 --json_output redis_ycsb.json
ERROR:root:Unknown command line flag 'json_output'
Usage: pkb.py ARGS
<perfkitbenchmarker.context.FlagValuesProxy object at 0x7f51910bc050>
```

@ehankland - do you have a minute to look at this? If not assign back to me.


Changed files:
perfkitbenchmarker/context.py

diff --git a/perfkitbenchmarker/context.py b/perfkitbenchmarker/context.py
index 22274e9b..c8a348b9 100644
--- a/perfkitbenchmarker/context.py
+++ b/perfkitbenchmarker/context.py
@@ -63,23 +63,24 @@ class FlagValuesProxy(object):
     else:
       return flags.FLAGS
 
-  def __setattr__(self, name, value):
-    self._thread_flag_values.__setattr__(name, value)
 
-  def __getattr__(self, name):
-    return self._thread_flag_values.__getattr__(name)
-
-  def __setitem__(self, key, value):
-    self._thread_flag_values.__setitem__(key, value)
-
-  def __getitem__(self, key):
-    return self._thread_flag_values.__getitem__(key)
-
-  def __call__(self, argv):
-    return self._thread_flag_values.__call__(argv)
-
-  def FlagDict(self):
-    return self._thread_flag_values.FlagDict()
+def _AddProxyMethod(f_name):
+  """Adds a method to FlagValuesProxy that forwards to _thread_flag_values."""
+  def f(self, *args, **kwargs):
+    return getattr(self._thread_flag_values, f_name)(*args, **kwargs)
+  f.__name__ = f_name
+  f.__doc__ = 'Proxied ' + f_name
+  setattr(FlagValuesProxy, f_name, f)
+
+
+# TODO: introduce a more generic proxy.
+for _f_name in ['FlagDict', 'Reset', 'SetDefault', 'RegisteredFlags',
+                'FlagValuesDict', '__contains__', '__iter__', '__call__',
+                '__setattr__', '__getattr__', '__setitem__', '__getitem__',
+                '__str__']:
+  _AddProxyMethod(_f_name)
+del _f_name
+del _AddProxyMethod
 
 
 BENCHMARK_FLAGS = FlagValuesProxy()
Repository: googlecloudplatform/perfkitbenchmarker
Issue: silo benchmark fails behind proxy

From @mateusz-blaszkowski in #475: 

> silo - failed with Clone of 'git://github.com/kohler/masstree-beta.git' into submodule path 'masstree' failed. I run the test behind the proxy and this is the case. I would have changed the path to Git repository to https:// but it is hidden somewhere in 'dbtest' (look a the command which failed: cd /tmp/pkb/silo && MODE=perf DEBUG=0 CHECK_INVARIANTS=0 make -j80 dbtest). Oh, i found that the exact path is specified here: https://github.com/stephentu/silo/blob/cc11ca1ea949ef266ee12a9b1c310392519d9e3b/.gitmodules

We should switch it to `https://`.


Changed files:
perfkitbenchmarker/linux_packages/silo.py

diff --git a/perfkitbenchmarker/linux_packages/silo.py b/perfkitbenchmarker/linux_packages/silo.py
index 2c14a0bc..fdb64fa6 100644
--- a/perfkitbenchmarker/linux_packages/silo.py
+++ b/perfkitbenchmarker/linux_packages/silo.py
@@ -33,6 +33,9 @@ def _Install(vm):
   vm.RemoteCommand('git clone {0} {1}'.format(GIT_REPO, SILO_DIR))
   vm.RemoteCommand('cd {0} && git checkout {1}'.format(SILO_DIR,
                                                        GIT_TAG))
+  # This is due to a failing clone command when executing behind a proxy.
+  # Replacing the protocol to https instead of git fixes the issue.
+  vm.RemoteCommand('git config --global url."https://".insteadOf git://')
   vm.RemoteCommand('cd {0} && MODE=perf DEBUG=0 CHECK_INVARIANTS=0 make\\
           -j{1} dbtest'.format(SILO_DIR, nthreads))
Repository: googlecloudplatform/perfkitbenchmarker
Issue: Object_storage failing with "STDOUT: STDERR: Warning: Permanently added '108.170.222.218' (ECDSA) to the list of known hosts. Usage: pip COMMAND [OPTIONS]  pip: error: No command by the name pip show   (maybe you meant "pip install show")"

./pkb.py --image=debian-7 --zones=us-central1-c --machine_type=n1-standard-8 --benchmarks=object_storage_service --storage=GCP --object_storage_scenario=api_data

...
STDOUT: STDERR: Warning: Permanently added '108.170.222.218' (ECDSA) to the list of known hosts. Usage: pip COMMAND [OPTIONS]  pip: error: No command by the name pip show   (maybe you meant "pip install show")"

...

perfkit@pkb-d8f9db39-0:~$ pip --version
pip 1.1 from /usr/lib/python2.7/dist-packages (python 2.7)

To fix the issue I upgraded pip...
$ sudo apt-get remove python-pip
$ wget https://bootstrap.pypa.io/get-pip.py
$ sudo python get-pip.py

It looks like the root issues is Debian-7 images are pulling a really old version of pip.  I suggest we update the package install to wget the latest like above.


Changed files:
perfkitbenchmarker/linux_packages/pip.py

diff --git a/perfkitbenchmarker/linux_packages/pip.py b/perfkitbenchmarker/linux_packages/pip.py
index fd824cb1..dfca23a6 100644
--- a/perfkitbenchmarker/linux_packages/pip.py
+++ b/perfkitbenchmarker/linux_packages/pip.py
@@ -25,6 +25,7 @@ from perfkitbenchmarker import vm_util
 def _Install(vm):
   """Install pip on the VM."""
   vm.InstallPackages('python-pip')
+  vm.RemoteCommand('sudo pip install -U pip')  # Make pip upgrade pip
   vm.RemoteCommand('mkdir -p {0} && pip freeze > {0}/requirements.txt'.format(
       vm_util.VM_TMP_DIR))
Repository: elastic/rally
Issue: Download documents.bz2 to file to .tmp and rename

I had a truncated .bz2 file and hit scary exceptions running esrally, I think because I ctrl+c'd once while the download was happening and then when I ran again esrally assumed the file was complete.

We should just download to temp file and rename in the end to prevent this?


Changed files:
rally/track/track.py

diff --git a/rally/track/track.py b/rally/track/track.py
index 4018aae2..b02506cc 100644
--- a/rally/track/track.py
+++ b/rally/track/track.py
@@ -266,15 +266,31 @@ class Marshal:
       raise RuntimeError("Cannot download mappings. No protocol handler for [%s] available." % url)
 
   def _do_download_via_http(self, url, data_set_path):
-    with urllib.request.urlopen(url) as response, open(data_set_path, 'wb') as out_file:
-      shutil.copyfileobj(response, out_file)
+    tmp_data_set_path = data_set_path + '.tmp'
+    try:
+      with urllib.request.urlopen(url) as response, open(tmp_data_set_path, 'wb') as out_file:
+        shutil.copyfileobj(response, out_file)
+    except:
+      logger.info('Removing temp file %s' % tmp_data_set_path)
+      os.remove(tmp_data_set_path)
+      raise
+    else:
+      os.rename(tmp_data_set_path, data_set_path)
 
   def _do_download_via_s3(self, track, url, data_set_path):
-    s3cmd = "s3cmd -v get %s %s" % (url, data_set_path)
-    success = rally.utils.process.run_subprocess(s3cmd)
-    # Exit code for s3cmd does not seem to be reliable so we also check the file size although this is rather fragile...
-    if not success or os.path.getsize(data_set_path) != track.compressed_size_in_bytes:
-      # cleanup probably corrupt data file...
-      if os.path.isfile(data_set_path):
-        os.remove(data_set_path)
-      raise RuntimeError("Could not get benchmark data from S3: '%s'. Is s3cmd installed and set up properly?" % s3cmd)
+    tmp_data_set_path = data_set_path + '.tmp'
+    s3cmd = "s3cmd -v get %s %s" % (url, tmp_data_set_path)
+    try:
+      success = rally.utils.process.run_subprocess(s3cmd)
+      # Exit code for s3cmd does not seem to be reliable so we also check the file size although this is rather fragile...
+      if not success or os.path.getsize(tmp_data_set_path) != track.compressed_size_in_bytes:
+        # cleanup probably corrupt data file...
+        if os.path.isfile(tmp_data_set_path):
+          os.remove(tmp_data_set_path)
+        raise RuntimeError("Could not get benchmark data from S3: '%s'. Is s3cmd installed and set up properly?" % s3cmd)
+    except:
+      logger.info('Removing temp file %s' % tmp_data_set_path)
+      os.remove(tmp_data_set_path)
+      raise
+    else:
+      os.rename(tmp_data_set_path, data_set_path)