internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: GoogleCloudPlatform/PerfKitBenchmarker

Issue:
The cluster boot benchmark should the num_cpus function in parallel
The cluster boot benchmark has the following code:

>  for vm in vms:
>    metadata = {'machine_type': vm.machine_type, 'num_cpus': vm.num_cpus,
>               'machine_instance': vm_number}
>   value = vm.TimeToBoot()

This looks great until you realize vm.num_cpus is a method on the virtual machine which in turn calls RemoteCommand leading to an ssh.  When large number of VM's boot the result is a long set of serially run ssh's to each VM.  This could be done a lot faster by moving the code into a method and then using RunThreaded.



Discussion:

Resolving patch:
diff --git a/perfkitbenchmarker/benchmarks/cluster_boot_benchmark.py b/perfkitbenchmarker/benchmarks/cluster_boot_benchmark.py
--- a/perfkitbenchmarker/benchmarks/cluster_boot_benchmark.py
+++ b/perfkitbenchmarker/benchmarks/cluster_boot_benchmark.py
@@ -17,6 +17,7 @@
 import logging
 
 from perfkitbenchmarker import flags
+from perfkitbenchmarker import vm_util
 
 FLAGS = flags.FLAGS
 BENCHMARK_INFO = {'name': 'cluster boot',
@@ -34,6 +35,14 @@ def Prepare(unused_benchmark_spec):
   pass
 
 
+def _GetTimeToBoot(vm, vm_index, result_list):
+  metadata = {'machine_type': vm.machine_type, 'num_cpus': vm.num_cpus,
+              'machine_instance': vm_index}
+  value = vm.TimeToBoot()
+  assert value is not None
+  result_list.append(('Boot Time', value, 'seconds', metadata))
+
+
 def Run(benchmark_spec):
   """Measure the boot time for all VMs.
 
@@ -49,18 +58,12 @@ def Run(benchmark_spec):
   """
 
   samples = []
-  vm_number = 0
   logging.info('Boot Results:')
   vms = benchmark_spec.vms
-  for vm in vms:
-    metadata = {'machine_type': vm.machine_type, 'num_cpus': vm.num_cpus,
-                'machine_instance': vm_number}
-    value = vm.TimeToBoot()
-    assert value is not None
-    samples.append(('Boot Time', value, 'seconds', metadata))
-    vm_number += 1
+  params = [((vm, i, samples), {}) for i, vm in enumerate(vms)]
+  vm_util.RunThreaded(_GetTimeToBoot, params)
   logging.info(samples)
-  assert vm_number == benchmark_spec.num_vms
+  assert len(samples) == benchmark_spec.num_vms
   return samples
 
 


Test patch:
Repository: GoogleCloudPlatform/PerfKitBenchmarker

Issue:
BaseVirtualMachine.__str__() method returns non-string object in some cases. 
I intentionally tried to throw an exception before running a test when one of the conditions was not met (see `_CheckPrerequisites()` method from `kubernetes_virtual_machine.py`). This exception was obscured by another exception:

```
2015-10-16 11:31:52,199 4efd9dfb MainThread sysbench_oltp(1/1) ERROR    Error during benchmark sysbench_oltp
Traceback (most recent call last):
  File "/root/mateuszb/perfkit_ww42_5/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 284, in RunBenchmark
    DoPreparePhase(benchmark, benchmark_name, spec, detailed_timer)
  File "/root/mateuszb/perfkit_ww42_5/PerfKitBenchmarker/perfkitbenchmarker/pkb.py", line 192, in DoPreparePhase
    spec.Prepare()
  File "/root/mateuszb/perfkit_ww42_5/PerfKitBenchmarker/perfkitbenchmarker/benchmark_spec.py", line 289, in Prepare
    vm_util.RunThreaded(self.PrepareVm, self.vms)
  File "/root/mateuszb/perfkit_ww42_5/PerfKitBenchmarker/perfkitbenchmarker/vm_util.py", line 345, in RunThreaded
    return RunParallelThreads(target_arg_tuples, max_concurrent_threads)
  File "/root/mateuszb/perfkit_ww42_5/PerfKitBenchmarker/perfkitbenchmarker/vm_util.py", line 273, in RunParallelThreads
    _GetCallString(target_arg_tuples[call_id]), os.linesep, stacktrace))
  File "/root/mateuszb/perfkit_ww42_5/PerfKitBenchmarker/perfkitbenchmarker/vm_util.py", line 192, in _GetCallString
    arg_strings = [str(a) for a in args]
TypeError: __str__ returned non-string (type NoneType)
```

The problem is that `BaseVirtualMachine.__str__()` method returns None when ip_address is not set. This creates additional exception (copied above) which obscures the real exception which should be visible to user.



Discussion:

Resolving patch:
diff --git a/perfkitbenchmarker/virtual_machine.py b/perfkitbenchmarker/virtual_machine.py
--- a/perfkitbenchmarker/virtual_machine.py
+++ b/perfkitbenchmarker/virtual_machine.py
@@ -123,7 +123,9 @@ def __repr__(self):
         self.ip_address, self.internal_ip)
 
   def __str__(self):
-    return self.ip_address
+    if self.ip_address:
+      return self.ip_address
+    return super(BaseVirtualMachine, self).__str__()
 
   def CreateScratchDisk(self, disk_spec):
     """Create a VM's scratch disk.


Test patch:
Repository: GoogleCloudPlatform/PerfKitBenchmarker

Issue:
Project id not respected when deleting disk in Cloudstack
I forgot to pass project_id as an agrugment when deleting volume. 



Discussion:

Resolving patch:
diff --git a/perfkitbenchmarker/cloudstack/cloudstack_disk.py b/perfkitbenchmarker/cloudstack/cloudstack_disk.py
--- a/perfkitbenchmarker/cloudstack/cloudstack_disk.py
+++ b/perfkitbenchmarker/cloudstack/cloudstack_disk.py
@@ -50,7 +50,7 @@ def __init__(self, disk_spec, name, zone_id, project_id=None):
     self.disk_offering_id = self._GetBestOfferingId(self.disk_size)
     assert self.disk_offering_id, "Unable get disk offering of given size"
 
-    if self.disk_spec.disk_type:
+    if disk_spec.disk_type:
         logging.warn("Cloudstack does not support disk types")
 
 
@@ -73,7 +73,7 @@ def _Create(self):
 
   def _Delete(self):
     """Deletes the disk."""
-    vol = self.cs.get_volume(self.name)
+    vol = self.cs.get_volume(self.name, self.project_id)
     if vol:
         self.cs.delete_volume(self.volume_id)
 


Test patch:
Repository: GoogleCloudPlatform/PerfKitBenchmarker

Issue:
Help doesn't render with FlagValuesProxy.
Example:

```
[:~/git/PerfKitBenchmarker] [perfkit] release-0.23.0+* 1 ± python pkb.py --benchmarks redis_ycsb  --machine_type n1-standard-4 --json_output redis_ycsb.json
ERROR:root:Unknown command line flag 'json_output'
Usage: pkb.py ARGS
<perfkitbenchmarker.context.FlagValuesProxy object at 0x7f51910bc050>
```

@ehankland - do you have a minute to look at this? If not assign back to me.



Discussion:

Resolving patch:
diff --git a/perfkitbenchmarker/context.py b/perfkitbenchmarker/context.py
--- a/perfkitbenchmarker/context.py
+++ b/perfkitbenchmarker/context.py
@@ -63,23 +63,24 @@ def _thread_flag_values(self):
     else:
       return flags.FLAGS
 
-  def __setattr__(self, name, value):
-    self._thread_flag_values.__setattr__(name, value)
 
-  def __getattr__(self, name):
-    return self._thread_flag_values.__getattr__(name)
-
-  def __setitem__(self, key, value):
-    self._thread_flag_values.__setitem__(key, value)
-
-  def __getitem__(self, key):
-    return self._thread_flag_values.__getitem__(key)
-
-  def __call__(self, argv):
-    return self._thread_flag_values.__call__(argv)
-
-  def FlagDict(self):
-    return self._thread_flag_values.FlagDict()
+def _AddProxyMethod(f_name):
+  """Adds a method to FlagValuesProxy that forwards to _thread_flag_values."""
+  def f(self, *args, **kwargs):
+    return getattr(self._thread_flag_values, f_name)(*args, **kwargs)
+  f.__name__ = f_name
+  f.__doc__ = 'Proxied ' + f_name
+  setattr(FlagValuesProxy, f_name, f)
+
+
+# TODO: introduce a more generic proxy.
+for _f_name in ['FlagDict', 'Reset', 'SetDefault', 'RegisteredFlags',
+                'FlagValuesDict', '__contains__', '__iter__', '__call__',
+                '__setattr__', '__getattr__', '__setitem__', '__getitem__',
+                '__str__']:
+  _AddProxyMethod(_f_name)
+del _f_name
+del _AddProxyMethod
 
 
 BENCHMARK_FLAGS = FlagValuesProxy()


Test patch:
Repository: GoogleCloudPlatform/PerfKitBenchmarker

Issue:
silo benchmark fails behind proxy
From @mateusz-blaszkowski in #475: 

> silo - failed with Clone of 'git://github.com/kohler/masstree-beta.git' into submodule path 'masstree' failed. I run the test behind the proxy and this is the case. I would have changed the path to Git repository to https:// but it is hidden somewhere in 'dbtest' (look a the command which failed: cd /tmp/pkb/silo && MODE=perf DEBUG=0 CHECK_INVARIANTS=0 make -j80 dbtest). Oh, i found that the exact path is specified here: https://github.com/stephentu/silo/blob/cc11ca1ea949ef266ee12a9b1c310392519d9e3b/.gitmodules

We should switch it to `https://`.



Discussion:
Adding something like:
`git config --global url.https://github.com/.insteadOf git://github.com/`
in packages/build_tools.py ( or maybe directly in silo benchmark ) should fix the issue.

ping ... and update on this?  Should we assign it out?


Resolving patch:
diff --git a/perfkitbenchmarker/linux_packages/silo.py b/perfkitbenchmarker/linux_packages/silo.py
--- a/perfkitbenchmarker/linux_packages/silo.py
+++ b/perfkitbenchmarker/linux_packages/silo.py
@@ -33,6 +33,9 @@ def _Install(vm):
   vm.RemoteCommand('git clone {0} {1}'.format(GIT_REPO, SILO_DIR))
   vm.RemoteCommand('cd {0} && git checkout {1}'.format(SILO_DIR,
                                                        GIT_TAG))
+  # This is due to a failing clone command when executing behind a proxy.
+  # Replacing the protocol to https instead of git fixes the issue.
+  vm.RemoteCommand('git config --global url."https://".insteadOf git://')
   vm.RemoteCommand('cd {0} && MODE=perf DEBUG=0 CHECK_INVARIANTS=0 make\
           -j{1} dbtest'.format(SILO_DIR, nthreads))
 


Test patch:
Repository: GoogleCloudPlatform/PerfKitBenchmarker

Issue:
" sudo apt-get update" is called even when there is nothing to install
It looks like VM's will " sudo apt-get update" even when there is nothing to install.  This is most notable on  the cluster_boot benchmark.

It seems like this should be deferred until and actual install is requested.



Discussion:
Maybe I'll take this one on.  The reason I opened it is I've seen a number of instances where boot stall because apt-get is failing.  Then either the benchmark fails or finished with a skewed value.

A skewed value for boot time? Updating packages isn't included in the boot time reported, though it of course affects end to end time.

If it's failing it's worth fixing soon. I decreased the priority because:
- AFAIK cluster boot is the only affected benchmark.
- Some providers don't bill exactly by the minute - GCP bills a minimum of 10, AWS in 1 hour increments, so we're not increasing cost in all situations.

I'm happy to fix.


Resolving patch:
diff --git a/perfkitbenchmarker/linux_virtual_machine.py b/perfkitbenchmarker/linux_virtual_machine.py
--- a/perfkitbenchmarker/linux_virtual_machine.py
+++ b/perfkitbenchmarker/linux_virtual_machine.py
@@ -647,9 +647,13 @@ class DebianMixin(BaseLinuxMixin):
 
   OS_TYPE = 'debian'
 
-  def SetupPackageManager(self):
-    """Runs apt-get update so InstallPackages shouldn't need to."""
-    self.AptUpdate()
+  def __init__(self, *args, **kwargs):
+    super(DebianMixin, self).__init__(*args, **kwargs)
+
+    # Whether or not apt-get update has been called.
+    # We defer running apt-get update until the first request to install a
+    # package.
+    self._apt_updated = False
 
   @vm_util.Retry(max_retries=UPDATE_RETRIES)
   def AptUpdate(self):
@@ -701,6 +705,11 @@ def Install(self, package_name):
     """Installs a PerfKit package on the VM."""
     if not self.install_packages:
       return
+
+    if not self._apt_updated:
+      self.AptUpdate()
+      self._apt_updated = True
+
     if package_name not in self._installed_packages:
       package = linux_packages.PACKAGES[package_name]
       package.AptInstall(self)


Test patch:
Repository: GoogleCloudPlatform/PerfKitBenchmarker

Issue:
iperf benchmark has race condition in saving process id
It's possible for iperf to save the wrong server id if two copies of the benchmark are running on the same machine.  Instead of using `pgrep -n`, use `$!` to get the process id.



Discussion:
Another option is to run the server inside of the timeout command (http://linux.die.net/man/1/timeout)


Resolving patch:
diff --git a/perfkitbenchmarker/linux_benchmarks/iperf_benchmark.py b/perfkitbenchmarker/linux_benchmarks/iperf_benchmark.py
--- a/perfkitbenchmarker/linux_benchmarks/iperf_benchmark.py
+++ b/perfkitbenchmarker/linux_benchmarks/iperf_benchmark.py
@@ -74,9 +74,8 @@ def Prepare(benchmark_spec):
     vm.Install('iperf')
     if vm_util.ShouldRunOnExternalIpAddress():
       vm.AllowPort(IPERF_PORT)
-    vm.RemoteCommand('nohup iperf --server --port %s &> /dev/null &' %
-                     IPERF_PORT)
-    stdout, _ = vm.RemoteCommand('pgrep -n iperf')
+    stdout, _ = vm.RemoteCommand(('nohup iperf --server --port %s &> /dev/null'
+                                  '& echo $!') % IPERF_PORT)
     # TODO store this in a better place once we have a better place
     vm.iperf_server_pid = stdout.strip()
 
@@ -193,4 +192,4 @@ def Cleanup(benchmark_spec):
   """
   vms = benchmark_spec.vms
   for vm in vms:
-    vm.RemoteCommand('kill -9 ' + vm.iperf_server_pid)
+    vm.RemoteCommand('kill -9 ' + vm.iperf_server_pid, ignore_failure=True)


Test patch:
Repository: GoogleCloudPlatform/PerfKitBenchmarker

Issue:
Don't print full usage on usage error
If a user provides an invalid flag to `pkb.py`, it prints a usage error along with all the valid flags. This makes it difficult for the user to see what the actual error was, as he/she must scroll up through the usage instructions to find the error.

It would be better to print minimal usage instructions when the user passes invalid flags, so that the error is clearly visible. 


Discussion:
+1
Also, it would be nice to be able to selectively print the flags info for a particular sub-module (e.g. provider, benchmarks, pkb flags, packages, etc) instead of the wall of text.

Something like `pkb.py --help=perfkitbenchmarker.providers.gcp` which would only show the flags of all the submodules matching the string `perfkitbenchmarker.providers.gcp`, and possibly the string could be a regexp.

Looks like the either method `FLAGS.ModuleHelp(module)`[1] or use one of the internal dicts in gflags that maps module names to a list of flags might do the trick.

[1] https://github.com/google/python-gflags/blob/master/gflags/flagvalues.py#L914-L925
I have a POC for my suggestion #1268 PTAL
Thanks! I'll take a look early next week.

Resolving patch:
diff --git a/perfkitbenchmarker/pkb.py b/perfkitbenchmarker/pkb.py
--- a/perfkitbenchmarker/pkb.py
+++ b/perfkitbenchmarker/pkb.py
@@ -263,7 +263,9 @@ def _ParseFlags(argv=sys.argv):
   try:
     argv = FLAGS(argv)
   except flags.FlagsError as e:
-    logging.error('%s\nUsage: %s ARGS\n%s', e, sys.argv[0], FLAGS)
+    logging.error(e)
+    logging.info('For usage instructions, use --helpmatch={module_name}')
+    logging.info('For example, ./pkb.py --helpmatch=benchmarks.fio')
     sys.exit(1)
 
 


Test patch:
Repository: GoogleCloudPlatform/PerfKitBenchmarker

Issue:
Default GCP config for gpu_pcie_bandwidth is broken
When running the gpu_pcie_bandwidth benchmark under the default GCP config, the following error is raised: 
```
ERROR: (gcloud.compute.instances.create) Some requests did not succeed:
- Invalid value for field 'resource.disks[0].initializeParams.sourceImage': 'https://www.googleapis.com/compute/v1/projects/bionic-baton-343/global/images//ubuntu-os-cloud/ubuntu-1604-xenial-v20161115'. The URL is malformed.
```
Until a patch is released, this error can be avoided by running the benchmark and passing in the desired image and image project on the CLI, like so:

`./pkb.py --benchmarks=gpu_pcie_bandwidth --cloud=GCP --image=ubuntu-1610-yakkety-v20170307 --image_project=ubuntu-os-cloud`


Discussion:

Resolving patch:
diff --git a/perfkitbenchmarker/linux_benchmarks/gpu_pcie_bandwidth_benchmark.py b/perfkitbenchmarker/linux_benchmarks/gpu_pcie_bandwidth_benchmark.py
--- a/perfkitbenchmarker/linux_benchmarks/gpu_pcie_bandwidth_benchmark.py
+++ b/perfkitbenchmarker/linux_benchmarks/gpu_pcie_bandwidth_benchmark.py
@@ -45,7 +45,8 @@
     default:
       vm_spec:
         GCP:
-          image: /ubuntu-os-cloud/ubuntu-1604-xenial-v20161115
+          image: ubuntu-1604-xenial-v20161115
+          image_project: ubuntu-os-cloud
           machine_type: n1-standard-4-k80x1
           zone: us-east1-d
           boot_disk_size: 200


Test patch:
Repository: GoogleCloudPlatform/PerfKitBenchmarker

Issue:
PodIP address is not populated while running iperf on Kubernetes
Referencing #1990. Sorry for late reply. Even after using ```--ip_addresses=INTERNAL```, IP is not getting populated.
I tried all the options for ```ip_addresses``` -> BOTH, INTERNAL, EXTERNAL, REACHABLE. All give the same issue.
Command used: ```./pkb.py --cloud=Kubernetes --benchmarks=iperf --kubeconfig=/root/.kube/config --image=ptest:v0.2  --ip_addresses=INTERNAL```
Notice: ```root@None``` in below logs

```
Running: ssh -A -p 22 root@None -2 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PreferredAuthentications=publickey -o PasswordAuthentication=no -o ConnectTimeout=5 -o GSSAPIAuthentication=no -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -i /tmp/perfkitbenchmarker/runs/f86259b8/perfkitbenchmarker_keyfile mkdir -p /tmp/pkb```


```

            }
        ],
        "hostIP": "91.106.194.55",
        "phase": "Running",
        "podIP": "10.233.84.192",
        "qosClass": "BestEffort",
        "startTime": "2019-11-20T05:57:07Z"
    }
}
```


Discussion:

Resolving patch:
diff --git a/perfkitbenchmarker/providers/kubernetes/kubernetes_virtual_machine.py b/perfkitbenchmarker/providers/kubernetes/kubernetes_virtual_machine.py
--- a/perfkitbenchmarker/providers/kubernetes/kubernetes_virtual_machine.py
+++ b/perfkitbenchmarker/providers/kubernetes/kubernetes_virtual_machine.py
@@ -186,6 +186,7 @@ def _GetInternalIp(self):
       raise Exception('Internal POD IP address not found. Retrying.')
 
     self.internal_ip = pod_ip
+    self.ip_address = pod_ip
 
   def _ConfigureProxy(self):
     """


Test patch:
Repository: GoogleCloudPlatform/PerfKitBenchmarker

Issue:
GCP  object_storage_service   MissingObjectCredentialException: Boto file cannot be found in [None, '~/.boto', '/etc/boto.cfg'].
* How to reproduce
  set flag `  gce_preemptible_vms: true`

*  errors
>  File "/Users/martin/work/PerfKitBenchmarker/perfkitbenchmarker/object_storage_service.py", line 383, in FindBotoFile
    'Boto file cannot be found in %s.' % paths_to_check)
> perfkitbenchmarker.errors.Benchmarks.MissingObjectCredentialException: Boto file cannot be found in [None, '~/.boto', '/etc/boto.cfg'].


* code 

https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/object_storage_service.py#L368-L383





Discussion:

Resolving patch:
diff --git a/perfkitbenchmarker/object_storage_service.py b/perfkitbenchmarker/object_storage_service.py
--- a/perfkitbenchmarker/object_storage_service.py
+++ b/perfkitbenchmarker/object_storage_service.py
@@ -376,8 +376,9 @@ def FindBotoFile():
   for path in paths_to_check:
     if not path:
       continue
-    if pathlib.Path(path).exists():
-      return path
+    absolute_path = pathlib.Path(path).expanduser().absolute()
+    if absolute_path.exists():
+      return absolute_path.as_posix()
 
   raise errors.Benchmarks.MissingObjectCredentialException(
       'Boto file cannot be found in %s.' % paths_to_check)


Test patch:
Repository: PaddlePaddle/PaddleSpeech

Issue:
Fix some problems in the ctc beam search decoder
- [x] Make character's index in FST starting from one, otherwise wrong decoding results would be produced especially when space is the first character in the vocabulary;
- [x] Add version check in the setup script;
- [x] Remove unused code. 



Discussion:

Resolving patch:
diff --git a/decoders/swig/setup.py b/decoders/swig/setup.py
--- a/decoders/swig/setup.py
+++ b/decoders/swig/setup.py
@@ -113,7 +113,7 @@ def compile_test(header, library):
 
 setup(
     name='swig_decoders',
-    version='1.0',
+    version='1.1',
     description="""CTC decoders""",
     ext_modules=decoders_module,
     py_modules=['swig_decoders'], )


Test patch:
Repository: PaddlePaddle/PaddleSpeech

Issue:
Need adapt demo_server.py



Discussion:

Resolving patch:
diff --git a/deploy/demo_server.py b/deploy/demo_server.py
--- a/deploy/demo_server.py
+++ b/deploy/demo_server.py
@@ -147,7 +147,8 @@ def start_server():
         augmentation_config='{}',
         specgram_type=args.specgram_type,
         num_threads=1,
-        keep_transcription_text=True)
+        keep_transcription_text=True,
+        num_conv_layers=args.num_conv_layers)
     # prepare ASR model
     ds2_model = DeepSpeech2Model(
         vocab_size=data_generator.vocab_size,
@@ -163,8 +164,20 @@ def start_server():
     # prepare ASR inference handler
     def file_to_transcript(filename):
         feature = data_generator.process_utterance(filename, "")
+        ins = []
+        conv0_h = (feature[0].shape[0] - 1) // 2 + 1
+        conv0_w = (feature[0].shape[1] - 1) // 3 + 1
+        ins += [feature[0], feature[1],
+                [0], [conv0_w],
+                [1, 32, 1, conv0_h, conv0_w + 1, conv0_w]]
+        pre_h = conv0_h
+        for i in xrange(args.num_conv_layers - 1):
+            h = (pre_h - 1) // 2 + 1
+            pre_h = h
+            ins += [[1, 32, 1, h, conv0_w + 1, conv0_w]]
+
         result_transcript = ds2_model.infer_batch(
-            infer_data=[feature],
+            infer_data=[ins],
             decoding_method=args.decoding_method,
             beam_alpha=args.alpha,
             beam_beta=args.beta,
@@ -173,7 +186,8 @@ def file_to_transcript(filename):
             cutoff_top_n=args.cutoff_top_n,
             vocab_list=vocab_list,
             language_model_path=args.lang_model_path,
-            num_processes=1)
+            num_processes=1,
+            feeding_dict=data_generator.feeding)
         return result_transcript[0]
 
     # warming up with utterrances sampled from Librispeech


Test patch: