internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: google/fuzzbench

Issue:
Dynamic report generation
Though we already provide things like ipython notebooks to generate reports, I think we should lower the bar so that people can modify the report from the experiment from the report page.
I can think of two ways to do this:
1. Allow dynamic report generation using some kind of webapp.
2. Having a link to an ipython notebook that uses the data for the report on each report page.

Option 1 is a bit of a challenge because we generate reports server side and they take a while to do.
Option 2 is not ideal because I assume there is startup time for ipython notebooks and because it puts a greater burden on users.


Discussion:
In particular we've gotten requests in #472 to exclude benchmarks from aggregate results (ie: filtering). I think this could be possible to support as aggregate ranking is not as complex as generating the rest of the graphs.
> In particular we've gotten requests in #472 to exclude benchmarks from aggregate results (ie: filtering). I think this could be possible to support as aggregate ranking is not as complex as generating the rest of the graphs.

I think users should be able to do this already by [generating their own report](https://google.github.io/fuzzbench/developing-fuzzbench/custom_analysis_and_reports#generating-reports) and [specifying the benchmarks](https://github.com/google/fuzzbench/blob/master/analysis/generate_report.py#L61-L65) to include. Would that work?




Resolving patch:
diff --git a/analysis/data_utils.py b/analysis/data_utils.py
--- a/analysis/data_utils.py
+++ b/analysis/data_utils.py
@@ -100,18 +100,17 @@ def filter_max_time(experiment_df, max_time):
 
 # Creating "snapshots" (see README.md for definition).
 
-_DEFAULT_BENCHMARK_SAMPLE_NUM_THRESHOLD = 1.0
+_MIN_FRACTION_OF_ALIVE_TRIALS_AT_SNAPSHOT = 0.5
 
 
 def get_benchmark_snapshot(benchmark_df,
-                           threshold=_DEFAULT_BENCHMARK_SAMPLE_NUM_THRESHOLD):
-    """Finds the latest time where all trials were still running
-    (or |threshold| * trials were still running). In most cases, this is the end
-    of the experiment. However, if some trials (or more than (1 - |threshold|) *
-    trials) ended earlier, we will use an earlier "snapshot" time for comparing
-    results.
-
-    Returns data frame that only contains the measurements of the picked
+                           threshold=_MIN_FRACTION_OF_ALIVE_TRIALS_AT_SNAPSHOT):
+    """Finds the latest time where |threshold| fraction of the trials were still
+    running. In most cases, this is the end of the experiment. However, if less
+    than |threshold| fraction of the trials reached the end of the experiment,
+    then we will use an earlier "snapshot" time for comparing results.
+
+    Returns a data frame that only contains the measurements of the picked
     snapshot time.
     """
     # Allow overriding threshold with environment variable as well.


Test patch:
Repository: google/fuzzbench

Issue:
Unique edges does not need two numbers after decimal place
These will never be anything other than 0. 
They also make numbers more likely to bleed into other bars.


Discussion:

Resolving patch:
diff --git a/analysis/plotting.py b/analysis/plotting.py
--- a/analysis/plotting.py
+++ b/analysis/plotting.py
@@ -371,7 +371,7 @@ def unique_coverage_ranking_plot(self,
 
         for patch in axes.patches:
             axes.annotate(
-                format(patch.get_height(), '.2f'),
+                format(patch.get_height(), '.0f'),
                 (patch.get_x() + patch.get_width() / 2., patch.get_height()),
                 ha='center',
                 va='center',


Test patch:
Repository: google/fuzzbench

Issue:
Make coverage graphs show numbers properly when we have large number of fuzzers
E.g.

![image](https://user-images.githubusercontent.com/8731250/91622668-1e665d00-e94d-11ea-9bb1-fe7232c05455.png)

![image](https://user-images.githubusercontent.com/8731250/91622674-22927a80-e94d-11ea-86dd-3da6f06f9dba.png)



Discussion:
It is ok with less fuzzers.


![image](https://user-images.githubusercontent.com/8731250/91622839-e3185e00-e94d-11ea-9232-6c048c890c76.png)

@lszekeres - can you please help with this post-perf/

Resolving patch:
diff --git a/analysis/plotting.py b/analysis/plotting.py
--- a/analysis/plotting.py
+++ b/analysis/plotting.py
@@ -421,4 +421,6 @@ def write_pairwise_unique_coverage_heatmap_plot(
             self, pairwise_unique_coverage_table, image_path):
         """Writes pairwise unique coverage heatmap plot."""
         self._write_plot_to_image(self.pairwise_unique_coverage_heatmap_plot,
-                                  pairwise_unique_coverage_table, image_path)
+                                  pairwise_unique_coverage_table,
+                                  image_path,
+                                  wide=True)


Test patch:
diff --git a/analysis/test_data/pairwise_unique_coverage_heatmap.png b/analysis/test_data/pairwise_unique_coverage_heatmap.png
new file mode 100644
Binary files /dev/null and b/analysis/test_data/pairwise_unique_coverage_heatmap.png differ
diff --git a/analysis/test_plotting.py b/analysis/test_plotting.py
new file mode 100644
--- /dev/null
+++ b/analysis/test_plotting.py
@@ -0,0 +1,36 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions andsss
+# limitations under the License.
+"""Plotting tests."""
+
+import matplotlib.testing.compare as plt_cmp
+import pandas as pd
+
+from analysis import plotting
+
+
+def test_pariwise_unique_coverage_heatmap_plot(tmp_path):
+    """Tests that pairwise unique coverage heatmap looks as expected (even with
+    a large number of fuzzers)."""
+    fuzzer_num = 22
+
+    fuzzers = [f'fuzzer-{i}' for i in range(fuzzer_num)]
+    table_data = [range(1000, 1000 + fuzzer_num)] * fuzzer_num
+    table_df = pd.DataFrame(table_data, index=fuzzers, columns=fuzzers)
+
+    plotter = plotting.Plotter(fuzzers)
+    image_path = tmp_path / 'out.png'
+    plotter.write_pairwise_unique_coverage_heatmap_plot(table_df, image_path)
+
+    golden_path = 'analysis/test_data/pairwise_unique_coverage_heatmap.png'
+    plt_cmp.compare_images(image_path, golden_path, tol=0.01)
Repository: google/fuzzbench

Issue:
Make coverage graphs show numbers properly when we have large number of fuzzers
E.g.

![image](https://user-images.githubusercontent.com/8731250/91622668-1e665d00-e94d-11ea-9bb1-fe7232c05455.png)

![image](https://user-images.githubusercontent.com/8731250/91622674-22927a80-e94d-11ea-86dd-3da6f06f9dba.png)



Discussion:
It is ok with less fuzzers.


![image](https://user-images.githubusercontent.com/8731250/91622839-e3185e00-e94d-11ea-9232-6c048c890c76.png)

@lszekeres - can you please help with this post-perf/

Resolving patch:
diff --git a/analysis/plotting.py b/analysis/plotting.py
--- a/analysis/plotting.py
+++ b/analysis/plotting.py
@@ -398,7 +398,9 @@ def write_unique_coverage_ranking_plot(self, unique_region_cov_df_combined,
                                            image_path):
         """Writes ranking plot for unique coverage."""
         self._write_plot_to_image(self.unique_coverage_ranking_plot,
-                                  unique_region_cov_df_combined, image_path)
+                                  unique_region_cov_df_combined,
+                                  image_path,
+                                  wide=True)
 
     def pairwise_unique_coverage_heatmap_plot(self,
                                               pairwise_unique_coverage_table,


Test patch:
diff --git a/analysis/test_data/unique_coverage_ranking.png b/analysis/test_data/unique_coverage_ranking.png
new file mode 100644
Binary files /dev/null and b/analysis/test_data/unique_coverage_ranking.png differ
diff --git a/analysis/test_plotting.py b/analysis/test_plotting.py
--- a/analysis/test_plotting.py
+++ b/analysis/test_plotting.py
@@ -34,3 +34,26 @@ def test_pariwise_unique_coverage_heatmap_plot(tmp_path):
 
     golden_path = 'analysis/test_data/pairwise_unique_coverage_heatmap.png'
     plt_cmp.compare_images(image_path, golden_path, tol=0.01)
+
+
+def test_unique_coverage_ranking_plot(tmp_path):
+    """Tests that unique coverage ranking plot looks as expected (even with a
+    large number of fuzzers)."""
+    fuzzer_num = 22
+
+    fuzzers = [f'fuzzer-{i}' for i in range(fuzzer_num)]
+    unique_regions = [10 * i for i in range(fuzzer_num)]
+    total_regions = [1000] * fuzzer_num
+
+    df = pd.DataFrame({
+        'fuzzer': fuzzers,
+        'unique_regions_covered': unique_regions,
+        'aggregated_edges_covered': total_regions
+    })
+
+    plotter = plotting.Plotter(fuzzers)
+    image_path = tmp_path / 'out.png'
+    plotter.write_unique_coverage_ranking_plot(df, image_path)
+
+    golden_path = 'analysis/test_data/unique_coverage_ranking.png'
+    plt_cmp.compare_images(image_path, golden_path, tol=0.01)
Repository: google/fuzzbench

Issue:
Dockerfile error in ffmpeg_ffmpeg_demuxer_fuzzer
There is an issue in the Dockerfile to build ffmpeg_ffmpeg_demuxer_fuzzer at this line:

https://github.com/google/fuzzbench/blob/master/benchmarks/ffmpeg_ffmpeg_demuxer_fuzzer/Dockerfile#L17

Consequently you cannot build this benchmark. I tried to fix it by using one of the two hashes, but the compilation does not work in any case (even if with two different errors).


Discussion:
Sorry about this.
I'll try to take a look today or tomorrow.

Resolving patch:
diff --git a/fuzzers/aflplusplus/fuzzer.py b/fuzzers/aflplusplus/fuzzer.py
--- a/fuzzers/aflplusplus/fuzzer.py
+++ b/fuzzers/aflplusplus/fuzzer.py
@@ -252,7 +252,7 @@ def fuzz(input_corpus,
         flags += ['-x', './afl++.dict']
 
     # Move the following to skip for upcoming _double tests:
-    if os.path.exists(cmplog_target_binary) and no_cmplog is not False:
+    if os.path.exists(cmplog_target_binary) and no_cmplog is False:
         flags += ['-c', cmplog_target_binary]
 
     if not skip:


Test patch:
Repository: google/fuzzbench

Issue:
Unbound variable in scheduler.py 
```
UnboundLocalError: local variable 'trial_instance_manager' referenced before assignment Message: Error occurred during scheduling.

at .schedule_loop ( [/work/src/experiment/scheduler.py:585](https://pantheon.corp.google.com/debug?referrer=fromlog&file=%2Fwork%2Fsrc%2Fexperiment%2Fscheduler.py&line=585&appModule=python&appVersion=&project=fuzzbench) )
```

@andreafioraldi I suspect this is related to #1356


Discussion:
It should never happen, handle_preempted is never True during local experiments. Is this a static analysis warning?
Ah yes got it, I could not trigger it with local experiments, PR coming

Resolving patch:
diff --git a/experiment/scheduler.py b/experiment/scheduler.py
--- a/experiment/scheduler.py
+++ b/experiment/scheduler.py
@@ -562,9 +562,11 @@ def schedule_loop(experiment_config: dict):
             pool_args = (processes, _process_init, (cores_queue,))
         else:
             pool_args = (runners_cpus,)
-            gce.initialize()
-            trial_instance_manager = TrialInstanceManager(
-                num_trials, experiment_config)
+
+    if not local_experiment:
+        gce.initialize()
+        trial_instance_manager = TrialInstanceManager(num_trials,
+                                                      experiment_config)
 
     experiment = experiment_config['experiment']
     with multiprocessing.Pool(*pool_args) as pool:


Test patch:
Repository: google/fuzzbench

Issue:
Write a validator to prevent issues requesting bug experiments
https://github.com/google/fuzzbench/commit/da5fab4dfcabcd856e0247b90122bb577f0c734e#diff-c89bd86bcf4447f199973358896b7a69d69d92f258653f28e77583ee8967dd13R49 has happened twice now, time to prevent it.


Discussion:

Resolving patch:
diff --git a/service/automatic_run_experiment.py b/service/automatic_run_experiment.py
--- a/service/automatic_run_experiment.py
+++ b/service/automatic_run_experiment.py
@@ -143,6 +143,18 @@ def _validate_individual_experiment_requests(experiment_requests):
                          experiment_type, benchmark_utils.BENCHMARK_TYPE_STRS)
             valid = False
 
+        benchmarks = request.get('benchmarks', [])
+        for benchmark in benchmarks:
+            benchmark_type = benchmark_utils.get_type(benchmark)
+            if (benchmark_type == benchmark_utils.BenchmarkType.BUG.value and
+                    experiment_type != benchmark_utils.BenchmarkType.BUG.value):
+                logger.error(
+                    'Benchmark %s is "type: bug". '
+                    'Experiment %s must be "type: bug" as well.', benchmark,
+                    experiment)
+                valid = False
+                break
+
     return valid
 
 


Test patch:
Repository: google/fuzzbench

Issue:
More configurable builds
Address #1512:
1. Make the worker pool configurable;
2. Make the number of  concurrent builds configurable.


Discussion:
CI tests passed, but shall we add some testcases?
Modified a testcase to test `worker_pool` (and all other `cloud_params`).
I noticed that there is no existing test for command line parameters (e.g. `concurrent_build`), so I reckon it is probably better to add them together in a different PR.
In addition, we are [validating the argument parameters manually](https://github.com/google/fuzzbench/blob/da05da0dcb3680deee9d894d1bfa61edbbf4a90c/experiment/run_experiment.py#L616), which is probably better done with [`argparser`'s type](https://docs.python.org/3/library/argparse.html#type).
I will do that in the new PR.
I reckon this PR is ready, please have a final look at it and let me know if there is anything else I can do.
Thanks!
I reckon this is the same as we discussed.
In the next PR, I will:
1. Merge the config `YAML` values and the command line parameters to the intermediate config file.
2. Create a separate module to set `ENV` values with the intermediate config file.
3. Use the module to access the settings from `ENV` values.

Please let me know if I missed anything, thanks! @jonathanmetzman 
> I reckon this is the same as we discussed. In the next PR, I will:
> 
> 1. Merge the config `YAML` values and the command line parameters to the intermediate config file.
> 2. Create a separate module to set `ENV` values with the intermediate config file.
> 3. Use the module to access the settings from `ENV` values.
> 
> Please let me know if I missed anything, thanks! @jonathanmetzman

I think this is it.
It just occur to me that I have not updated the config YAML accordingly.
I think that is why our experiment with the new version of `libFuzzer` did not launch.
Let me fix that in a new PR later.

Resolving patch:
diff --git a/experiment/run_experiment.py b/experiment/run_experiment.py
--- a/experiment/run_experiment.py
+++ b/experiment/run_experiment.py
@@ -172,6 +172,8 @@ def read_and_validate_experiment_config(config_filename: str) -> Dict:
             Requirement(not local_experiment, str, False, ''),
         'experiment':
             Requirement(False, str, False, ''),
+        'cloud_sql_instance_connection_name':
+            Requirement(False, str, True, ''),
         'snapshot_period':
             Requirement(False, int, False, ''),
         'local_experiment':
@@ -180,6 +182,8 @@ def read_and_validate_experiment_config(config_filename: str) -> Dict:
             Requirement(False, bool, False, ''),
         'merge_with_nonprivate':
             Requirement(False, bool, False, ''),
+        'preemptible_runners':
+            Requirement(False, bool, False, ''),
     }
 
     all_params_valid = _validate_config_parameters(config, config_requirements)


Test patch:
Repository: canonical/microk8s

Issue:
Indentation error in yaml output of microk8s.status
The output of `microk8s.status` is

```
microk8s:
  running: False
high-availability:
  enabled: False
   message: microk8s is not running. Use microk8s inspect for a deeper inspection.
```

which confuses some parsers (e.g. the built-in Python 3.8) due to the extraneous space before `message`.


Discussion:

Resolving patch:
diff --git a/scripts/wrappers/status.py b/scripts/wrappers/status.py
--- a/scripts/wrappers/status.py
+++ b/scripts/wrappers/status.py
@@ -98,7 +98,7 @@ def print_short_yaml(isReady, enabled_addons, disabled_addons):
             print("  {}: disabled".format(disabled["name"]))
     else:
         print(
-            "{:>2} {} {}".format(
+            "{:>2}{} {}".format(
                 "",
                 "message:",
                 "microk8s is not running. Use microk8s inspect for a deeper inspection.",
@@ -135,7 +135,7 @@ def print_yaml(isReady, enabled_addons, disabled_addons):
             print("{:>4}status: disabled".format(""))
     else:
         print(
-            "{:>2} {} {}".format(
+            "{:>2}{} {}".format(
                 "",
                 "message:",
                 "microk8s is not running. Use microk8s inspect for a deeper inspection.",


Test patch:
Repository: canonical/microk8s

Issue:
latest/edge: server cert generation failure too verbose
```
Restarting cluster-agent to load new server certificate

error: error running snapctl: snap "microk8s" has "service-control" change in progress

Traceback (most recent call last):

  File "/snap/microk8s/3125/scripts/cluster/add_token.py", line 190, in <module>

    subprocess.check_call(["snapctl", "restart", "microk8s.daemon-cluster-agent"])

  File "/snap/microk8s/3125/usr/lib/python3.6/subprocess.py", line 311, in check_call

    raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '['snapctl', 'restart', 'microk8s.daemon-cluster-agent']' returned non-zero exit status 1.

```

The action failing is fine, but the call stack being printed is un-necessary 


Discussion:

Resolving patch:
diff --git a/scripts/cluster/add_token.py b/scripts/cluster/add_token.py
--- a/scripts/cluster/add_token.py
+++ b/scripts/cluster/add_token.py
@@ -5,9 +5,6 @@
 import time
 import argparse
 import subprocess
-import hashlib
-import ssl
-import http.client
 
 from common.utils import is_node_running_dqlite
 
@@ -175,20 +172,6 @@ def print_short(token, check):
     add_token_with_expiry(token, cluster_tokens_file, ttl)
     check = run_util("server_cert_check")
 
-    # NOTE(akolaitis): if the server certificate has changed after the cluster-agent
-    # service started, the server hash will be out of date. make sure to restart
-    # the cluster-agent in this case, otherwise the joining nodes will be unable
-    # to verify.
-    context = ssl._create_unverified_context()
-    conn = http.client.HTTPSConnection("127.0.0.1:25000", context=context)
-    conn.connect()
-    der_cert_bin = conn.sock.getpeercert(True)
-    conn.close()
-    peer_cert_hash = hashlib.sha256(der_cert_bin).hexdigest()
-    if not peer_cert_hash.startswith(check):
-        print("Restarting cluster-agent to load new server certificate")
-        subprocess.check_call(["snapctl", "restart", "microk8s.daemon-cluster-agent"])
-
     if args.format == "pretty":
         print_pretty(token, check)
     elif args.format == "short":


Test patch:
Repository: canonical/microk8s

Issue:
latest/edge: skip-verify flag not honoured as expected
```
❯ microk8s join 10.126.166.1:25000/25a8af1d57e5b04e558a929cce4c0ed3/6b2e92ea203f --skip-verify

Contacting cluster at 10.126.166.1

Joining cluster failed. Could not verify the identity of 10.126.166.1. Use '--skip-verify' to skip server certificate check.
```

On `latest/edge` branch


Discussion:

Resolving patch:
diff --git a/scripts/cluster/join.py b/scripts/cluster/join.py
--- a/scripts/cluster/join.py
+++ b/scripts/cluster/join.py
@@ -723,7 +723,9 @@ def join_dqlite(connection_parts, verify=False, worker=False):
     fingerprint = None
     if len(connection_parts) > 2:
         fingerprint = connection_parts[2]
-        verify = True
+    else:
+        # we do not have a fingerprint, do not attempt to verify the remote cert
+        verify = False
 
     print("Contacting cluster at {}".format(master_ip))
 


Test patch:
Repository: canonical/microk8s

Issue:
Kubelet node-ip parameter overrided while joining and join don't work
#### Summary
When joining a node that have `--node-ip` parameter set in `/var/snap/microk8s/current/args/kubelet` this parameter is overrided and take the value of the joined master; the node never join and microk8s crash on new node.

In `join.py`, i suspect the problem to come from the values in `info["kubelet_args"]` given by `get_connection_info` function to be written in `kubelet` file by the `store_base_kubelet_args` function.

We may respect the actual `node-ip` parameter configured before join.

#### Initial problem
I have to use two network interface and the only way to make it work is to hardcode ip: `--node-ip` in `args/kubelet file` and `--advertise-address` in `args/kube-apiserver`.


Discussion:
may be related to: https://github.com/canonical/microk8s/issues/2402 see this comment: https://github.com/canonical/microk8s/issues/2402#issuecomment-875086145

Resolving patch:
diff --git a/scripts/cluster/join.py b/scripts/cluster/join.py
--- a/scripts/cluster/join.py
+++ b/scripts/cluster/join.py
@@ -550,6 +550,17 @@ def store_base_kubelet_args(args_string):
     try_set_file_permissions(args_file)
 
 
+def update_kubelet_node_ip(args_string, hostname_override):
+    """
+    Update the kubelet --node-ip argument if it was set on the node that we join.
+
+    :param args_string: the kubelet arguments
+    :param hostname_override: the source IP address used by the node when joining
+    """
+    if "--node-ip" in args_string:
+        set_arg("--node-ip", hostname_override, "kubelet")
+
+
 def replace_admin_token(token):
     """
     Replaces the admin token in the known tokens
@@ -822,7 +833,7 @@ def join_dqlite_worker_node(info, master_ip, master_port, token):
     store_cert("serviceaccount.key", info["service_account_key"])
 
     store_base_kubelet_args(info["kubelet_args"])
-
+    update_kubelet_node_ip(info["kubelet_args"], hostname_override)
     update_cert_auth_kubeproxy(token, info["ca"], master_ip, master_port, hostname_override)
     update_cert_auth_kubelet(token, info["ca"], master_ip, master_port)
 
@@ -868,6 +879,7 @@ def join_dqlite_master_node(info, master_ip, token):
 
     create_admin_kubeconfig(info["ca"], info["admin_token"])
     store_base_kubelet_args(info["kubelet_args"])
+    update_kubelet_node_ip(info["kubelet_args"], hostname_override)
     store_callback_token(info["callback_token"])
     update_dqlite(info["cluster_cert"], info["cluster_key"], info["voters"], hostname_override)
     # We want to update the local CNI yaml but we do not want to apply it.
@@ -892,6 +904,8 @@ def join_etcd(connection_parts, verify=True):
     hostname_override = None
     if "hostname_override" in info:
         hostname_override = info["hostname_override"]
+        update_kubelet_node_ip(info["kubelet_args"], hostname_override)
+
     store_remote_ca(info["ca"])
     update_flannel(info["etcd"], master_ip, master_port, token)
     update_kubeproxy(info["kubeproxy"], info["ca"], master_ip, info["apiport"], hostname_override)


Test patch: