Repository: google/fuzzbench

Issue:
Interpretation of the FuzzBench Reports
The available fuzzer configurations and the results might be confusing for someone who looks at a random FuzzBench report for the first time. To avoid misinterpretation of the results, maybe there could be
1. a description of the intent of the evaluation and/or an explanation of the evaluated fuzzer configurations, and
2. a distinction between "master/official/representative" and "developer" evaluations and fuzzer variants. 


Discussion:
I think I agree with this overall. 1 seems easy to do, we should probably link to some place in the docs where we explain what the report is and provide necessary caveats. I'm having a hard time imagining what 2 looks like in practice though. Maybe people can tag fuzzers as "official" or not and that tag can make its way into reports. On some level though I think adding this functionality might not be the highest priority.
What about naming fuzzer configurations accordingly?
Ah. So for example, if one of the fuzzbench devs integrates ankou, we would name the fuzzer "ankou_experimental"? If this isn't what you have in mind could you provide a specific example?
Yes, for instance. There was a discussion on Twitter where aflplusplus_optimal (which is highly overfitted to the benchmark just to represent the best version of AFL++, and @vanhauser-thc uses more as a reference to find good "mainline" AFL++ default settings) was misunderstood as the mainline AFL++ version.
this also happened because the aflplusplus instance was not in the general report. as I previously stated (months ago), both should be in there, aflplusplus and aflplusplus_optimal for fairness.

- for 1) we could introduce a `description.txt` in every fuzzer directory that'd describe the config and include that in the report
- for 2) we've been discussing having a list of the "official" fuzzers (that'd be included by default in the "official" reports) in a file, but we could do this too with having special files in the fuzzer dirs (e.g. just leaving an empty `.experimental` file or something)


for 2), we have official list in https://github.com/google/fuzzbench/blob/master/service/core-fuzzers.yaml
Nice! For 1) we could use that list to put "experimental" reports under say `fuzzbench.com/reports/experimental`

For 2, @vanhauser-thc, would you mind adding a super short `description.md` under the `fuzzer/aflplusplus_*` dirs? 
(Feel free to use markdown to include links and such) 
I can try to describe the rest of them.

> For 2, @vanhauser-thc, would you mind adding a super short `description.md` under the `fuzzer/aflplusplus_*` dirs?
> (Feel free to use markdown to include links and such)
> I can try to describe the rest of them.

sure I will add that in the next push.
do you want to also for temporary variants that are just there for a single test?
> sure I will add that in the next push.

Thanks!

> do you want to also for temporary variants that are just there for a single test?

Yes, the goal is to not leave anything unexplained in any report :) thanks!
It might also help to hyperlink fuzzer.py in each report.

You could mark fuzzers that do not support a particular benchmark (explaining an unusually low rank) -- but (probably needing a separate Issue) I actually think that fuzzers should be able to "register" for certain *categories* of benchmarks (e.g., only those that take structured file formats), but not specific benchmarks. Fuzzers would only be evaluated within their specific categories.

[Edit: Whoops, wrong button.]
I put a description.md with a detailed description. for future temporary variants I will keep it shorter though :)


Resolving patch:
diff --git a/analysis/experiment_results.py b/analysis/experiment_results.py
--- a/analysis/experiment_results.py
+++ b/analysis/experiment_results.py
@@ -72,6 +72,19 @@ def __init__(  # pylint: disable=too-many-arguments
     def _get_full_path(self, filename):
         return os.path.join(self._output_directory, filename)
 
+    def linkify_names(self, df):
+        """For any DataFrame which is indexed by fuzzer names, turns the fuzzer
+        names into links to their directory with a description on GitHub."""
+        assert df.index.name == 'fuzzer'
+
+        def description_link(commit, fuzzer):
+            return (f'<a href="https://github.com/google/fuzzbench/blob/'
+                    f'{commit}/fuzzers/{fuzzer}">{fuzzer}</a>')
+
+        commit = self.git_hash if self.git_hash else 'master'
+        df.index = df.index.map(lambda fuzzer: description_link(commit, fuzzer))
+        return df
+
     @property
     @functools.lru_cache()
     # TODO(lszekeres): With python3.8+, replace above two decorators with:


Test patch:
diff --git a/analysis/test_experiment_results.py b/analysis/test_experiment_results.py
new file mode 100644
--- /dev/null
+++ b/analysis/test_experiment_results.py
@@ -0,0 +1,33 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions andsss
+# limitations under the License.
+"""Tests for experiment_results.py"""
+
+from analysis import experiment_results
+from analysis import test_data_utils
+
+
+def test_linkify_fuzzer_names_in_ranking():
+    """Tests turning fuzzer names into links."""
+    experiment_df = test_data_utils.create_experiment_data()
+    results = experiment_results.ExperimentResults(experiment_df,
+                                                   coverage_dict=None,
+                                                   output_directory=None,
+                                                   plotter=None)
+    ranking = results.rank_by_median_and_average_rank
+
+    ranking = results.linkify_names(ranking)
+
+    assert ranking.index[0] == (
+        '<a href="https://github.com/google/fuzzbench/blob/'
+        'master/fuzzers/afl">afl</a>')