Repository: google/fuzzbench

Issue:
Support specifying which benchmarks to include in the report 
The current set of benchmarks is diverse, which is great. However, since some fuzzers may not work very well with all types of input formats (e.g., context-free grammar inputs vs chunk-based inputs, binary inputs vs text inputs), it would be great if there is an option for us to choose which benchmarks should be included in the report. Of course, when we submit papers for review, we should explain why we ignore some benchmarks from our evaluation.


Discussion:
Should be a simple change to do, similar to https://github.com/google/fuzzbench/commit/2db4efc668f23a5dde4ae10de30998af19cf739e ? That will allow you to generate report for only a set of benchmarks.
Thanks, Abhishek.
Yes, perfect! A change imilar to 2db4efc should work even though the argument could be very long :) There are several benchmarks and their names are longer than the fuzzers' names. 

Resolving patch:
diff --git a/analysis/data_utils.py b/analysis/data_utils.py
--- a/analysis/data_utils.py
+++ b/analysis/data_utils.py
@@ -29,10 +29,16 @@ def drop_uninteresting_columns(experiment_df):
 
 
 def filter_fuzzers(experiment_df, included_fuzzers):
-    """Returns table with only rows where fuzzer in in |included_fuzzers|."""
+    """Returns table with only rows where fuzzer is in |included_fuzzers|."""
     return experiment_df[experiment_df['fuzzer'].isin(included_fuzzers)]
 
 
+def filter_benchmarks(experiment_df, included_benchmarks):
+    """Returns table with only rows where benchmark is in
+    |included_benchmarks|."""
+    return experiment_df[experiment_df['benchmark'].isin(included_benchmarks)]
+
+
 def label_fuzzers_by_experiment(experiment_df):
     """Returns table where every fuzzer is labeled by the experiment it
     was run in."""
diff --git a/experiment/generate_report.py b/experiment/generate_report.py
--- a/experiment/generate_report.py
+++ b/experiment/generate_report.py
@@ -53,10 +53,16 @@ def get_arg_parser():
         action='store_true',
         default=False,
         help='If set, plots are created faster, but contain less details.')
-    parser.add_argument('-f',
-                        '--fuzzers',
-                        nargs='*',
-                        help='Names of the fuzzers to include in the report. ')
+    parser.add_argument(
+        '-b',
+        '--benchmarks',
+        nargs='*',
+        help='Names of the benchmarks to include in the report.')
+    parser.add_argument(
+        '-f',
+        '--fuzzers',
+        nargs='*',
+        help='Names of the fuzzers to include in the report.')
     parser.add_argument(
         '-l',
         '--label-by-experiment',
@@ -80,6 +86,7 @@ def generate_report(experiment_names,
                     report_directory,
                     report_name=None,
                     label_by_experiment=False,
+                    benchmarks=None,
                     fuzzers=None,
                     report_type='default',
                     quick=False,
@@ -97,6 +104,9 @@ def generate_report(experiment_names,
         # Save the raw data along with the report.
         experiment_df.to_csv(data_path)
 
+    if benchmarks is not None:
+        experiment_df = data_utils.filter_benchmarks(experiment_df, benchmarks)
+
     if fuzzers is not None:
         experiment_df = data_utils.filter_fuzzers(experiment_df, fuzzers)
 
@@ -123,7 +133,7 @@ def main():
     args = parser.parse_args()
 
     generate_report(args.experiments, args.report_dir, args.report_name,
-                    args.label_by_experiment, args.fuzzers, args.report_type,
+                    args.label_by_experiment, args.benchmarks, args.fuzzers, args.report_type,
                     args.quick, args.from_cached_data)
 
 


Test patch: