Repository: facebookresearch/CompilerGym

Issue:
[llbm] miBench dataset has optnone function attribute set
## 🐛 Bug

I am not sure whether it is a bug, or perhaps it is due to the dataset?

I run the e-greedy algorithm https://github.com/facebookresearch/CompilerGym/blob/development/leaderboard/llvm_instcount/e_greedy/e_greedy.py on  mibench. The results seems does not make sense at all: all rewards are 0 in terms of instruction count.

I don't think it is due the implementation of e_greedy.py. Actually I wrote one version of greedy algorithm and it has the same results.

Why this happen? any problem in the benchmark? should we add some notes about this in the website? 

## To Reproduce

Steps to reproduce the behavior:

1. `python e_greedy.py --n=1 --epsilon=0 --test_dataset=mibench-v0 --leaderboard_results=greedy_mibench.csv`


[greedy_mibench.csv](https://github.com/facebookresearch/CompilerGym/files/7677924/greedy_mibench.csv)


## Environment

Please fill in this checklist:

- CompilerGym: 0.2.1
- How you installed CompilerGym (conda, pip, source): pip
- OS: Ubuntu 18.04
- Python version: 3.6.9
- Build command you used (if compiling from source):
- GCC/clang version (if compiling from source):
- Bazel version (if compiling from source):
- Versions of any other relevant libraries:


Discussion:
Hi @yqtianust, thanks for the report!

This is indeed a bug. All mibench programs have the same size when optimized by -Oz. Test script:

```py
import compiler_gym
env = compiler_gym.make("llvm-v0")
for benchmark in env.datasets["mibench-v0"].benchmarks():
    env.reset(benchmark=benchmark)
    print(benchmark, env.observation.IrInstructionCount(), env.observation.IrInstructionCountOz())
```

Produces output:

```
benchmark://mibench-v0/bitcount-1 124 124
benchmark://mibench-v0/bitcount-2 131 131
benchmark://mibench-v0/dijkstra 121 121
benchmark://mibench-v0/fft-1 131 131
benchmark://mibench-v0/fft-2 362 362
benchmark://mibench-v0/ghostscript 194 194
benchmark://mibench-v0/gsm-1 218 218
benchmark://mibench-v0/gsm-2 222 222
benchmark://mibench-v0/jpeg-c-1 177 177
benchmark://mibench-v0/jpeg-c-2 347 347
benchmark://mibench-v0/jpeg-c-3 325 325
benchmark://mibench-v0/jpeg-c 645 645
benchmark://mibench-v0/lame-newmdct-1 569 569
benchmark://mibench-v0/lame-newmdct-2 189 189
benchmark://mibench-v0/lame-psymodel 1563 1562
benchmark://mibench-v0/lame-quantize-1 161 161
benchmark://mibench-v0/lame-quantize-2 129 129
benchmark://mibench-v0/lame-takehiro-1 386 388
benchmark://mibench-v0/lame-takehiro-2 162 162
benchmark://mibench-v0/mad-1 391 391
benchmark://mibench-v0/mad-2 243 243
benchmark://mibench-v0/pgp-d 149 149
benchmark://mibench-v0/pgp-e-1 149 149
benchmark://mibench-v0/pgp-e-2 158 158
benchmark://mibench-v0/pgp-e-3 171 171
benchmark://mibench-v0/qsort1 114 114
benchmark://mibench-v0/rsynth-1 224 224
benchmark://mibench-v0/rsynth-2 155 155
benchmark://mibench-v0/susan-e-1 215 215
benchmark://mibench-v0/susan-e-2 1629 1629
benchmark://mibench-v0/susan-s 231 231
benchmark://mibench-v0/tiff2rgba 340 340
benchmark://mibench-v0/tiffdither-1 331 331
benchmark://mibench-v0/tiffdither-2 331 331
benchmark://mibench-v0/tiffdither-3 244 244
benchmark://mibench-v0/tiffmedian-1 229 229
benchmark://mibench-v0/tiffmedian-2 158 158
benchmark://mibench-v0/tiffmedian-3 220 220
benchmark://mibench-v0/tiffmedian-4 251 251
benchmark://mibench-v0/tiffmedian-5 181 181
```

The problem is that mibench bitcodes have been prepared without the right flags, causing the `optnone` function attribute on all bitcodes:

```llvm
; Function Attrs: noinline nounwind optnone ssp uwtable
define void @astex_codelet__6(%struct.colorbox* %0, %struct.colorbox* %1, i32 %2, %struct.colorbox** %3, i32* %4) #0 {
```

For more context, see [here](https://github.com/facebookresearch/CompilerGym/blob/development/compiler_gym/envs/llvm/llvm_benchmark.py#L150-L169). I will fix this before the next release, either by recompiling the bitcodes or removing the dataset.

Good catch!

Cheers,
Chris
Thanks for the detailed feedback. Good to know it is a bug. 

Resolving patch:
diff --git a/compiler_gym/envs/llvm/datasets/__init__.py b/compiler_gym/envs/llvm/datasets/__init__.py
--- a/compiler_gym/envs/llvm/datasets/__init__.py
+++ b/compiler_gym/envs/llvm/datasets/__init__.py
@@ -109,6 +109,30 @@ def __init__(self, site_data_base: Path, sort_order: int = 0):
 
 
 class MibenchDataset(TarDatasetWithManifest):
+    def __init__(self, site_data_base: Path, sort_order: int = 0):
+        super().__init__(
+            name="benchmark://mibench-v1",
+            tar_urls=[
+                "https://dl.fbaipublicfiles.com/compiler_gym/llvm_bitcodes-10.0.0-mibench-v1.tar.bz2"
+            ],
+            tar_sha256="795b80d3198bc96e394823a4cb294d256845beffccce52fea0e3446395212bb5",
+            manifest_urls=[
+                "https://dl.fbaipublicfiles.com/compiler_gym/llvm_bitcodes-10.0.0-mibench-v0-manifest.bz2"
+            ],
+            manifest_sha256="8ed985d685b48f444a3312cd84ccc5debda4a839850e442a3cdc93910ba0dc5f",
+            references={
+                "Paper": "http://vhosts.eecs.umich.edu/mibench/Publications/MiBench.pdf"
+            },
+            license="BSD 3-Clause",
+            strip_prefix="mibench-v1",
+            description="C benchmarks",
+            benchmark_file_suffix=".bc",
+            site_data_base=site_data_base,
+            sort_order=sort_order,
+        )
+
+
+class MibenchV0Dataset(TarDatasetWithManifest):
     def __init__(self, site_data_base: Path, sort_order: int = 0):
         super().__init__(
             name="benchmark://mibench-v0",
@@ -129,6 +153,7 @@ def __init__(self, site_data_base: Path, sort_order: int = 0):
             benchmark_file_suffix=".bc",
             site_data_base=site_data_base,
             sort_order=sort_order,
+            deprecated="Please use mibench-v1",
         )
 
 
@@ -259,6 +284,7 @@ def get_llvm_datasets(site_data_base: Optional[Path] = None) -> Iterable[Dataset
     yield LinuxDataset(site_data_base=site_data_base, sort_order=0)
     yield LlvmStressDataset(site_data_base=site_data_base, sort_order=0)
     yield MibenchDataset(site_data_base=site_data_base, sort_order=0)
+    yield MibenchV0Dataset(site_data_base=site_data_base, sort_order=100)
     yield NPBDataset(site_data_base=site_data_base, sort_order=0)
     yield OpenCVDataset(site_data_base=site_data_base, sort_order=0)
     yield POJ104Dataset(site_data_base=site_data_base, sort_order=0)


Test patch:
diff --git a/examples/llvm_rl/config/testing/all.yaml b/examples/llvm_rl/config/testing/all.yaml
--- a/examples/llvm_rl/config/testing/all.yaml
+++ b/examples/llvm_rl/config/testing/all.yaml
@@ -21,7 +21,7 @@ benchmarks:
       max_benchmarks: 50
     - dataset: generator://llvm-stress-v0
       max_benchmarks: 50
-    - dataset: benchmark://mibench-v0
+    - dataset: benchmark://mibench-v1
       max_benchmarks: 50
     - dataset: benchmark://npb-v0
       max_benchmarks: 50
diff --git a/tests/llvm/datasets/llvm_datasets_test.py b/tests/llvm/datasets/llvm_datasets_test.py
--- a/tests/llvm/datasets/llvm_datasets_test.py
+++ b/tests/llvm/datasets/llvm_datasets_test.py
@@ -19,7 +19,7 @@ def test_default_dataset_list():
             "benchmark://clgen-v0",
             "benchmark://github-v0",
             "benchmark://linux-v0",
-            "benchmark://mibench-v0",
+            "benchmark://mibench-v1",
             "benchmark://npb-v0",
             "benchmark://opencv-v0",
             "benchmark://poj104-v1",