Repository: alibaba/BladeDISC (language: python)

Issue:
[RFC] Add TorchBlade benchmarks based on TorchBench
### Why TorchBench?
1. A benchmark maintained by the PyTorch community officially
2. `torchbenchmark/models` contains copies of popular or exemplary workloads 
3. `torch.utils.benchmark.Timer` has priorities than `timeit.Timer` in PyTorch
4. Easier to generate, save and compare benchmark reports
### Goals of Benchmark
1. Test TorchBlade on community models
2. Compare with other optimization methods
3. Provide guidance on the improvement of TorchBlade contiguously
4. Survey on new techs, such as TorchDynamo and training
### TorchBench & TorchDynamo
TorchDynamo is a Python-level JIT compiler designed to make unmodified PyTorch programs faster.  It does performance measurements based on TorchBench. We are doing a survey on enabling TorchBlade in TorchDynamo infrastructure recently.
### TorchBlade & TorchBench
TorchBlade is a optimzation tool for PyTorch program.  We want to benchmark the performance compared to other methods. Given the usage of TorchBench in TorchDynamo, we would like to construct our benchmarks based on TorchBench as well. For now, we will add the following benchmarks:
1. TorchBlade benchmark based on torchdynamo
2. TorchBlade benchmark based on TorchScript
### What is needed to build the benchmark
1. A Github CI job, daily or weekly. The job will upload reports to Alibaba Cloud Object Storage Service or save reports on Github
2. A docker image installed with pre-built PyTorch(or some stable version) & TorchBlade
3. Benchmark scripts based on TorchBench
  ○ TorchDynamo/TorchScript
  ○ CUDA/CPU
  ○ TensorRT/DISC
  ○ F32/F16/Int8
4. Report performance, collect perf statistics
5. Add custom model list

### Tasks
- [x] Build a docker image prebuilt by BladeDISC daily
- [x] Construct a CI cronjob that is based on the image
- [x] Add torchdynamo benchmark launch scripts #465
- [x] Add torchdynamo benchmark to cronjobs #485
- [x] Add torchscript benchmark launch scripts to cronjobs
- [x] Add more different configurations(cpu/gpu, tensorrt/disc, fp32/fp16/int8) etc.
- [x] Add perf statistics(#cluster   #mem-intensive-kernel   time-of-mem-intensive-kernel   #compute-intensive-kernel   time-of-compute-intensive-kernel)
- [x] Generate human-readable reports
- [x] Upload reports && summaries https://github.com/alibaba/BladeDISC/pull/607
- [ ] More custom model list
- [ ] Others ....(need refinements)

### References
https://pytorch.org/tutorials/recipes/recipes/benchmark.html#steps
https://github.com/pytorch/benchmark.git
https://github.com/pytorch/torchdynamo.git


Discussion / hints:
> TorchDynamo is a Python-level JIT compiler designed to make unmodified PyTorch programs faster. It does performance measurements based on TorchBench. We are doing a survey on enabling TorchBlade in TorchDynamo infrastructure recently.

It seems to enable TorchBlade in TorchDynamo is an individual topic, maybe we can talk about that in another issue.

> A Github CI job, daily or weekly. The job will upload reports to OSS or save reports on Github

Do you want to compare the latest report to previous one, that helps us to find the PR that hurt the performance?

> A docker image installed with pre-built PyTorch(or some stable version)

We have deployed the latest runtime Docker image, can we reuse that?


> A Github CI job, daily or weekly. The job will upload reports to OSS or save reports on Github

nit: `OSS` usually means `Open Source Software` in community. Maybe Alibaba cloud Object Storage Service is better?


> > TorchDynamo is a Python-level JIT compiler designed to make unmodified PyTorch programs faster. It does performance measurements based on TorchBench. We are doing a survey on enabling TorchBlade in TorchDynamo infrastructure recently.
> 
> It seems to enable TorchBlade in TorchDynamo is an individual topic, maybe we can talk about that in another issue.
> 
> > A Github CI job, daily or weekly. The job will upload reports to OSS or save reports on Github
> 
> Do you want to compare the latest report to previous one, that helps us to find the PR that hurt the performance?
> 
> > A docker image installed with pre-built PyTorch(or some stable version)
> 
> We have deployed the latest runtime Docker image, can we reuse that?

1. TorchDynamo is mentioned here because it's one of the goals of adding TorchBench to TorchBlade.
2. Yes, TorchBench provides good utils to make comparisons between multiple reports. See https://pytorch.org/tutorials/recipes/recipes/benchmark.html#comparing-benchmark-results
3. This is a good suggestion!
> A docker image installed with pre-built PyTorch(or some stable version)

@zzpmiracle Please have a try on this image bladedisc/bladedisc:latest-runtime-torch1.12.0-cu113
Related PR to torchbench repo: https://github.com/pytorch/benchmark/pull/1045 , https://github.com/pytorch/benchmark/pull/1072, https://github.com/pytorch/benchmark/pull/1073

Fix patch:
diff --git a/.github/workflows/cronjobs.yml b/.github/workflows/cronjobs.yml
index 5dd27be84f5..c7861056a8c 100644
--- a/.github/workflows/cronjobs.yml
+++ b/.github/workflows/cronjobs.yml
@@ -17,6 +17,7 @@ jobs:
       base_image: nvcr.io/nvidia/pytorch:22.01-py3
       device: gpu-t4
       dockerfile: docker/cronjobs/Dockerfile.torch.ngc
+      extra_build_args: --gpus all
       exec_command: bash -c "cd pytorch_blade && bash ./benchmark/run_benchmark.sh"
       push_command: source scripts/ci/deploy_docker.sh && push_images
       remote_dev_docker: bladedisc:disc-dev-pt-ngc
@@ -31,8 +32,9 @@ jobs:
     with:
       name: torch-offcial-benchmark
       base_image: bladedisc/bladedisc:latest-runtime-torch1.12.0-cu113
-      device: gpu-t4
+      device: gpu-a10
       dockerfile: docker/cronjobs/Dockerfile.torch.bench
+      extra_build_args: --cap-add=SYS_ADMIN --gpus all
       exec_command: bash ./pytorch_blade/benchmark/TorchBench/test_torch_bench.sh
       push_command: ""
       remote_dev_docker: ""
diff --git a/.github/workflows/reusable.yml b/.github/workflows/reusable.yml
index c8bb3b307d3..49b7e4780d3 100644
--- a/.github/workflows/reusable.yml
+++ b/.github/workflows/reusable.yml
@@ -63,7 +63,7 @@ jobs:
     - name: Run Job 
       run: |
         set -e
-        docker run --gpus all --rm -t --user $(id -u):$(id -g) \
+        docker run --rm -t --user $(id -u):$(id -g) \
           -v $HOME/.cache:$HOME/.cache \
           -v /etc/passwd:/etc/passwd:ro \
           -v /etc/group:/etc/group:ro \
diff --git a/docker/cronjobs/Dockerfile.torch.bench b/docker/cronjobs/Dockerfile.torch.bench
index 1e2de31882a..d5ffd36a8fa 100644
--- a/docker/cronjobs/Dockerfile.torch.bench
+++ b/docker/cronjobs/Dockerfile.torch.bench
@@ -1,6 +1,22 @@
 ARG BASEIMAGE=bladedisc/bladedisc:latest-runtime-torch1.12.0-cu113
 FROM ${BASEIMAGE}
 
-RUN apt-get install -y git git-lfs libglib2.0-0 libsndfile1 libgl1
+RUN apt-get install -y git git-lfs libglib2.0-0 libsndfile1 libgl1 && \
+    rm -rf /var/lib/apt/lists/*
+# install nsys
+RUN apt-get update -y && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+        apt-transport-https \
+        ca-certificates \
+        gnupg \
+        wget && \
+    rm -rf /var/lib/apt/lists/*
+RUN wget -qO - https://developer.download.nvidia.com/devtools/repos/ubuntu2004/amd64/nvidia.pub | apt-key add - && \
+    echo "deb https://developer.download.nvidia.com/devtools/repos/ubuntu2004/amd64/ /" >> /etc/apt/sources.list.d/nsight.list && \
+    apt-get update -y && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+        nsight-systems-2022.2.1 && \
+    rm -rf /var/lib/apt/lists/* 
+ENV PATH="/opt/nvidia/nsight-systems/2022.2.1/bin:${PATH}"
 
 RUN python3 -m pip install virtualenv
\ No newline at end of file
diff --git a/pytorch_blade/benchmark/TorchBench/backends.py b/pytorch_blade/benchmark/TorchBench/backends.py
deleted file mode 100644
index c71847a27e6..00000000000
--- a/pytorch_blade/benchmark/TorchBench/backends.py
+++ /dev/null
@@ -1,34 +0,0 @@
-# Copyright 2022 The BladeDISC Authors. All rights reserved.
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-# http://www.apache.org/licenses/LICENSE-2.0
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from torchdynamo.optimizations.backends import create_backend
-import torch
-
-
-@create_backend
-def blade_disc_compiler(subgraph):
-    import torch_blade
-    from torch_blade import optimize as blade_optimize
-
-    opt_disc_config = torch_blade.config.Config()
-    with opt_disc_config, torch.no_grad():
-        optimized_model = blade_optimize(
-            subgraph.model.eval(),
-            allow_tracing=True,
-            model_inputs=tuple(subgraph.example_inputs),
-        )
-
-    # with open(f'model.code.py', 'a') as writer:
-    #     writer.write(str(optimized_model.code))
-    # with open(f'model.graph.txt', 'a') as writer:
-    #     writer.write(str(optimized_model.graph))
-
-    return optimized_model
diff --git a/pytorch_blade/benchmark/TorchBench/blade_bench.py b/pytorch_blade/benchmark/TorchBench/blade_bench.py
deleted file mode 100644
index d75c69192cb..00000000000
--- a/pytorch_blade/benchmark/TorchBench/blade_bench.py
+++ /dev/null
@@ -1,32 +0,0 @@
-# Copyright 2022 The BladeDISC Authors. All rights reserved.
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-# http://www.apache.org/licenses/LICENSE-2.0
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import logging
-import warnings
-import sys
-import os
-from os.path import abspath
-
-original_dir = abspath(os.getcwd())
-sys.path.append(os.path.join(original_dir, "dynamo", "benchmarks"))
-
-from backends import blade_disc_compiler
-from dynamo.benchmarks.torchbench import TorchBenchmarkRunner
-from dynamo.benchmarks import common
-import torchdynamo
-
-# config log dir
-torchdynamo.config.base_dir = original_dir
-
-if __name__ == "__main__":
-    logging.basicConfig(level=logging.WARNING)
-    warnings.filterwarnings("ignore")
-    common.main(TorchBenchmarkRunner(), original_dir)
diff --git a/pytorch_blade/benchmark/TorchBench/requirements.txt b/pytorch_blade/benchmark/TorchBench/requirements.txt
index 7569b278890..b5628e076b2 100644
--- a/pytorch_blade/benchmark/TorchBench/requirements.txt
+++ b/pytorch_blade/benchmark/TorchBench/requirements.txt
@@ -4,4 +4,3 @@ torchvision==0.13.0+cu113
 torchaudio==0.12.0+cu113
 torchtext==0.13.0
 torchdynamo
-pycocotools==2.0.3


Test patch:
diff --git a/pytorch_blade/benchmark/TorchBench/test_torch_bench.sh b/pytorch_blade/benchmark/TorchBench/test_torch_bench.sh
index 5f41c21b798..e070449b73d 100755
--- a/pytorch_blade/benchmark/TorchBench/test_torch_bench.sh
+++ b/pytorch_blade/benchmark/TorchBench/test_torch_bench.sh
@@ -17,19 +17,20 @@ python3 -m pip install -q -r $script_dir/requirements.txt
 
 # setup for torchbenchmark
 benchmark_repo_dir=$HOME/.cache/torchbenchmark
-if [ -d $benchmark_repo_dir ]; then
-    rm -rf $benchmark_repo_dir
+if [ ! -d $benchmark_repo_dir ]; then
+    git clone -q https://github.com/pai-disc/torchbenchmark.git --recursive $benchmark_repo_dir
 fi
-git clone -q https://github.com/pai-disc/torchbenchmark.git --recursive $benchmark_repo_dir
 # CI git-lfs permission problems
 cd $benchmark_repo_dir && export HOME=$(pwd) && git lfs install --force
 git pull && git submodule update --init --recursive --depth 1 && python3 install.py --continue_on_fail
+# fix pycocotools after install
+python3 -m pip install -U numpy
 pushd $script_dir # pytorch_blade/benchmark/TorchBench
-# setup for torchdynamo
 ln -s $benchmark_repo_dir torchbenchmark
 
-# torchscript/torchdynamo frontend and disc backend
+# benchmark 
 TORCHBENCH_ATOL=1e-2 TORCHBENCH_RTOL=1e-2 python3 torchbenchmark/.github/scripts/run-config.py -c blade_bench.yaml -b ./torchbenchmark/ --output-dir .
 # results
+cat eval-cuda-fp16/summary.csv
 cat eval-cuda-fp32/summary.csv
 popd