Repository: Xilinx/inference-server (language: python)

Issue:
Use Google Benchmark for C++ benchmarks
Using [Google Benchmark](https://github.com/google/benchmark) to run the C++ benchmarks will help standardize the output and make it easier to collect and parse. The library is also related to GTest which we're already using.

Related to #140 


Discussion / hints:

Fix patch:
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index af47f2979..7072b5025 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -18,28 +18,61 @@ Contributing
 ============
 
 AMD Inference Server is in active development and many things need to be done so thanks for helping!
-You can contribute in a variety of ways.
 
-Ways to Contribute
+
+Ways to contribute
 ------------------
 
-* Raise issues to report bugs! If you find a bug, give us as much information as you can about your environment, hardware, steps to reproduce and relevant logs. If you can point to the code that's causing a problem, that helps a lot too!
-* Raise issues to suggest or request new features and changes to existing ones to improve functionality, performance and/or quality. Comment and vote up issues that you think should be prioritized.
-* Submit pull requests against active issues. See `Contributing Code`_ for more information.
+You can contribute in a variety of ways depending on your experience, time, and permissions.
+The easiest way to help out is react and vote up issues and discussions that are important to you.
 
-Contributing Code
-------------------
+Idea generation
+^^^^^^^^^^^^^^^
+
+If you have ideas on new things that may be needed or changing something, `start a discussion <https://github.com/Xilinx/inference-server/discussions/new?category=ideas>`_.
+New idea discussions don't need to be fully fleshed out or limited in scope.
+Instead, use it as a notepad for ideas that you have so they can be remembered and tracked.
+Having it as a discussion allows the community and maintainers to elaborate on the idea and determine if it's something of interest.
+
+Raise issues
+^^^^^^^^^^^^
+
+An issue can be a bug report, a new feature request, or a change to documentation.
+They're used to track concrete measurable tasks that need to get done, which means there should be a clear way to determine that the issue is resolved.
+If you find a bug, give us as much information as you can about your environment, hardware, steps to reproduce and relevant logs.
+If you can point to the code that's causing a problem, that helps a lot too!
+
+Issues may be raised independently from ideas if the scope is already well-defined.
+But ideas will also naturally result in new issues aimed at implementing the idea.
+These related issues aimed at implementing this idea should link back to the original idea for context.
+If an issue is better suited as a discussion before being formalized as an issue, it will be moved.
+
+Triage
+^^^^^^
+
+Managing and organizing the repository are tasks for those with appropriate permissions:
+
+* Issues that are better suited for discussions should be moved
+* Issues should be labeled appropriately to categorize them and marked active once approved
+* Projects should be used to track upcoming releases or deadlines
+* Issues should be added to the appropriate project(s) if they're committed to for it
 
-The suggested model of contributing new code is to pick an issue (or raise a new one) and comment about wanting to tackle it.
-This lets others know that there's active development on it. Then, you can fork this repository, make your additions and submit a pull request for merging.
-When sending code sign your work as described below.
+Raise pull requests
+^^^^^^^^^^^^^^^^^^^
+
+A pull request should be made against an active issue.
+An active issue indicates that it has gone through a discussion already and is approved for development.
+You can fork the repository, make your additions and raise a pull request.
+Remember to `sign your git commits <Sign your work>`_.
+The pull request will be reviewed by an owner and eventually approved for testing.
+If the automated testing passes, then it can be merged in.
 All code is licensed under the terms of the LICENSE file included in the repository.
 Your contribution will be accepted under the same license.
 
-Sign Your Work
-^^^^^^^^^^^^^^
+Sign your work
+""""""""""""""
 
-Please use the *Signed-off-by* line at the end of your patch which indicates that you accept the Developer Certificate of Origin (DCO) defined by https://developercertificate.org/ reproduced below:
+Please use the *Signed-off-by* line at the end of your patch which indicates that you accept the `Developer Certificate of Origin (DCO) <https://developercertificate.org/>`_, reproduced below:
 
 .. code-block:: text
 
@@ -82,7 +115,7 @@ Please use the *Signed-off-by* line at the end of your patch which indicates tha
         this project or the open source license(s) involved.
 
 
-Here is an example Signed-off-by line which indicates that the contributor accepts DCO:
+Here is an example commit message that indicates that the contributor accepts the DCO:
 
 .. code-block:: text
 
@@ -90,7 +123,16 @@ Here is an example Signed-off-by line which indicates that the contributor accep
 
     Signed-off-by: Jane Doe <jane.doe@example.com>
 
-Style Guide
+You can also add the sign-off statement automatically when committing with git:
+
+.. code-block:: console
+
+    $ git commit -s -m "This is my commit message"
+
+Consider signing your commit with GPG as well.
+You can see more information about commit signature verification on `Github <https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits>`_.
+
+Style guide
 -----------
 
 ``pre-commit`` is used to enforce style and is included in the development container.
@@ -103,6 +145,19 @@ Documentation
 The documentation for the AMD Inference Server is written in reStructuredText and is located in the ``docs/`` directory.
 If you are unfamiliar with reStructuredText, check out a `basic tutorial <https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_
 
+Headers
+"""""""
+
+Headers are denoted in reStructuredText with a series of punctuation characters at least as long as the title.
+While it does not enforce that a particular character denotes a particular hierarchy, you should use the following convention that matches the `Python convention <https://devguide.python.org/documentation/markup/#sections>`_:
+
+* ``#`` with overline, for parts
+* ``*`` with overline, for chapters
+* ``=``, for sections
+* ``-``, for subsections
+* ``^``, for subsubsections
+* ``"``, for paragraphs
+
 Admonitions
 """""""""""
 
@@ -117,16 +172,11 @@ While reStructuredText supports many types of admonitions, the following groups
 
 For maintaining visual consistency, admonitions in the same class should convey the same relative importance.
 
-.. list-table::
-    :header-rows: 1
-
-    * - Admonition
-      - Usage
-    * - Blue
-      - Contains useful non-essential information and does not suggest an action for the reader to take
-    * - Green
-      - Contains useful information or suggests an action for the reader to take
-    * - Yellow
-      - Contains important information or highlights unexpected side effects of events
-    * - Red
-      - Contains critical information
+========== =====
+Admonition Usage
+========== =====
+Blue       Contains useful non-essential information and does not suggest an action for the reader to take
+Green      Contains useful information or suggests an action for the reader to take
+Yellow     Contains important information or highlights unexpected side effects of events
+Red        Contains critical information
+========== =====
diff --git a/cmake/AddTest.cmake b/cmake/AddTest.cmake
index 55815df1d..24d81e874 100644
--- a/cmake/AddTest.cmake
+++ b/cmake/AddTest.cmake
@@ -66,6 +66,22 @@ function(amdinfer_add_unit_test test)
   _amdinfer_add_test(${test} "unit")
 endfunction()
 
+function(amdinfer_add_benchmark test)
+  amdinfer_get_test_target(target ${test} benchmark)
+
+  add_executable(${target} benchmark_${test}.cpp)
+  target_include_directories(
+    ${target} BEFORE PRIVATE ${AMDINFER_TEST_INCLUDE_DIRS}
+  )
+  target_include_directories(${target} AFTER PRIVATE ${AMDINFER_INCLUDE_DIRS})
+
+  target_link_libraries(
+    ${target} PRIVATE benchmark::benchmark benchmark::benchmark_main
+  )
+
+  set_target_options(${target})
+endfunction()
+
 function(amdinfer_add_system_tests tests)
   foreach(test ${tests})
     amdinfer_add_system_test(${test})
@@ -84,3 +100,16 @@ function(amdinfer_add_unit_tests tests tests_libs)
     target_link_libraries(${target} PRIVATE ${libs})
   endforeach()
 endfunction()
+
+function(amdinfer_add_benchmarks tests tests_libs)
+  foreach(test lib_str IN ZIP_LISTS tests tests_libs)
+
+    string(REPLACE " " "" libs_no_spaces ${lib_str})
+    string(REPLACE "~" ";" libs ${libs_no_spaces})
+
+    amdinfer_add_benchmark(${test})
+    amdinfer_get_test_target(target ${test} benchmark)
+
+    target_link_libraries(${target} PRIVATE ${libs})
+  endforeach()
+endfunction()
diff --git a/docker/template.dockerfile b/docker/template.dockerfile
index ba0b882e2..6dc223bcb 100644
--- a/docker/template.dockerfile
+++ b/docker/template.dockerfile
@@ -432,6 +432,16 @@ RUN wget -O half_2.2.0.zip https://sourceforge.net/projects/half/files/latest/do
     && mv half/include/half.hpp ${COPY_DIR}/usr/local/include/half \
     && cd /tmp && rm -fr /tmp/*
 
+# install google benchmark for benchmarking apps
+RUN wget --quiet https://github.com/google/benchmark/archive/refs/tags/v1.7.1.tar.gz \
+    && tar -xzf v1.7.1.tar.gz \
+    && cd benchmark-1.7.1 \
+    && cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DBENCHMARK_ENABLE_TESTING=OFF \
+    && cmake --build build --target install -- -j$(($(nproc) - 1)) \
+    && cat build/install_manifest.txt | xargs -i bash -c "if [ -f {} ]; then cp --parents -P {} ${COPY_DIR}; fi" \
+    && cat build/install_manifest.txt > ${MANIFESTS_DIR}/benchmark.txt \
+    && cd /tmp && rm -fr /tmp/*
+
 # install yaml-cpp for yaml parsing
 # RUN wget --quet https://github.com/jbeder/yaml-cpp/archive/refs/tags/yaml-cpp-0.7.0.tar.gz \
 #     && tar -xzf yaml-cpp-0.7.0.tar.gz \


Test patch:
diff --git a/tests/performance/CMakeLists.txt b/tests/performance/CMakeLists.txt
index c1d1bd4f5..5d8ce17e9 100644
--- a/tests/performance/CMakeLists.txt
+++ b/tests/performance/CMakeLists.txt
@@ -13,5 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+find_package(benchmark)
+
 add_subdirectory(batching)
 add_subdirectory(models)
diff --git a/tests/performance/batching/CMakeLists.txt b/tests/performance/batching/CMakeLists.txt
index a1a41cca7..14659fee9 100644
--- a/tests/performance/batching/CMakeLists.txt
+++ b/tests/performance/batching/CMakeLists.txt
@@ -24,4 +24,4 @@ list(
       ~$<TARGET_OBJECTS:native_internal>~timer"
 )
 
-amdinfer_add_unit_tests("${tests}" "${tests_libs}")
+amdinfer_add_benchmarks("${tests}" "${tests_libs}")
diff --git a/tests/performance/batching/test_soft.cpp b/tests/performance/batching/benchmark_soft.cpp
similarity index 58%
rename from tests/performance/batching/test_soft.cpp
rename to tests/performance/batching/benchmark_soft.cpp
index bc05ffe44..c8de44559 100644
--- a/tests/performance/batching/test_soft.cpp
+++ b/tests/performance/batching/benchmark_soft.cpp
@@ -13,6 +13,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+#include <benchmark/benchmark.h>
+
 #include <array>     // for array
 #include <chrono>    // for duration, operator-
 #include <cstddef>   // for size_t
@@ -37,61 +39,36 @@
 #include "amdinfer/core/worker_info.hpp"         // for WorkerInfo
 #include "amdinfer/declarations.hpp"             // for BufferPtrs
 #include "amdinfer/observation/logging.hpp"      // for LogOptions, initLogger
-#include "gtest/gtest.h"                         // for UnitTest, EXPECT_NEAR
+#include "gtest/gtest.h"
 
 namespace amdinfer {
 
 constexpr auto kTimeoutMs = 1000;  // timeout in ms
-// timeout in us with safety factor
-constexpr auto kTimeoutUs = kTimeoutMs * 1000 * 5;
 
-class PerfSoftBatcherFixture
-  : public testing::TestWithParam<std::tuple<int, int, int>> {
+class PerfSoftBatcherFixture : public ::benchmark::Fixture {
  public:
-  int enqueue() {
-    auto start_time = std::chrono::high_resolution_clock::now();
-    std::chrono::nanoseconds duration = std::chrono::nanoseconds(0);
-
-    int count = 0;
-
-    do {
+  void enqueue(int count) {
+    for (auto i = 0; i < count; ++i) {
       auto req = std::make_unique<CppNativeApi>(this->request_);
       batcher_->enqueue(std::move(req));
-      count++;
-
-      auto end_time = std::chrono::high_resolution_clock::now();
-      duration = end_time - start_time;
-    } while (duration < std::chrono::nanoseconds(std::chrono::seconds(1)));
-
-    return count;
+    }
   }
 
-  int dequeue() {
-    const auto [batch_size, num_buffers, delay] = GetParam();
-    bool valid_read = false;
-    int count = 0;
-    do {
+  void dequeue(int count, int delay) {
+    for (auto i = 0; i < count; ++i) {
       BatchPtr batch;
-      valid_read =
-        batcher_->getOutputQueue()->wait_dequeue_timed(batch, kTimeoutUs);
+      batcher_->getOutputQueue()->wait_dequeue(batch);
       std::this_thread::sleep_for(std::chrono::microseconds(delay));
-      if (valid_read) {
-        count++;
-      }
-    } while (valid_read);
-    return count;
+    }
   }
 
- protected:
-  void SetUp() override {
-    const auto [batch_size, num_buffers, delay] = GetParam();
+  void SetUp(const ::benchmark::State& state) override {
+    auto batch_size = static_cast<int>(state.range(0));
+    auto buffer_num = static_cast<size_t>(state.range(1));
 
-    const auto buffer_num = static_cast<size_t>(num_buffers);
     const auto data_shape = {1UL, 2UL, 50UL};
     const auto data_size = 100;
 
-    this->data_size_ = data_size;
-
     ParameterMap parameters;
     parameters.put("timeout", kTimeoutMs);
     parameters.put("batch_size", batch_size);
@@ -124,7 +101,7 @@ class PerfSoftBatcherFixture
 
     data_.resize(data_size);
     for (auto i = 0; i < data_size; i++) {
-      data_[i] = i;
+      data_[i] = static_cast<uint8_t>(i);
     }
 
     this->request_ = InferenceRequest();
@@ -132,54 +109,52 @@ class PerfSoftBatcherFixture
                                   DataType::Uint8);
   }
 
-  void TearDown() override {
+  void TearDown([[maybe_unused]] const ::benchmark::State& state) override {
     batcher_->enqueue(nullptr);
     batcher_->end();
   }
 
-  int data_size_ = 0;
+ private:
   std::vector<uint8_t> data_;
   InferenceRequest request_;
   std::optional<WorkerInfo> worker_;
   std::optional<SoftBatcher> batcher_;
 };
 
-// @pytest.mark.perf(group="batcher")
-TEST_P(PerfSoftBatcherFixture, BasicBatching) {  // NOLINT
-  const auto [batch_size, num_buffers, delay] = GetParam();
-  auto start_time = std::chrono::high_resolution_clock::now();
-
-  auto enqueue =
-    std::async(std::launch::async, &PerfSoftBatcherFixture::enqueue, this);
-  auto dequeue =
-    std::async(std::launch::async, &PerfSoftBatcherFixture::dequeue, this);
+BENCHMARK_DEFINE_F(PerfSoftBatcherFixture, BasicBatching)
+(benchmark::State& st) {  // NOLINT
 
-  auto enqueue_count = enqueue.get();
-  auto dequeue_count = dequeue.get();
+  const auto enqueue_count = 1000;
 
-  auto end_time = std::chrono::high_resolution_clock::now();
-  std::chrono::duration<double> duration = end_time - start_time;
-  auto throughput = static_cast<double>(enqueue_count) / duration.count();
+  for ([[maybe_unused]] auto _ : st) {
+    auto batch_size = static_cast<int>(st.range(0));
+    const auto dequeue_count = static_cast<int>(
+      std::ceil(enqueue_count / static_cast<double>(batch_size)));
+    auto enqueue =
+      std::async(std::launch::async, &PerfSoftBatcherFixture::enqueue, this,
+                 enqueue_count);
+    auto dequeue =
+      std::async(std::launch::async, &PerfSoftBatcherFixture::dequeue, this,
+                 dequeue_count, st.range(2));
 
-  EXPECT_NEAR(enqueue_count / float(batch_size), dequeue_count,
-              enqueue_count * 0.01);
-
-  std::cerr << "Enqueue count: " << enqueue_count << std::endl;
-  std::cerr << "Dequeue count: " << dequeue_count << std::endl;
-  std::cerr << "Throughput (req/s): " << throughput << std::endl;
+    enqueue.get();
+    dequeue.get();
+  }
 }
 
-const std::array kBatchSizes{1, 2, 4};
+const std::initializer_list<int64_t> kBatchSizes{1, 2, 4};
 
-const std::array kBufferNums{1, 10, 100};
+const std::initializer_list<int64_t> kBufferNums{1, 10, 100};
 
 // delay after dequeue per request in microseconds
-const std::array kWorkDelay{0, 1, 10};
+const std::initializer_list<int64_t> kWorkDelay{0, 1, 10};
+
+// NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
+BENCHMARK_REGISTER_F(PerfSoftBatcherFixture, BasicBatching)
+  ->ArgsProduct({kBatchSizes, kBufferNums, kWorkDelay})
+  ->Unit(benchmark::kMillisecond);
 
-// NOLINTNEXTLINE(cert-err58-cpp)
-INSTANTIATE_TEST_SUITE_P(Datatypes, PerfSoftBatcherFixture,
-                         testing::Combine(testing::ValuesIn(kBatchSizes),
-                                          testing::ValuesIn(kBufferNums),
-                                          testing::ValuesIn(kWorkDelay)));
+// NOLINTNEXTLINE
+BENCHMARK_MAIN();
 
 }  // namespace amdinfer
diff --git a/tests/performance/models/CMakeLists.txt b/tests/performance/models/CMakeLists.txt
index 672662d2c..42a6c697c 100644
--- a/tests/performance/models/CMakeLists.txt
+++ b/tests/performance/models/CMakeLists.txt
@@ -15,9 +15,10 @@
 list(APPEND tests resnet50)
 
 foreach(test ${tests})
-  amdinfer_add_system_test(${test})
-  amdinfer_get_test_target(target ${test})
+  amdinfer_add_benchmark(${test})
+  amdinfer_get_test_target(target ${test} benchmark)
   target_link_libraries(
-    ${target} PRIVATE opencv_imgcodecs opencv_imgproc opencv_core
+    ${target} PRIVATE opencv_imgcodecs opencv_imgproc opencv_core amdinfer
+                      testing
   )
 endforeach()
diff --git a/tests/performance/models/test_resnet50.cpp b/tests/performance/models/benchmark_resnet50.cpp
similarity index 60%
rename from tests/performance/models/test_resnet50.cpp
rename to tests/performance/models/benchmark_resnet50.cpp
index e779eb19b..09a2648f9 100644
--- a/tests/performance/models/test_resnet50.cpp
+++ b/tests/performance/models/benchmark_resnet50.cpp
@@ -14,9 +14,11 @@
 
 /**
  * @file
- * @brief Performance testing for the native client
+ * @brief Performance testing for resnet50
  */
 
+#include <benchmark/benchmark.h>
+
 #include <algorithm>            // for copy, max
 #include <array>                // for array
 #include <cassert>              // for assert
@@ -32,13 +34,13 @@
 #include <string>               // for string, allocator
 #include <thread>               // for sleep_for
 #include <tuple>                // for tuple, _Swallow_as...
+#include <utility>              // for forward
 #include <variant>              // for variant, visit
 #include <vector>               // for vector
 
 #include "amdinfer/amdinfer.hpp"                   // for InferenceRequest
 #include "amdinfer/pre_post/image_preprocess.hpp"  // for ImagePreprocessOpt...
 #include "amdinfer/testing/get_path_to_asset.hpp"  // for getPathToAsset
-#include "amdinfer/testing/gtest_fixtures.hpp"     // for ParamIteratorInter...
 
 namespace fs = std::filesystem;
 
@@ -53,6 +55,9 @@ struct Config {
   }
 };
 
+const std::array<Config, 4> kConfigs = {Config{1, 10}, Config{2, 20},
+                                        Config{20, 40}, Config{40, 40}};
+
 template <typename T, int kChannels>
 using ImagePreprocessOptions =
   amdinfer::pre_post::ImagePreprocessOptions<T, kChannels>;
@@ -72,7 +77,8 @@ struct PtzendnnWorker : public Workers {
     assert(root != nullptr);
     extension = "ptzendnn";
     name = "ptzendnn";
-    graph = fs::path{root} / "external/pytorch_models/resnet50_pretrained.pt";
+    graph =
+      fs::path{root} / "external/artifacts/pytorch/resnet50_pretrained.pt";
 
     parameters.put("model", graph.string());
 
@@ -100,7 +106,7 @@ struct TfzendnnWorker : public Workers {
     extension = "tfzendnn";
     name = "tfzendnn";
     graph = fs::path{root} /
-            "external/tensorflow_models/resnet_v1_50_baseline_6.96B_922.pb";
+            "external/artifacts/tensorflow/resnet_v1_50_baseline_6.96B_922.pb";
 
     // arbitrarily set to 64
     const int inter_op = 64;
@@ -119,12 +125,12 @@ struct TfzendnnWorker : public Workers {
   }
 };
 
-class PerfModelsResnetBaseFixture
-  : public BaseFixtureWithParams<std::tuple<Config, const Workers*>> {};
-class PerfModelsResnetHttpFixture
-  : public HttpFixtureWithParams<std::tuple<Config, const Workers*>> {};
-class PerfModelsResnetGrpcFixture
-  : public GrpcFixtureWithParams<std::tuple<Config, const Workers*>> {};
+// NOLINTNEXTLINE(cert-err58-cpp)
+const PtzendnnWorker kPtZendnn{};
+// NOLINTNEXTLINE(cert-err58-cpp)
+const TfzendnnWorker kTfZendnn{};
+
+const std::array<const Workers*, 2> kWorkers{&kPtZendnn, &kTfZendnn};
 
 template <class... Fs>
 struct Overload : Fs... {
@@ -133,11 +139,15 @@ struct Overload : Fs... {
 template <class... Fs>
 Overload(Fs...) -> Overload<Fs...>;
 
-void test(amdinfer::Client* client, const Config& config,
-          const Workers* worker) {
-  if (!amdinfer::serverHasExtension(client, worker->extension)) {
-    GTEST_SKIP() << worker->extension << " support required but not found.\n";
-  }
+template <class... Args>
+void resnet50(benchmark::State& st, Args&&... args) {
+  auto args_tuple = std::make_tuple(std::forward<Args>(args)...);
+  int client_index = std::get<0>(args_tuple);
+
+  auto config_index = st.range(0);
+  auto worker_index = st.range(1);
+  const auto& config = kConfigs.at(config_index);
+  const auto* worker = kWorkers.at(worker_index);
 
   const auto image_location =
     amdinfer::getPathToAsset("asset_dog-3619020_640.jpg");
@@ -147,7 +157,6 @@ void test(amdinfer::Client* client, const Config& config,
   const auto batch_size = config.batch_size;
 
   const auto warmup_requests = 4;
-  const auto num_requests = config.requests;
 
   const auto& name = worker->name;
 
@@ -156,17 +165,38 @@ void test(amdinfer::Client* client, const Config& config,
   parameters.put("output_classes", output_classes);
   parameters.put("batch_size", batch_size);
 
+  const auto default_http_port = 8998;
+  const auto default_grpc_port = 50'051;
+
+  amdinfer::Server server;
+  std::unique_ptr<amdinfer::Client> client;
+  if (client_index == 0) {
+    client = std::make_unique<amdinfer::NativeClient>(&server);
+  } else if (client_index == 1) {
+    server.startHttp(default_http_port);
+    client = std::make_unique<amdinfer::HttpClient>("http://127.0.0.1:8998");
+  } else if (client_index == 2) {
+    server.startGrpc(default_grpc_port);
+    client = std::make_unique<amdinfer::GrpcClient>("127.0.0.1:50051");
+  } else {
+    st.SkipWithError("Unknown client index passed");
+    return;
+  }
+
+  amdinfer::waitUntilServerReady(client.get());
+
   auto endpoint = client->workerLoad(name, &parameters);
-  EXPECT_EQ(endpoint, name);
+  assert(endpoint == name);
 
   std::vector<std::string> paths{image_location};
   const auto& options = worker->preprocessing;
 
   amdinfer::InferenceRequest request;
+  std::vector<std::vector<float>> images;
   std::visit(
     Overload{
       [&](const ImagePreprocessOptions<float, channels>& opts) {
-        auto images = amdinfer::pre_post::imagePreprocess(paths, opts);
+        images = amdinfer::pre_post::imagePreprocess(paths, opts);
         request.addInputTensor(images[0].data(),
                                {channels, input_size, input_size},
                                amdinfer::DataType::Fp32);
@@ -179,90 +209,46 @@ void test(amdinfer::Client* client, const Config& config,
     std::ignore = client->modelInfer(endpoint, request);
   }
 
-  auto start = std::chrono::high_resolution_clock::now();
-  for (auto i = 0; i < num_requests; ++i) {
-    auto response = client->modelInfer(endpoint, request);
-    EXPECT_FALSE(response.isError());
+  for (auto _ : st) {
+    if (!amdinfer::serverHasExtension(client.get(), worker->extension)) {
+      std::string error = worker->extension + " support required but not found";
+      st.SkipWithError(error.c_str());
+      return;
+    }
+
+    auto num_requests = config.requests;
+    for (auto i = 0; i < num_requests; ++i) {
+      auto response = client->modelInfer(endpoint, request);
+      assert(!response.isError());
+    }
   }
-  auto stop = std::chrono::high_resolution_clock::now();
-  std::chrono::duration<double, std::milli> duration = stop - start;
-  std::cout << "-----\n";
-  std::cout << "sync time taken for " << num_requests
-            << " images with batch size " << batch_size << " on " << name
-            << ": " << duration.count() << "ms" << std::endl;
-  std::cout << "-----\n";
-
-  std::vector<amdinfer::InferenceRequest> requests;
-  requests.reserve(num_requests);
-  for (auto i = 0; i < num_requests; ++i) {
-    requests.push_back(request);
-  }
-  start = std::chrono::high_resolution_clock::now();
-  auto responses = amdinfer::inferAsyncOrdered(client, endpoint, requests);
-  stop = std::chrono::high_resolution_clock::now();
-  duration = stop - start;
-  std::cout << "-----\n";
-  std::cout << "async time taken for " << num_requests
-            << " images with batch size " << batch_size << " on " << name
-            << ": " << duration.count() << "ms" << std::endl;
-  std::cout << "-----\n";
-
   client->workerUnload(endpoint);
   while (client->modelReady(endpoint)) {
     std::this_thread::sleep_for(std::chrono::seconds(1));
   }
 }
 
-const std::array<Config, 4> kConfigs = {Config{1, 10}, Config{2, 20},
-                                        Config{20, 40}, Config{40, 40}};
-
-// NOLINTNEXTLINE(cert-err58-cpp)
-const PtzendnnWorker kPtZendnn{};
-// NOLINTNEXTLINE(cert-err58-cpp)
-const TfzendnnWorker kTfZendnn{};
-
-const std::array<const Workers*, 1> kWorkers = {
-  // TODO(amuralee): why does TFzendnn slow down dramatically if included
-  // together
-  //  &kPtZendnn, &kTfZendnn
-  &kTfZendnn};
-
-#ifdef AMDINFER_ENABLE_GRPC
-
-// @pytest.mark.perf(group="clients")
-TEST_P(PerfModelsResnetGrpcFixture, ModelInfer) {  // NOLINT
-  const auto& [config, worker] = GetParam();
-  test(client_.get(), config, worker);
-}
-
 // NOLINTNEXTLINE(cert-err58-cpp)
-INSTANTIATE_TEST_SUITE_P(PerfModelsResnetGrpc, PerfModelsResnetGrpcFixture,
-                         testing::Combine(testing::ValuesIn(kConfigs),
-                                          testing::ValuesIn(kWorkers)));
-#endif
-
-// @pytest.mark.perf(group="clients")
-TEST_P(PerfModelsResnetBaseFixture, ModelInfer) {  // NOLINT
-  amdinfer::NativeClient client(&server_);
-  const auto& [config, worker] = GetParam();
-  test(&client, config, worker);
-}
-
-// NOLINTNEXTLINE(cert-err58-cpp)
-INSTANTIATE_TEST_SUITE_P(PerfModelsResnetBase, PerfModelsResnetBaseFixture,
-                         testing::Combine(testing::ValuesIn(kConfigs),
-                                          testing::ValuesIn(kWorkers)));
+const std::initializer_list<std::vector<int64_t>> kRange{
+  benchmark::CreateDenseRange(0, kConfigs.size() - 1, 1),
+  benchmark::CreateDenseRange(0, kWorkers.size() - 1, 1)};
 
+// NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
+BENCHMARK_CAPTURE(resnet50, Native, 0)
+  ->ArgsProduct(kRange)
+  ->Unit(benchmark::kMillisecond);
 #ifdef AMDINFER_ENABLE_HTTP
-// @pytest.mark.perf(group="clients")
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
-TEST_P(PerfModelsResnetHttpFixture, ModelInfer) {  // NOLINT
-  const auto& [config, worker] = GetParam();
-  test(client_.get(), config, worker);
-}
-
-// NOLINTNEXTLINE(cert-err58-cpp)
-INSTANTIATE_TEST_SUITE_P(PerfModelsResnetHttp, PerfModelsResnetHttpFixture,
-                         testing::Combine(testing::ValuesIn(kConfigs),
-                                          testing::ValuesIn(kWorkers)));
+BENCHMARK_CAPTURE(resnet50, HTTP, 1)
+  ->ArgsProduct(kRange)
+  ->Unit(benchmark::kMillisecond);
 #endif
+#ifdef AMDINFER_ENABLE_GRPC
+// NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
+BENCHMARK_CAPTURE(resnet50, gRPC, 2)
+  ->ArgsProduct(kRange)
+  ->Unit(benchmark::kMillisecond);
+#endif
+
+// NOLINTNEXTLINE
+BENCHMARK_MAIN();
diff --git a/tests/workers/test_echo.py b/tests/workers/test_echo.py
index e58572e60..363034a3b 100644
--- a/tests/workers/test_echo.py
+++ b/tests/workers/test_echo.py
@@ -196,9 +196,10 @@ def test_echo_4(self):
     @pytest.mark.benchmark(group="echo")
     def test_benchmark_echo_0(self, benchmark):
         request = self.construct_request(False, False, False)
+        model, parameters = self.get_config()
         options = {
-            "model": self.model,
-            "parameters": self.parameters,
+            "model": model,
+            "parameters": parameters,
             "type": "rest (pytest)",
             "config": "N/A",
         }