Repository: Xilinx/inference-server (language: python)

Issue:
Add model chaining
Chaining should let a user define a pipeline of steps that could include pre- and post-processing steps to run end-to-end inference or send the output of one model to another.


Discussion / hints:

Fix patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 07e5b7f49..cdddba642 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -30,6 +30,8 @@ Added
 
 * An example MLPerf app using the inference server API (:pr:`129`)
 * Google Benchmark for writing performance-tracking tests (:pr:`147`)
+* Custom memory storage classes in the memory pool (:pr:`166`)
+* C++ worker for executing C++ "models" (:pr:`172`)
 
 Changed
 ^^^^^^^
@@ -38,6 +40,10 @@ Changed
 * Require the server as an argument for creating the NativeClient (:pr:`125`)
 * Use a global memory pool to allocate memory for incoming requests (:pr:`149`)
 * Resolve the request at the incoming server rather than the batcher (:pr:`164`)
+* Add flags to run container-based tests in parallel (:pr:`168`)
+* Bump up to Vitis AI 3.0 (:pr:`169`)
+* Refactor inference request objects and tensors (:pr:`172`)
+* Use const references throughout for ParameterMap (:pr:`172`)
 
 Deprecated
 ^^^^^^^^^^
@@ -49,6 +55,9 @@ Fixed
 ^^^^^
 
 * Use the right unit for batcher timeout (:pr:`129`)
+* Don't call ``next`` and ``prev`` on end iterators (:pr:`166`)
+* Use the right package name for ``g++`` in CentOS (:pr:`168`)
+* Fix building with different CMake options (:pr:`170`)
 
 Security
 ^^^^^^^^
diff --git a/docker/generate.py b/docker/generate.py
index e1767e00d..24d94a6de 100644
--- a/docker/generate.py
+++ b/docker/generate.py
@@ -707,6 +707,7 @@ def install_python_packages():
                 exhale \\
                 sphinx-tabs \\
                 sphinxcontrib-openapi \\
+                sphinxcontrib-jquery \\
                 # install linting tools
                 black \\
                 cpplint \\
diff --git a/docs/conf.py b/docs/conf.py
index 091997541..b2a3e3f8f 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -50,6 +50,7 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
+    "sphinxcontrib.jquery",
     "breathe",
     "exhale",
     # adds argparse directive to parse CLIs
diff --git a/docs/dependencies.rst b/docs/dependencies.rst
index 1c403727d..54c03a342 100644
--- a/docs/dependencies.rst
+++ b/docs/dependencies.rst
@@ -140,6 +140,7 @@ The following packages (and any dependencies) are installed from the Python Pack
     :pypiPackages:`sphinx-issues`,latest,MIT,Sphinx plugin for links to the project's Github issue tracker\ :superscript:`d 0`
     :pypiPackages:`sphinx-tabs`,latest,MIT,Sphinx plugin to create tabs\ :superscript:`d 0`
     :pypiPackages:`sphinxcontrib-confluencebuilder`,latest,BSD-2,Sphinx plugin to export documentation to Confluence\ :superscript:`d 0`
+    :pypiPackages:`sphinxcontrib-jquery`,latest,BSD-0,Sphinx plugin to add JQuery\ :superscript:`d 0`
     :pypiPackages:`sphinxcontrib-openapi`,latest,BSD-2,Sphinx plugin to build OpenAPI docs\ :superscript:`d 0`
     :pypiPackages:`wheel`,latest,MIT,Support wheels for Python packages\ :superscript:`d 0`
 
diff --git a/docs/hello_world_echo.rst b/docs/hello_world_echo.rst
index 6a578daf6..ba03a10ed 100644
--- a/docs/hello_world_echo.rst
+++ b/docs/hello_world_echo.rst
@@ -21,12 +21,17 @@ Hello World - Python
 
 AMD Inference Server's Python API allows you to start the server and send requests to it from Python.
 This example walks you through how to start the server and use Python to send requests to a simple model.
-This example and script are intended to run in the development container.
 The complete script used here is available in the :amdinferTree:`repository <examples/hello_world>`.
 
 Import the library
 ------------------
 
+You can install the Python client library with ``pip``:
+
+.. code-block:: console
+
+    $ pip install amdinfer
+
 In the development container, the Python library is automatically built and installed as part of the CMake build process.
 
 .. literalinclude:: ../examples/hello_world/echo.py
@@ -67,7 +72,8 @@ Workers are started as threads in AMD Inference Server and have a defined lifecy
 Before making an inference request to a worker, it must first be loaded.
 Loading a worker returns an endpoint that the client should use for future operations.
 
-This worker, Echo, is a simple example worker that accepts an integer as input, adds one to it and returns the sum.
+This simple worker, "cplusplus", is a worker that accepts a C++-based "model" which is specified in the parameters.
+This model, "echo" is a simple test model that accepts an integer as input, adds one to it and returns the sum.
 
 .. literalinclude:: ../examples/hello_world/echo.py
     :start-after: +load worker
@@ -79,7 +85,7 @@ Inference
 
 Once the worker is ready, you can make an inference request to it.
 To do so, first construct a request.
-We construct a request that contains an integer and send it to AMD Inference Server.
+We construct a request that contains an integer and send it to the server.
 
 .. literalinclude:: ../examples/hello_world/echo.py
     :start-after: +inference
@@ -103,7 +109,7 @@ After making the inference, you should check what the response was.
 First, make sure the inference didn't fail by checking if the response is erroneous.
 Then, you can get the outputs and examine them.
 The format and contents of the output data will depend on the worker and model used for inference.
-In this case, the Echo worker returns a single output tensor back with one integer that should be one larger than what was sent.
+In this case, the worker returns a single output tensor back with one integer that should be one larger than what was sent.
 The assertions here check these cases.
 
 .. literalinclude:: ../examples/hello_world/echo.py
@@ -115,9 +121,13 @@ Clean up
 --------
 
 Workers that are loaded in AMD Inference Server will persist until the server shuts down or they're explicitly unloaded.
-While it's not shown here, the Python API provides an ``unload()`` method for this purpose.
 As the script ends, the Server object's destructor will clean up any active workers on the server and it will shut down
 
+.. literalinclude:: ../examples/hello_world/echo.py
+    :start-after: +unload
+    :end-before: -unload
+    :dedent: 4
+
 Next steps
 ----------
 
diff --git a/docs/quickstart_inference.rst b/docs/quickstart_inference.rst
index c5d7f912d..3fa66a22e 100644
--- a/docs/quickstart_inference.rst
+++ b/docs/quickstart_inference.rst
@@ -78,6 +78,7 @@ These examples assume a ResNet50 model trained on the ImageNet dataset is availa
 
     $ wget :amdinferRawFull:`examples/resnet50/imagenet_classes.txt`
     $ wget :amdinferRawFull:`tests/assets/dog-3619020_640.jpg`
+    $ wget :amdinferRawFull:`examples/resnet50/resnet.py`
 
 .. tabs::
 
diff --git a/examples/hello_world/README.rst b/examples/hello_world/README.rst
index 8327cda2a..776f9a54e 100644
--- a/examples/hello_world/README.rst
+++ b/examples/hello_world/README.rst
@@ -21,5 +21,5 @@ These examples demonstrate how to perform some basic tasks with the Inference Se
 echo.py
 ^^^^^^^
 
-This example describes making a simple inference to the "echo" worker on the Inference Server.
+This example describes making a simple inference to a C++-based worker running a trivial model on the Inference Server.
 It shows you how to start the server from Python, construct an HTTP/REST client, make a request, and process the results.
diff --git a/examples/hello_world/echo.py b/examples/hello_world/echo.py
index 3f08fb565..c960be712 100644
--- a/examples/hello_world/echo.py
+++ b/examples/hello_world/echo.py
@@ -20,8 +20,6 @@
 """
 
 # +imports
-from time import sleep
-
 import numpy as np
 
 import amdinfer
@@ -43,7 +41,7 @@ def make_request(data):
     request = amdinfer.InferenceRequest()
     # each request has one or more input tensors, depending on the worker/model that's going to process it
     input_0 = amdinfer.InferenceRequestInput()
-    input_0.name = f"input0"
+    input_0.name = "input0"
     input_0.setUint32Data(np.array([data], np.uint32))
     input_0.datatype = amdinfer.DataType.UINT32
     input_0.shape = [1]
@@ -69,8 +67,10 @@ def main():
     # -start server:
     print("Server ready!")
 
-    # +load worker: load the Echo worker which accepts a number, adds 1, and returns the sum
-    endpoint = client.workerLoad("echo")
+    # +load worker: load the worker with a model which accepts a number, adds 1, and returns the sum
+    parameters = amdinfer.ParameterMap()
+    parameters.put("model", "echo")
+    endpoint = client.workerLoad(client, "cplusplus", parameters)
     amdinfer.waitUntilModelReady(client, endpoint)
     # -load worker
     print("Model ready!")
@@ -93,6 +93,10 @@ def main():
     # -validate
     print("Results validated!")
 
+    # +unload: unload the worker
+    client.workerUnload(endpoint)
+    # -unload
+
 
 if __name__ == "__main__":
     main()
diff --git a/include/amdinfer/clients/client.hpp b/include/amdinfer/clients/client.hpp
index acb2ac3d3..acebcdd11 100644
--- a/include/amdinfer/clients/client.hpp
+++ b/include/amdinfer/clients/client.hpp
@@ -32,6 +32,7 @@
 namespace amdinfer {
 
 class ParameterMap;
+class Endpoint;
 
 /**
  * @brief The base Client class defines the set of methods that all client
@@ -180,6 +181,35 @@ void waitUntilServerReady(const Client* client);
  * @param model the model/worker to wait for
  */
 void waitUntilModelReady(const Client* client, const std::string& model);
+/**
+ * @brief Blocks until the named model/worker is not ready
+ *
+ * @param client a pointer to a client object
+ * @param model the model/worker to wait for
+ */
+void waitUntilModelNotReady(const Client* client, const std::string& model);
+
+/**
+ * @brief Load an ensemble - a chain of connected workers. This implementation
+ * uses the simplest case where the ensemble is a single linear graph.
+ *
+ * @param client a pointer to a client object
+ * @param workers the list of workers to connect
+ * @param parameters the list of parameters corresponding to each worker
+ * @return std::vector<std::string> the endpoints for each loaded worker
+ */
+std::vector<std::string> loadEnsemble(const Client* client,
+                                      std::vector<std::string> workers,
+                                      std::vector<ParameterMap> parameters);
+
+/**
+ * @brief Unload a list of models. This list may be from an ensemble or
+ * individually loaded workers or models.
+ *
+ * @param client a pointer to a client object
+ * @param models a list of models to unload
+ */
+void unloadModels(const Client* client, const std::vector<std::string>& models);
 
 /**
  * @brief Makes inference requests in parallel to the specified model. All
diff --git a/include/amdinfer/core/data_types.hpp b/include/amdinfer/core/data_types.hpp
index cab2a1931..6c9844a08 100644
--- a/include/amdinfer/core/data_types.hpp
+++ b/include/amdinfer/core/data_types.hpp
@@ -142,7 +142,7 @@ class DataType {
       case DataType::Fp64:
         return sizeof(double);
       case DataType::String:
-        return sizeof(std::string);
+        return sizeof(char);
       default:
         throw invalid_argument("Unknown datatype passed");
     }
diff --git a/include/amdinfer/core/inference_request.hpp b/include/amdinfer/core/inference_request.hpp
index 68a810e64..8a205752a 100644
--- a/include/amdinfer/core/inference_request.hpp
+++ b/include/amdinfer/core/inference_request.hpp
@@ -38,8 +38,6 @@ class InferenceRequestInput : public InferenceTensor {
   /// Constructs a new InferenceRequestInput object
   InferenceRequestInput();
   /// Constructs a new InferenceRequestInput object
-  explicit InferenceRequestInput(const InferenceTensor &tensor);
-  /// Constructs a new InferenceRequestInput object
   explicit InferenceRequestInput(const Tensor &tensor);
 
   /**
@@ -137,6 +135,8 @@ class InferenceRequest {
   // Construct a new InferenceRequest object
   InferenceRequest() = default;
 
+  InferenceRequestPtr propagate();
+
   /**
    * @brief Sets the request's callback function used by the last worker to
    * respond back to the client
diff --git a/include/amdinfer/core/parameters.hpp b/include/amdinfer/core/parameters.hpp
index d4d6c6e85..0f4433b46 100644
--- a/include/amdinfer/core/parameters.hpp
+++ b/include/amdinfer/core/parameters.hpp
@@ -47,42 +47,39 @@ class ParameterMap : public Serializable {
   using ConstIterator = Container::const_iterator;
 
  public:
+  ParameterMap() = default;
   /**
-   * @brief Puts in a key-value pair
+   * @brief Construct a new ParameterMap object with initial values. The sizes
+   * of the keys and values vectors must match.
    *
-   * @param key key used to store and retrieve the value
-   * @param value value to store
-   */
-  void put(const std::string &key, bool value);
-  /**
-   * @brief Puts in a key-value pair
+   * @details Until C++20, passing const char* to this constructor will convert
+   * it to a bool instead of a string. Explicitly convert any string literals to
+   * a string before passing them to this constructor.
    *
-   * @param key key used to store and retrieve the value
-   * @param value value to store
+   * @param keys
+   * @param values
    */
-  void put(const std::string &key, double value);
+  ParameterMap(const std::vector<std::string> &keys,
+               const std::vector<Parameter> &values);
+
   /**
-   * @brief Puts in a key-value pair
+   * @brief Put in a key-value pair
    *
    * @param key key used to store and retrieve the value
    * @param value value to store
    */
-  void put(const std::string &key, int32_t value);
+  void put(const std::string &key, Parameter value);
   /**
-   * @brief Puts in a key-value pair
+   * @brief Put in a key-value pair
    *
-   * @param key key used to store and retrieve the value
-   * @param value value to store
-   */
-  void put(const std::string &key, const std::string &value);
-  /**
-   * @brief Puts in a key-value pair
+   * @details This overload is needed because C++ converts const char* to bool
+   * instead of string when both types are present in the variant. This behavior
+   * has been fixed in C++20.
    *
-   * @param key key used to store and retrieve the value
-   * @param value value to store
+   * @param key
+   * @param value
    */
   void put(const std::string &key, const char *value);
-
   /**
    * @brief Get the named parameter
    *
diff --git a/src/amdinfer/CMakeLists.txt b/src/amdinfer/CMakeLists.txt
index c2596e38a..8620450dc 100644
--- a/src/amdinfer/CMakeLists.txt
+++ b/src/amdinfer/CMakeLists.txt
@@ -47,6 +47,10 @@ foreach(lib ${output_libraries})
     ${lib} PROPERTIES VERSION ${amdinfer_VERSION}
                       SOVERSION ${amdinfer_VERSION_MAJOR}
   )
+
+  set_target_properties(
+    ${lib} PROPERTIES BUILD_RPATH ${PROJECT_BINARY_DIR}/src/amdinfer/workers
+  )
 endforeach()
 
 # the compiled server binary
diff --git a/src/amdinfer/batching/batch.cpp b/src/amdinfer/batching/batch.cpp
index 1f55b3186..99f39de29 100644
--- a/src/amdinfer/batching/batch.cpp
+++ b/src/amdinfer/batching/batch.cpp
@@ -28,29 +28,26 @@ namespace amdinfer {
 
 void Batch::addRequest(InferenceRequestPtr request) {
   requests_.push_back(std::move(request));
+  // models_.emplace_back();
 }
 
-std::vector<Buffer*> Batch::getRawInputBuffers() const {
-  std::vector<Buffer*> buffers;
-  buffers.reserve(input_buffers_.size());
-  for (const auto& buffer : input_buffers_) {
-    buffers.push_back(buffer.get());
-  }
-  return buffers;
-}
-
-BufferPtrs Batch::getInputBuffers() { return std::move(input_buffers_); }
+std::unique_ptr<Batch> Batch::propagate() {
+  const auto batch_size = this->size();
+  auto new_batch = std::make_unique<Batch>();
+  new_batch->models_.resize(batch_size);
 
-std::vector<Buffer*> Batch::getRawOutputBuffers() const {
-  std::vector<Buffer*> buffers;
-  buffers.reserve(output_buffers_.size());
-  for (const auto& buffer : output_buffers_) {
-    buffers.push_back(buffer.get());
+  for (auto i = 0U; i < batch_size; ++i) {
+    new_batch->setModel(i, this->getModel(i));
+#ifdef AMDINFER_ENABLE_METRICS
+    new_batch->addTime(this->getTime(i));
+#endif
   }
-  return buffers;
+  return new_batch;
 }
 
-BufferPtrs Batch::getOutputBuffers() { return std::move(output_buffers_); }
+const BufferPtrs& Batch::getInputBuffers() const { return input_buffers_; }
+
+const BufferPtrs& Batch::getOutputBuffers() const { return output_buffers_; }
 
 const std::vector<InferenceRequestPtr>& Batch::getRequests() const {
   return requests_;
@@ -63,12 +60,7 @@ const InferenceRequestPtr& Batch::getRequest(size_t index) {
 bool Batch::empty() const { return requests_.empty(); }
 
 size_t Batch::size() const {
-#ifdef AMDINFER_ENABLE_TRACING
-  assert(requests_.size() == traces_.size());
-#endif
-#ifdef AMDINFER_ENABLE_METRICS
-  assert(requests_.size() == start_times_.size());
-#endif
+  // assert(requests_.size() == models_.size());
 
   return requests_.size();
 }
@@ -82,6 +74,21 @@ void Batch::setBuffers(BufferPtrs inputs, BufferPtrs outputs) {
   output_buffers_ = std::move(outputs);
 }
 
+const std::string& Batch::getModel(size_t index) const {
+  return models_.at(index);
+}
+
+void Batch::setModel(size_t index, std::string model) {
+  auto& old_model = models_.at(index);
+  if (old_model.empty()) {
+    old_model = std::move(model);
+  }
+}
+
+void Batch::addModel(std::string model) {
+  models_.emplace_back(std::move(model));
+}
+
 #ifdef AMDINFER_ENABLE_TRACING
 void Batch::addTrace(TracePtr trace) { traces_.push_back(std::move(trace)); }
 
diff --git a/src/amdinfer/batching/batch.hpp b/src/amdinfer/batching/batch.hpp
index 89c85e430..a3223e8c4 100644
--- a/src/amdinfer/batching/batch.hpp
+++ b/src/amdinfer/batching/batch.hpp
@@ -25,8 +25,6 @@
 
 namespace amdinfer {
 
-class WorkerInfo;
-
 /**
  * @brief The Batch is what the batcher produces and pushes to the workers. It
  * represents the requests, the buffers associated with the request and other
@@ -36,20 +34,23 @@ class WorkerInfo;
 class Batch {
  public:
   void addRequest(InferenceRequestPtr request);
+  std::unique_ptr<Batch> propagate();
 
   void setBuffers(BufferPtrs inputs, BufferPtrs outputs);
   [[nodiscard]] const InferenceRequestPtr& getRequest(size_t index);
   [[nodiscard]] const std::vector<InferenceRequestPtr>& getRequests() const;
-  [[nodiscard]] std::vector<BufferPtr> getInputBuffers();
-  [[nodiscard]] std::vector<BufferPtr> getOutputBuffers();
-  [[nodiscard]] std::vector<Buffer*> getRawInputBuffers() const;
-  [[nodiscard]] std::vector<Buffer*> getRawOutputBuffers() const;
+  [[nodiscard]] const std::vector<BufferPtr>& getInputBuffers() const;
+  [[nodiscard]] const std::vector<BufferPtr>& getOutputBuffers() const;
 
   [[nodiscard]] bool empty() const;
   [[nodiscard]] size_t size() const;
   [[nodiscard]] size_t getInputSize() const;
   [[nodiscard]] size_t getOutputSize() const;
 
+  const std::string& getModel(size_t index) const;
+  void setModel(size_t index, std::string model);
+  void addModel(std::string model);
+
 #ifdef AMDINFER_ENABLE_TRACING
   void addTrace(TracePtr trace);
   TracePtr& getTrace(size_t index);
@@ -63,10 +64,10 @@ class Batch {
   [[nodiscard]] auto end() const { return requests_.end(); }
 
  private:
-  const WorkerInfo* worker_;
   std::vector<InferenceRequestPtr> requests_;
   std::vector<BufferPtr> input_buffers_;
   std::vector<BufferPtr> output_buffers_;
+  std::vector<std::string> models_;
 #ifdef AMDINFER_ENABLE_TRACING
   std::vector<TracePtr> traces_;
 #endif
diff --git a/src/amdinfer/batching/hard.cpp b/src/amdinfer/batching/hard.cpp
index 623d86e4f..2040353c6 100644
--- a/src/amdinfer/batching/hard.cpp
+++ b/src/amdinfer/batching/hard.cpp
@@ -53,7 +53,6 @@ void HardBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
     auto batch = std::make_unique<Batch>();
     size_t batch_size = 0;
 
-    std::vector<BufferPtr> input_buffers;
     std::vector<size_t> input_offset;
     std::vector<size_t> output_offset;
 
@@ -86,6 +85,7 @@ void HardBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
       }
 
       if (first_request) {
+        std::vector<BufferPtr> input_buffers;
         input_buffers.reserve(input_size);
         // auto output_sizes = req->getOutputSizes();
         // TODO(varunsh): the spec does not require the request to have outputs
@@ -105,26 +105,25 @@ void HardBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
         batch->setBuffers(std::move(input_buffers), {});
       }
 
-      auto raw_inputs = batch->getRawInputBuffers();
-      auto raw_outputs = batch->getRawOutputBuffers();
+      const auto& input_buffers = batch->getInputBuffers();
 
       auto old_input_offset = input_offset;
       for (auto i = 0U; i < input_size; ++i) {
         const auto& input = inputs[i];
-        auto* raw_input = raw_inputs[i];
+        const auto& input_buffer = input_buffers.at(i);
         auto& offset = input_offset[i];
 
         auto new_offset =
-          raw_input->write(input.getData(), offset,
-                           input.getSize() * input.getDatatype().size());
-        pool_->put(
-          std::make_unique<CpuBuffer>(input.getData(), MemoryAllocators::Cpu));
-        request->setInputTensorData(i, raw_input->data(offset));
+          input_buffer->write(input.getData(), offset,
+                              input.getSize() * input.getDatatype().size());
+        pool_->put(MemoryAllocators::Cpu, input.getData());
+        request->setInputTensorData(i, input_buffer->data(offset));
         offset = new_offset;
       }
 
       batch->addRequest(request);
       batch_size++;
+      batch->addModel("");
 #ifdef AMDINFER_ENABLE_TRACING
       trace->endSpan();
       batch->addTrace(std::move(trace));
diff --git a/src/amdinfer/batching/soft.cpp b/src/amdinfer/batching/soft.cpp
index c4ef5e987..c16ce2f22 100644
--- a/src/amdinfer/batching/soft.cpp
+++ b/src/amdinfer/batching/soft.cpp
@@ -68,7 +68,6 @@ void SoftBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
     auto batch = std::make_unique<Batch>();
     size_t batch_size = 0;
 
-    std::vector<BufferPtr> input_buffers;
     std::vector<size_t> input_offset;
     std::vector<size_t> output_offset;
 
@@ -118,6 +117,7 @@ void SoftBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
       }
 
       if (first_request) {
+        std::vector<BufferPtr> input_buffers;
         input_buffers.reserve(input_size);
         // auto output_sizes = req->getOutputSizes();
         // TODO(varunsh): the spec does not require the request to have outputs
@@ -137,8 +137,7 @@ void SoftBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
         batch->setBuffers(std::move(input_buffers), {});
       }
 
-      auto raw_inputs = batch->getRawInputBuffers();
-      auto raw_outputs = batch->getRawOutputBuffers();
+      const auto& input_buffers = batch->getInputBuffers();
 
 #ifdef AMDINFER_ENABLE_TRACING
       auto& trace = req->trace;
@@ -154,20 +153,20 @@ void SoftBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
 
       for (auto i = 0U; i < input_size; ++i) {
         const auto& input = inputs[i];
-        auto* raw_input = raw_inputs[i];
+        const auto& input_buffer = input_buffers.at(i);
         auto& offset = input_offset[i];
 
         auto new_offset =
-          raw_input->write(input.getData(), offset,
-                           input.getSize() * input.getDatatype().size());
-        pool_->put(
-          std::make_unique<CpuBuffer>(input.getData(), MemoryAllocators::Cpu));
-        request->setInputTensorData(i, raw_input->data(offset));
+          input_buffer->write(input.getData(), offset,
+                              input.getSize() * input.getDatatype().size());
+        pool_->put(MemoryAllocators::Cpu, input.getData());
+        request->setInputTensorData(i, input_buffer->data(offset));
         offset = new_offset;
       }
 
       batch->addRequest(request);
       batch_size++;
+      batch->addModel("");
 #ifdef AMDINFER_ENABLE_TRACING
       trace->endSpan();
       batch->addTrace(std::move(trace));
diff --git a/src/amdinfer/bindings/python/clients/client.cpp b/src/amdinfer/bindings/python/clients/client.cpp
index 27184c6e6..a06692e67 100644
--- a/src/amdinfer/bindings/python/clients/client.cpp
+++ b/src/amdinfer/bindings/python/clients/client.cpp
@@ -24,6 +24,7 @@
 #include <pybind11/pybind11.h>  // for module_, sequence, class_, pybind11
 #include <pybind11/stl.h>       // IWYU pragma: keep
 
+#include "amdinfer/bindings/python/helpers/docstrings.hpp"  // for DOCS
 #include "amdinfer/core/inference_request.hpp"   // for InferenceRequest
 #include "amdinfer/core/inference_response.hpp"  // for InferenceResponse
 
@@ -31,7 +32,7 @@ namespace py = pybind11;
 
 namespace amdinfer {
 
-void wrapClient(py::module_ &m) {
+void wrapClient(py::module_& m) {
   py::class_<Client> client{m, "Client"};
 
   m.def("serverHasExtension", &serverHasExtension, py::arg("client"),
@@ -39,12 +40,18 @@ void wrapClient(py::module_ &m) {
   m.def("waitUntilServerReady", &waitUntilServerReady, py::arg("client"));
   m.def("waitUntilModelReady", &waitUntilModelReady, py::arg("client"),
         py::arg("model"));
+  m.def("waitUntilModelNotReady", &waitUntilModelNotReady, py::arg("client"),
+        py::arg("model"));
 
   m.def("inferAsyncOrdered", &inferAsyncOrdered, py::arg("client"),
         py::arg("model"), py::arg("requests"));
   m.def("inferAsyncOrderedBatched", &inferAsyncOrderedBatched,
         py::arg("client"), py::arg("model"), py::arg("requests"),
         py::arg("batch_sizes"));
+
+  m.def("loadEnsemble", &loadEnsemble, py::arg("client"), py::arg("models"),
+        py::arg("parameters"));
+  m.def("unloadModels", &unloadModels, py::arg("client"), py::arg("models"));
 }
 
 }  // namespace amdinfer
diff --git a/src/amdinfer/bindings/python/core/inference_request.cpp b/src/amdinfer/bindings/python/core/inference_request.cpp
index 55fa3719f..9f35b0a6b 100644
--- a/src/amdinfer/bindings/python/core/inference_request.cpp
+++ b/src/amdinfer/bindings/python/core/inference_request.cpp
@@ -56,14 +56,11 @@ void setData(amdinfer::InferenceRequestInput &self, py::array_t<T> &b) {
 void wrapInferenceRequestInput(py::module_ &m) {
   py::class_<InferenceRequestInput, InferenceTensor>(m, "InferenceRequestInput")
     .def(py::init<>(), DOCS(InferenceRequestInput, InferenceRequestInput))
-    .def(py::init<const InferenceTensor &>(),
-         DOCS(InferenceRequestInput, InferenceRequestInput, 2),
-         py::arg("tensor"))
     .def(py::init<const Tensor &>(),
-         DOCS(InferenceRequestInput, InferenceRequestInput, 3),
+         DOCS(InferenceRequestInput, InferenceRequestInput, 2),
          py::arg("tensor"))
     .def(py::init<void *, std::vector<uint64_t>, DataType, std::string>(),
-         DOCS(InferenceRequestInput, InferenceRequestInput, 4), py::arg("data"),
+         DOCS(InferenceRequestInput, InferenceRequestInput, 3), py::arg("data"),
          py::arg("shape"), py::arg("data_type"), py::arg("data") = "")
     .def("setUint8Data", &setData<uint8_t>, KeepAliveAssign())
     .def("setUint16Data", &setData<uint16_t>, KeepAliveAssign())
@@ -120,6 +117,8 @@ void wrapInferenceRequest(py::module_ &m) {
       &InferenceRequest::addInputTensor);
   py::class_<InferenceRequest>(m, "InferenceRequest")
     .def(py::init<>(), DOCS(InferenceRequest, InferenceRequest))
+    .def("propagate", &InferenceRequest::propagate,
+         DOCS(InferenceRequest, propagate))
     .def_property("id", &InferenceRequest::getID, &InferenceRequest::setID)
     .def_property("parameters", &InferenceRequest::getParameters,
                   &InferenceRequest::setParameters)
diff --git a/src/amdinfer/bindings/python/core/parameters.cpp b/src/amdinfer/bindings/python/core/parameters.cpp
index 966f0e5f6..fe8503b6f 100644
--- a/src/amdinfer/bindings/python/core/parameters.cpp
+++ b/src/amdinfer/bindings/python/core/parameters.cpp
@@ -42,24 +42,18 @@ void wrapParameterMap(py::module_ &m) {
   using amdinfer::ParameterMap;
 
   py::class_<ParameterMap, std::shared_ptr<ParameterMap>>(m, "ParameterMap")
-    .def(py::init<>(), DOCS(ParameterMap))
-    .def("put",
-         py::overload_cast<const std::string &, bool>(&ParameterMap::put),
-         DOCS(ParameterMap, put))
-    .def("put",
-         py::overload_cast<const std::string &, double>(&ParameterMap::put),
-         DOCS(ParameterMap, put, 2))
-    .def("put",
-         py::overload_cast<const std::string &, int32_t>(&ParameterMap::put),
-         DOCS(ParameterMap, put, 3))
-    .def("put",
-         py::overload_cast<const std::string &, const std::string &>(
-           &ParameterMap::put),
-         DOCS(ParameterMap, put, 4))
+    .def(py::init<>(), DOCS(ParameterMap, ParameterMap))
+    .def(py::init<const std::vector<std::string> &,
+                  const std::vector<Parameter>>(),
+         py::arg("keys"), py::arg("values"),
+         DOCS(ParameterMap, ParameterMap, 2))
     .def(
       "put",
       py::overload_cast<const std::string &, const char *>(&ParameterMap::put),
-      DOCS(ParameterMap, put, 5))
+      DOCS(ParameterMap, put))
+    .def("put",
+         py::overload_cast<const std::string &, Parameter>(&ParameterMap::put),
+         DOCS(ParameterMap, put))
     .def("getBool", &ParameterMap::get<bool>, DOCS(ParameterMap, get))
     .def("getFloat", &ParameterMap::get<double>, DOCS(ParameterMap, get))
     .def("getInt", &ParameterMap::get<int32_t>, DOCS(ParameterMap, get))
diff --git a/src/amdinfer/buffers/CMakeLists.txt b/src/amdinfer/buffers/CMakeLists.txt
index 2a84a9eb1..a2a7c9814 100644
--- a/src/amdinfer/buffers/CMakeLists.txt
+++ b/src/amdinfer/buffers/CMakeLists.txt
@@ -14,7 +14,7 @@
 # limitations under the License.
 
 set(base_targets buffer)
-set(derived_targets cpu)
+set(derived_targets cpu vector)
 if(${AMDINFER_ENABLE_VITIS})
   list(APPEND derived_targets vart_tensor)
 endif()
diff --git a/src/amdinfer/buffers/buffer.cpp b/src/amdinfer/buffers/buffer.cpp
index 1cd094e1f..a0300acd5 100644
--- a/src/amdinfer/buffers/buffer.cpp
+++ b/src/amdinfer/buffers/buffer.cpp
@@ -22,6 +22,8 @@
 
 #include <cstring>  // for memcpy, size_t
 
+#include "amdinfer/core/memory_pool/pool.hpp"
+
 namespace amdinfer {
 
 Buffer::Buffer(MemoryAllocators allocator) : allocator_(allocator) {}
@@ -33,4 +35,8 @@ size_t Buffer::write(void* data, size_t offset, size_t size) {
 
 MemoryAllocators Buffer::getAllocator() const { return allocator_; }
 
+void Buffer::setPool(const MemoryPool* pool) { pool_ = pool; }
+
+const MemoryPool* Buffer::getPool() const { return pool_; }
+
 }  // namespace amdinfer
diff --git a/src/amdinfer/buffers/buffer.hpp b/src/amdinfer/buffers/buffer.hpp
index 0b7c98af7..db6c6bca2 100644
--- a/src/amdinfer/buffers/buffer.hpp
+++ b/src/amdinfer/buffers/buffer.hpp
@@ -33,6 +33,8 @@
 
 namespace amdinfer {
 
+class MemoryPool;
+
 /**
  * @brief The base buffer class. Buffer implementations should extend this class
  * and override the methods.
@@ -93,8 +95,16 @@ class Buffer {
 
   MemoryAllocators getAllocator() const;
 
+  void setPool(const MemoryPool* pool);
+
+  virtual void free() = 0;
+
+ protected:
+  const MemoryPool* getPool() const;
+
  private:
   MemoryAllocators allocator_;
+  const MemoryPool* pool_ = nullptr;
 };
 
 }  // namespace amdinfer
diff --git a/src/amdinfer/buffers/cpu.cpp b/src/amdinfer/buffers/cpu.cpp
index 44cb2517a..18e3666ea 100644
--- a/src/amdinfer/buffers/cpu.cpp
+++ b/src/amdinfer/buffers/cpu.cpp
@@ -23,6 +23,8 @@
 #include <memory>     // for make_unique
 #include <utility>    // for move
 
+#include "amdinfer/core/memory_pool/pool.hpp"
+
 namespace amdinfer {
 
 CpuBuffer::CpuBuffer(void* data, MemoryAllocators allocator)
@@ -30,4 +32,11 @@ CpuBuffer::CpuBuffer(void* data, MemoryAllocators allocator)
 
 void* CpuBuffer::data(size_t offset) { return data_ + offset; }
 
+void CpuBuffer::free() {
+  auto* pool = getPool();
+  if (pool != nullptr) {
+    pool->put(getAllocator(), data_);
+  }
+}
+
 }  // namespace amdinfer
diff --git a/src/amdinfer/buffers/cpu.hpp b/src/amdinfer/buffers/cpu.hpp
index ad5a40769..488d90338 100644
--- a/src/amdinfer/buffers/cpu.hpp
+++ b/src/amdinfer/buffers/cpu.hpp
@@ -40,8 +40,8 @@ class CpuBuffer : public Buffer {
   /**
    * @brief Construct a new Vector Buffer object
    *
-   * @param elements number of elements to store
-   * @param data_type type of element to store
+   * @param data non-owning pointer to data
+   * @param data_type type of memory allocator
    */
   CpuBuffer(void* data, MemoryAllocators allocator);
 
@@ -52,6 +52,8 @@ class CpuBuffer : public Buffer {
    */
   void* data(size_t offset) override;
 
+  void free() override;
+
  private:
   std::byte* data_;
 };
diff --git a/src/amdinfer/buffers/vart_tensor.cpp b/src/amdinfer/buffers/vart_tensor.cpp
index 86ee92de0..07fe5f5e1 100644
--- a/src/amdinfer/buffers/vart_tensor.cpp
+++ b/src/amdinfer/buffers/vart_tensor.cpp
@@ -23,6 +23,8 @@
 #include <vector>                 // for vector
 #include <xir/tensor/tensor.hpp>  // for Tensor
 
+#include "amdinfer/core/memory_pool/pool.hpp"
+
 namespace amdinfer {
 
 VartTensorBuffer::VartTensorBuffer(void* data, MemoryAllocators allocator)
@@ -55,4 +57,11 @@ void* VartTensorBuffer::data(size_t offset) {
 
 vart::TensorBuffer* VartTensorBuffer::getTensorBuffer() { return data_; }
 
+void VartTensorBuffer::free() {
+  auto* pool = getPool();
+  if (pool != nullptr) {
+    pool->put(getAllocator(), data_);
+  }
+}
+
 }  // namespace amdinfer
diff --git a/src/amdinfer/buffers/vart_tensor.hpp b/src/amdinfer/buffers/vart_tensor.hpp
index a6f91ac3a..3220110c4 100644
--- a/src/amdinfer/buffers/vart_tensor.hpp
+++ b/src/amdinfer/buffers/vart_tensor.hpp
@@ -49,6 +49,8 @@ class VartTensorBuffer : public Buffer {
 
   vart::TensorBuffer* getTensorBuffer();
 
+  void free() override;
+
  private:
   vart::TensorBuffer* data_;
 };
diff --git a/src/amdinfer/buffers/vector.cpp b/src/amdinfer/buffers/vector.cpp
new file mode 100644
index 000000000..3ec3a6c17
--- /dev/null
+++ b/src/amdinfer/buffers/vector.cpp
@@ -0,0 +1,36 @@
+// Copyright 2023 Advanced Micro Devices, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/**
+ * @file
+ * @brief Implements the VectorBuffer class
+ */
+
+#include "amdinfer/buffers/vector.hpp"
+
+#include <algorithm>  // for max
+#include <memory>     // for make_unique
+#include <utility>    // for move
+
+namespace amdinfer {
+
+VectorBuffer::VectorBuffer(size_t size) : Buffer(MemoryAllocators::Cpu) {
+  data_.resize(size);
+}
+
+void* VectorBuffer::data(size_t offset) { return data_.data() + offset; }
+
+void VectorBuffer::free() { data_.clear(); }
+
+}  // namespace amdinfer
diff --git a/src/amdinfer/buffers/vector.hpp b/src/amdinfer/buffers/vector.hpp
new file mode 100644
index 000000000..16cf9b018
--- /dev/null
+++ b/src/amdinfer/buffers/vector.hpp
@@ -0,0 +1,62 @@
+// Copyright 2023 Advanced Micro Devices, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/**
+ * @file
+ * @brief
+ */
+
+#ifndef GUARD_AMDINFER_BUFFERS_VECTOR
+#define GUARD_AMDINFER_BUFFERS_VECTOR
+
+#include <cstddef>  // for size_t, byte
+#include <vector>   // for vector
+
+#include "amdinfer/buffers/buffer.hpp"   // IWYU pragma: export
+#include "amdinfer/core/data_types.hpp"  // for DataType
+#include "amdinfer/util/queue.hpp"       // for BufferPtrsQueue
+
+namespace amdinfer {
+
+enum class MemoryAllocators;
+
+/**
+ * @brief CpuBuffer uses vectors for storing data
+ *
+ */
+class VectorBuffer : public Buffer {
+ public:
+  /**
+   * @brief Construct a new Vector Buffer object
+   *
+   * @param size size of the buffer in bytes
+   */
+  explicit VectorBuffer(size_t size);
+
+  /**
+   * @brief Returns a pointer to the underlying data
+   *
+   * @return void*
+   */
+  void* data(size_t offset) override;
+
+  void free() override;
+
+ private:
+  std::vector<std::byte> data_;
+};
+
+}  // namespace amdinfer
+
+#endif  // GUARD_AMDINFER_BUFFERS_VECTOR
diff --git a/src/amdinfer/clients/client.cpp b/src/amdinfer/clients/client.cpp
index d92fec3de..f692f9392 100644
--- a/src/amdinfer/clients/client.cpp
+++ b/src/amdinfer/clients/client.cpp
@@ -44,6 +44,41 @@ void initializeClientLogging() {
 #endif
 }
 
+std::vector<std::string> loadEnsemble(const Client* client,
+                                      std::vector<std::string> workers,
+                                      std::vector<ParameterMap> parameters) {
+  if (workers.size() != parameters.size()) {
+    throw invalid_argument("The number of workers and parameters must match");
+  }
+
+  std::vector<std::string> endpoints;
+  endpoints.resize(workers.size());
+
+  std::string next;
+  // reverse iterate through vectors using "goes to" operator
+  for (auto i = workers.size(); i-- > 0;) {
+    const auto& worker = workers.at(i);
+    auto& parameter = parameters.at(i);
+
+    if (!next.empty()) {
+      parameter.put("next", next);
+    }
+    auto endpoint = client->workerLoad(worker, parameter);
+    next = endpoint;
+    endpoints.at(i) = std::move(endpoint);
+    waitUntilModelReady(client, next);
+  }
+
+  return endpoints;
+}
+
+void unloadModels(const Client* client,
+                  const std::vector<std::string>& models) {
+  for (const auto& model : models) {
+    client->modelUnload(model);
+  }
+}
+
 Client::Client() { initializeClientLogging(); }
 
 bool serverHasExtension(const Client* client, const std::string& extension) {
@@ -65,8 +100,21 @@ void waitUntilServerReady(const Client* client) {
 
 void waitUntilModelReady(const Client* client, const std::string& model) {
   bool ready = false;
+  // arbitrarily set to 1ms
+  const auto sleep_time = std::chrono::milliseconds(1);
   while (!ready) {
     ready = client->modelReady(model);
+    std::this_thread::sleep_for(sleep_time);
+  }
+}
+
+void waitUntilModelNotReady(const Client* client, const std::string& model) {
+  bool ready = true;
+  // arbitrarily set to 1ms
+  const auto sleep_time = std::chrono::milliseconds(1);
+  while (ready) {
+    ready = client->modelReady(model);
+    std::this_thread::sleep_for(sleep_time);
   }
 }
 
diff --git a/src/amdinfer/core/endpoints.cpp b/src/amdinfer/core/endpoints.cpp
index df9775b3a..f28054dec 100644
--- a/src/amdinfer/core/endpoints.cpp
+++ b/src/amdinfer/core/endpoints.cpp
@@ -28,6 +28,7 @@
 #include "amdinfer/core/parameters.hpp"         // for ParameterMap
 #include "amdinfer/core/request_container.hpp"  // for RequestContainer
 #include "amdinfer/core/worker_info.hpp"        // for WorkerInfo
+#include "amdinfer/util/string.hpp"             // for startsWith
 #include "amdinfer/util/thread.hpp"             // for setThreadName
 
 namespace amdinfer {
@@ -234,13 +235,30 @@ std::string Endpoints::unsafeLoad(const std::string& worker,
   std::string worker_name = endpoint;
   if (parameters->has("worker")) {
     worker_name = parameters->get<std::string>("worker");
+  } else {
+    parameters->put("worker", worker_name);
   }
 
   // if the worker doesn't exist yet, we need to create it
   try {
     if (worker_info == nullptr) {
-      auto new_worker =
-        std::make_unique<WorkerInfo>(worker_name, parameters, &pool_);
+      BatchPtrQueue* next = nullptr;
+      std::vector<MemoryAllocators> next_allocators;
+      if (parameters->has("next")) {
+        auto next_endpoint = parameters->get<std::string>("next");
+        const auto* next_info = this->unsafeGet(next_endpoint);
+        next = next_info->getInputQueue();
+        next_allocators = next_info->getAllocators();
+      } else if (worker != "responder") {
+        const auto* next_info = this->unsafeGet("responder");
+        next = next_info->getInputQueue();
+        next_allocators = next_info->getAllocators();
+      } else {
+        // worker being loaded is the responder so we don't do anything
+      }
+
+      auto new_worker = std::make_unique<WorkerInfo>(
+        worker_name, parameters, &pool_, next, next_allocators);
       this->workers_.try_emplace(endpoint, std::move(new_worker));
       // if the worker exists but the share parameter is false, we need to add
       // one
@@ -301,7 +319,9 @@ void Endpoints::unsafeList(std::vector<std::string>* list) const {
   assert(list->empty());
   list->reserve(workers_.size());
   for (const auto& [worker, _] : workers_) {
-    list->push_back(worker);
+    if (!util::startsWith(worker, "responder")) {
+      list->push_back(worker);
+    }
   }
 }
 
diff --git a/src/amdinfer/core/inference_request.cpp b/src/amdinfer/core/inference_request.cpp
index 2c9007c17..aa04b95ac 100644
--- a/src/amdinfer/core/inference_request.cpp
+++ b/src/amdinfer/core/inference_request.cpp
@@ -25,6 +25,18 @@
 
 namespace amdinfer {
 
+InferenceRequestPtr InferenceRequest::propagate() {
+  auto new_request = std::make_shared<InferenceRequest>();
+  new_request->setCallback(this->getCallback());
+  new_request->setID(this->getID());
+  const auto outputs = this->getOutputs();
+  for (const auto &output : outputs) {
+    new_request->addOutputTensor(output);
+  }
+
+  return new_request;
+}
+
 void InferenceRequest::setCallback(Callback &&callback) {
   callback_ = std::move(callback);
 }
@@ -86,8 +98,6 @@ InferenceRequestInput::InferenceRequestInput(void *data,
   : InferenceTensor(std::move(name), std::move(shape), data_type),
     data_(data) {}
 
-InferenceRequestInput::InferenceRequestInput(const InferenceTensor &tensor)
-  : InferenceTensor(tensor) {}
 InferenceRequestInput::InferenceRequestInput(const Tensor &tensor)
   : InferenceTensor(tensor) {}
 
diff --git a/src/amdinfer/core/memory_pool/pool.cpp b/src/amdinfer/core/memory_pool/pool.cpp
index fdca2c3ac..0e88a4dcc 100644
--- a/src/amdinfer/core/memory_pool/pool.cpp
+++ b/src/amdinfer/core/memory_pool/pool.cpp
@@ -19,6 +19,8 @@
 
 #include "amdinfer/core/memory_pool/pool.hpp"
 
+#include <cassert>
+
 #include "amdinfer/buffers/cpu.hpp"
 #include "amdinfer/core/exceptions.hpp"
 #include "amdinfer/core/memory_pool/cpu_allocator.hpp"
@@ -40,9 +42,12 @@ MemoryPool::MemoryPool() {
 std::unique_ptr<Buffer> MemoryPool::get(
   const std::vector<MemoryAllocators>& allocators, const Tensor& tensor,
   size_t batch_size) const {
+  assert(!allocators.empty());
   for (const auto& allocator : allocators) {
     try {
-      return allocators_.at(allocator)->get(tensor, batch_size);
+      auto buffer = allocators_.at(allocator)->get(tensor, batch_size);
+      buffer->setPool(this);
+      return buffer;
     } catch (const runtime_error&) {
       continue;
     }
@@ -50,10 +55,8 @@ std::unique_ptr<Buffer> MemoryPool::get(
   throw runtime_error("Memory could not be allocated");
 }
 
-void MemoryPool::put(std::unique_ptr<Buffer> memory) const {
-  const auto allocator = memory->getAllocator();
-  const auto* address = memory->data(0);
-  allocators_.at(allocator)->put(address);
+void MemoryPool::put(MemoryAllocators allocator, void* memory) const {
+  allocators_.at(allocator)->put(memory);
 }
 
 }  // namespace amdinfer
diff --git a/src/amdinfer/core/memory_pool/pool.hpp b/src/amdinfer/core/memory_pool/pool.hpp
index 7623b9df2..f379507ef 100644
--- a/src/amdinfer/core/memory_pool/pool.hpp
+++ b/src/amdinfer/core/memory_pool/pool.hpp
@@ -37,7 +37,7 @@ class MemoryPool {
 
   std::unique_ptr<Buffer> get(const std::vector<MemoryAllocators>& allocators,
                               const Tensor& tensor, size_t batch_size) const;
-  void put(std::unique_ptr<Buffer> memory) const;
+  void put(MemoryAllocators allocator, void* memory) const;
 
  private:
   std::unordered_map<MemoryAllocators, std::unique_ptr<MemoryAllocator>>
diff --git a/src/amdinfer/core/memory_pool/vart_tensor_allocator.cpp b/src/amdinfer/core/memory_pool/vart_tensor_allocator.cpp
index 279a4b756..091c049ad 100644
--- a/src/amdinfer/core/memory_pool/vart_tensor_allocator.cpp
+++ b/src/amdinfer/core/memory_pool/vart_tensor_allocator.cpp
@@ -79,18 +79,15 @@ BufferPtr VartTensorAllocator::get(const Tensor& tensor, size_t batch_size) {
 void VartTensorAllocator::put(const void* address) {
   const std::lock_guard lock{mutex_};
   const auto end = headers_.end();
-  auto found = headers_.end();
   for (auto it = headers_.begin(); it != end; it++) {
     if (it->address == address) {
-      found->free = true;
+      it->free = true;
       // std::cout << "Freed memory\n";
       return;
     }
   }
 
-  if (found == end) {
-    throw runtime_error("Address not found");
-  }
+  throw runtime_error("Address not found");
 }
 
 }  // namespace amdinfer
diff --git a/src/amdinfer/core/parameters.cpp b/src/amdinfer/core/parameters.cpp
index 292f9509f..2ae8ab74d 100644
--- a/src/amdinfer/core/parameters.cpp
+++ b/src/amdinfer/core/parameters.cpp
@@ -26,24 +26,25 @@
 #include <type_traits>  // for add_const<>::type, decay_t
 #include <vector>       // for vector
 
-#include "amdinfer/util/memory.hpp"  // for copy
+#include "amdinfer/core/exceptions.hpp"  // for invalid_argument
+#include "amdinfer/util/memory.hpp"      // for copy
 
 namespace amdinfer {
 
-void ParameterMap::put(const std::string &key, bool value) {
-  this->parameters_.try_emplace(key, value);
-}
-
-void ParameterMap::put(const std::string &key, double value) {
-  this->parameters_.try_emplace(key, value);
-}
+ParameterMap::ParameterMap(const std::vector<std::string> &keys,
+                           const std::vector<Parameter> &values) {
+  if (keys.size() != values.size()) {
+    throw invalid_argument("Keys and values sizes must match");
+  }
 
-void ParameterMap::put(const std::string &key, int32_t value) {
-  this->parameters_.try_emplace(key, value);
+  const auto size = keys.size();
+  for (auto i = 0U; i < size; ++i) {
+    ParameterMap::put(keys.at(i), values.at(i));
+  }
 }
 
-void ParameterMap::put(const std::string &key, const std::string &value) {
-  this->parameters_.try_emplace(key, value);
+void ParameterMap::put(const std::string &key, Parameter value) {
+  this->parameters_.try_emplace(key, std::move(value));
 }
 
 void ParameterMap::put(const std::string &key, const char *value) {
@@ -153,8 +154,6 @@ std::byte *ParameterMap::serialize(std::byte *data_out) const {
  * @tparam Ts: the types of the variant. Order matters!
  * @param i index of the variant type to create
  * @return std::variant<Ts...>
- *
- * https://www.reddit.com/r/cpp/comments/f8cbzs/comment/fimjm2f/?context=3
  */
 template <typename... Ts>
 [[nodiscard]] std::variant<Ts...> expandType(std::size_t i) {
diff --git a/src/amdinfer/core/worker_info.cpp b/src/amdinfer/core/worker_info.cpp
index fb8a35f04..ab58c8b56 100644
--- a/src/amdinfer/core/worker_info.cpp
+++ b/src/amdinfer/core/worker_info.cpp
@@ -105,7 +105,9 @@ workers::Worker* getWorker(const std::string& name) {
 }
 
 WorkerInfo::WorkerInfo(const std::string& name, ParameterMap* parameters,
-                       MemoryPool* pool) {
+                       MemoryPool* pool, BatchPtrQueue* next,
+                       const std::vector<MemoryAllocators>& next_allocators)
+  : next_(next), next_allocators_(next_allocators) {
   this->addAndStartWorker(name, parameters, pool);
 }
 
@@ -133,7 +135,8 @@ void WorkerInfo::addAndStartWorker(const std::string& name,
   }
 
   this->batch_size_ = worker->getBatchSize();
-  worker->setPool(pool);
+  worker->setNext(next_);
+  worker->setNextAllocators(next_allocators_);
 
   if (this->batchers_.empty()) {
     int32_t batcher_count = 1;
@@ -153,7 +156,8 @@ void WorkerInfo::addAndStartWorker(const std::string& name,
       batcher->start(allocators);
     }
   }
-  auto thread = worker->spawn(this->batchers_[0]->getOutputQueue());
+  std::thread thread{&workers::Worker::run, worker,
+                     this->batchers_[0]->getOutputQueue(), pool};
 
   auto thread_id = thread.get_id();
 
@@ -163,6 +167,10 @@ void WorkerInfo::addAndStartWorker(const std::string& name,
 
 Batcher* WorkerInfo::getBatcher() { return this->batchers_[0].get(); }
 
+BatchPtrQueue* WorkerInfo::getInputQueue() const {
+  return batchers_[0]->getOutputQueue();
+}
+
 void WorkerInfo::join(std::thread::id id) {
   auto& thread = worker_threads_.at(id);
   if (thread.joinable()) {
@@ -236,4 +244,9 @@ ModelMetadata WorkerInfo::getMetadata() const {
   return worker_class->getMetadata();
 }
 
+std::vector<MemoryAllocators> WorkerInfo::getAllocators() const {
+  const auto* worker_class = workers_.begin()->second;
+  return worker_class->getAllocators();
+}
+
 }  // namespace amdinfer
diff --git a/src/amdinfer/core/worker_info.hpp b/src/amdinfer/core/worker_info.hpp
index 88961abd8..daaaa1d94 100644
--- a/src/amdinfer/core/worker_info.hpp
+++ b/src/amdinfer/core/worker_info.hpp
@@ -29,8 +29,9 @@
 #include <thread>   // for thread, thread::id
 #include <vector>   // for vector
 
-#include "amdinfer/declarations.hpp"  // for BufferPtr
-#include "amdinfer/util/queue.hpp"    // for BufferPtrsQueuePtr
+#include "amdinfer/batching/batcher.hpp"  // for BatchPtrQueue
+#include "amdinfer/declarations.hpp"      // for BufferPtr
+#include "amdinfer/util/queue.hpp"        // for BufferPtrsQueuePtr
 
 namespace amdinfer {
 class Batcher;
@@ -52,7 +53,8 @@ class WorkerInfo {
  public:
   /// Construct a new WorkerInfo object
   WorkerInfo(const std::string& name, ParameterMap* parameters,
-             MemoryPool* pool);
+             MemoryPool* pool, BatchPtrQueue* next,
+             const std::vector<MemoryAllocators>& next_allocators);
   ~WorkerInfo();                           ///> Destroy a WorkerInfo object
   WorkerInfo(WorkerInfo const&) = delete;  ///< Copy constructor
   /// Copy assignment constructor
@@ -67,6 +69,12 @@ class WorkerInfo {
    * @return InferenceRequestPtrQueue*
    */
   Batcher* getBatcher();
+  /**
+   * @brief Get the queue associated with this worker to send it batches
+   *
+   * @return BatchPtrQueue*
+   */
+  BatchPtrQueue* getInputQueue() const;
   /// Blocks until the associated worker's thread joins
   void join(std::thread::id id);
   void joinAll();  ///< Blocks until all workers in the group join
@@ -92,6 +100,8 @@ class WorkerInfo {
   /// get the batch size of the worker group
   [[nodiscard]] auto getBatchSize() const { return this->batch_size_; }
 
+  std::vector<MemoryAllocators> getAllocators() const;
+
   ModelMetadata getMetadata() const;
 
  private:
@@ -99,6 +109,8 @@ class WorkerInfo {
   std::map<std::thread::id, workers::Worker*> workers_;
   std::vector<std::unique_ptr<Batcher>> batchers_;
   size_t batch_size_ = 1;
+  BatchPtrQueue* next_;
+  std::vector<MemoryAllocators> next_allocators_;
 
   friend class Manager;
 };
diff --git a/src/amdinfer/models/CMakeLists.txt b/src/amdinfer/models/CMakeLists.txt
index 8a339656a..b38e6cfc4 100644
--- a/src/amdinfer/models/CMakeLists.txt
+++ b/src/amdinfer/models/CMakeLists.txt
@@ -14,7 +14,7 @@
 
 include(GNUInstallDirs)
 
-set(filenames echo echo_multi)
+set(filenames base64_encode base64_decode echo echo_multi invert_image)
 set(targets "")
 
 function(amdinfer_get_target_name target filename)
@@ -30,6 +30,12 @@ foreach(filename ${filenames})
   list(APPEND targets ${target})
 endforeach()
 
+# link these models against OpenCV
+foreach(filename base64_encode base64_decode invert_image)
+  amdinfer_get_target_name(target ${filename})
+  target_link_libraries(${target} PRIVATE opencv_core opencv_imgcodecs)
+endforeach()
+
 if(NOT SKBUILD)
   install(TARGETS ${targets} LIBRARY COMPONENT amdinfer_Runtime
           LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}"
diff --git a/src/amdinfer/models/base64_decode.cpp b/src/amdinfer/models/base64_decode.cpp
new file mode 100644
index 000000000..1efc34a9a
--- /dev/null
+++ b/src/amdinfer/models/base64_decode.cpp
@@ -0,0 +1,142 @@
+// Copyright 2023 Advanced Micro Devices, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/**
+ * @file
+ * @brief Implements the base64_codec model
+ */
+
+#include <opencv2/core.hpp>       // for bitwise_not, Mat
+#include <opencv2/imgcodecs.hpp>  // for imdecode, imencode, IMRE...
+
+#include "amdinfer/batching/batch.hpp"
+#include "amdinfer/buffers/vector.hpp"
+#include "amdinfer/core/inference_request.hpp"
+#include "amdinfer/core/inference_response.hpp"
+#include "amdinfer/core/parameters.hpp"
+#include "amdinfer/observation/logging.hpp"
+#include "amdinfer/observation/metrics.hpp"
+#include "amdinfer/observation/tracing.hpp"
+#include "amdinfer/util/base64.hpp"  // for base64_decode, base64_e...
+#include "amdinfer/util/containers.hpp"
+#include "amdinfer/util/memory.hpp"
+#include "amdinfer/util/timer.hpp"
+
+extern "C" {
+
+std::vector<amdinfer::Tensor> getInputs() {
+  // intentionally return an empty shape. This forces the worker to use
+  // different logic to invoke the run method.
+  return {};
+}
+
+std::vector<amdinfer::Tensor> getOutputs() { return {}; }
+
+// Support up to Full HD
+const auto kMaxImageHeight = 1080;
+const auto kMaxImageWidth = 1920;
+const auto kMaxImageChannels = 3;
+const auto kMaxImageSize = kMaxImageHeight * kMaxImageWidth * kMaxImageChannels;
+
+amdinfer::BatchPtr run(amdinfer::Batch* batch) {
+  amdinfer::Logger logger{amdinfer::Loggers::Server};
+
+  auto new_batch = batch->propagate();
+  const auto batch_size = batch->size();
+
+  // const auto data_size = amdinfer::DataType("Uint8").size();
+  // std::vector<amdinfer::BufferPtr> input_buffers;
+  // input_buffers.emplace_back(std::make_unique<amdinfer::VectorBuffer>(
+  //   kMaxImageSize * batch_size * data_size));
+
+  std::vector<cv::Mat> decoded_images;
+  size_t max_decoded_size = 0;
+  const auto channels = 3;  // assuming 3 channels
+  for (unsigned int j = 0; j < batch_size; j++) {
+    const auto& req = batch->getRequest(j);
+#ifdef AMDINFER_ENABLE_TRACING
+    const auto& trace = batch->getTrace(j);
+    trace->startSpan("base64_decode");
+#endif
+    const auto& inputs = req->getInputs();
+    if (inputs.size() != 1) {
+      req->runCallbackError("Only one input tensor should be present");
+      continue;
+    }
+    auto& input = inputs.at(0);
+    const auto& input_shape = input.getShape();
+    auto input_size = amdinfer::util::containerProduct(input_shape);
+
+    auto new_request = req->propagate();
+
+    auto* input_data = static_cast<char*>(input.getData());
+    auto decoded_str = amdinfer::util::base64Decode(input_data, input_size);
+    std::vector<char> data(decoded_str.begin(), decoded_str.end());
+    cv::Mat img;
+    try {
+      img = cv::imdecode(data, cv::IMREAD_UNCHANGED);
+    } catch (const cv::Exception& e) {
+      AMDINFER_LOG_ERROR(logger, e.what());
+      req->runCallbackError("Failed to decode base64 image data");
+      continue;
+    }
+
+    if (img.empty()) {
+      const char* error = "Decoded image is empty";
+      AMDINFER_LOG_ERROR(logger, error);
+      req->runCallbackError(error);
+      continue;
+    }
+
+    const size_t decoded_size = img.rows * img.cols * channels;
+    if (decoded_size > max_decoded_size) {
+      max_decoded_size = decoded_size;
+    }
+
+    std::vector<uint64_t> shape{static_cast<uint64_t>(img.rows),
+                                static_cast<uint64_t>(img.cols), 3};
+
+    new_request->addInputTensor(nullptr, shape, amdinfer::DataType::Uint8,
+                                "output");
+    decoded_images.emplace_back(std::move(img));
+
+    new_batch->addRequest(new_request);
+
+#ifdef AMDINFER_ENABLE_TRACING
+    trace->endSpan();
+#endif
+  }
+
+  const auto data_size = amdinfer::DataType("Uint8").size();
+  std::vector<amdinfer::BufferPtr> input_buffers;
+  input_buffers.emplace_back(std::make_unique<amdinfer::VectorBuffer>(
+    max_decoded_size * batch_size * data_size));
+
+  for (auto j = 0U; j < batch_size; j++) {
+    auto* data_ptr =
+      input_buffers.at(0)->data(j * max_decoded_size * data_size);
+    const auto& req = new_batch->getRequest(j);
+    req->setInputTensorData(j, data_ptr);
+    const auto& data = decoded_images.at(j);
+    const auto decoded_size = data.rows * data.cols * channels;
+    amdinfer::util::copy(data.data, static_cast<std::byte*>(data_ptr),
+                         decoded_size);
+  }
+
+  new_batch->setBuffers(std::move(input_buffers), {});
+
+  return new_batch;
+}
+
+}  // extern "C"
diff --git a/src/amdinfer/models/base64_encode.cpp b/src/amdinfer/models/base64_encode.cpp
new file mode 100644
index 000000000..d0d69057e
--- /dev/null
+++ b/src/amdinfer/models/base64_encode.cpp
@@ -0,0 +1,119 @@
+// Copyright 2023 Advanced Micro Devices, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/**
+ * @file
+ * @brief Implements the base64_codec model
+ */
+
+#include <opencv2/core.hpp>       // for bitwise_not, Mat
+#include <opencv2/imgcodecs.hpp>  // for imdecode, imencode, IMRE...
+
+#include "amdinfer/batching/batch.hpp"
+#include "amdinfer/buffers/vector.hpp"
+#include "amdinfer/core/inference_request.hpp"
+#include "amdinfer/core/inference_response.hpp"
+#include "amdinfer/core/parameters.hpp"
+#include "amdinfer/observation/logging.hpp"
+#include "amdinfer/observation/metrics.hpp"
+#include "amdinfer/observation/tracing.hpp"
+#include "amdinfer/util/base64.hpp"  // for base64_decode, base64_e...
+#include "amdinfer/util/containers.hpp"
+#include "amdinfer/util/memory.hpp"
+#include "amdinfer/util/timer.hpp"
+
+extern "C" {
+
+std::vector<amdinfer::Tensor> getInputs() {
+  // intentionally return an empty shape. This forces the worker to use
+  // different logic to invoke the run method.
+  return {};
+}
+
+std::vector<amdinfer::Tensor> getOutputs() { return {}; }
+
+// Support up to Full HD
+const auto kMaxImageHeight = 1080;
+const auto kMaxImageWidth = 1920;
+const auto kMaxImageChannels = 3;
+const auto kMaxImageSize = kMaxImageHeight * kMaxImageWidth * kMaxImageChannels;
+
+amdinfer::BatchPtr run(amdinfer::Batch* batch) {
+  amdinfer::Logger logger{amdinfer::Loggers::Server};
+
+  auto new_batch = batch->propagate();
+  const auto batch_size = batch->size();
+
+  std::vector<std::string> encoded_images;
+  size_t max_encoded_size = 0;
+  for (unsigned int j = 0; j < batch_size; j++) {
+    const auto& req = batch->getRequest(j);
+#ifdef AMDINFER_ENABLE_TRACING
+    const auto& trace = batch->getTrace(j);
+    trace->startSpan("base64_encode");
+#endif
+    const auto& inputs = req->getInputs();
+    if (inputs.size() != 1) {
+      req->runCallbackError("Only one input tensor should be present");
+      continue;
+    }
+    auto& input = inputs.at(0);
+    auto input_shape = input.getShape();
+
+    auto new_request = req->propagate();
+
+    auto img =
+      cv::Mat{static_cast<int>(input_shape[0]),
+              static_cast<int>(input_shape[1]), CV_8UC3, input.getData()};
+    std::vector<unsigned char> buf;
+    cv::imencode(".jpg", img, buf);
+    const auto* enc_msg = reinterpret_cast<const char*>(buf.data());
+    auto encoded = amdinfer::util::base64Encode(enc_msg, buf.size());
+    const auto encoded_size = encoded.size();
+    if (encoded_size > max_encoded_size) {
+      max_encoded_size = encoded_size;
+    }
+
+    new_request->addInputTensor(nullptr, {encoded_size},
+                                amdinfer::DataType::String, "output");
+    encoded_images.emplace_back(std::move(encoded));
+
+    new_batch->addRequest(new_request);
+
+#ifdef AMDINFER_ENABLE_TRACING
+    trace->endSpan();
+#endif
+  }
+
+  const auto data_size = amdinfer::DataType("Uint8").size();
+  std::vector<amdinfer::BufferPtr> input_buffers;
+  input_buffers.emplace_back(std::make_unique<amdinfer::VectorBuffer>(
+    max_encoded_size * batch_size * data_size));
+
+  for (auto j = 0U; j < batch_size; j++) {
+    auto* data_ptr =
+      input_buffers.at(0)->data(j * max_encoded_size * data_size);
+    const auto& req = new_batch->getRequest(j);
+    req->setInputTensorData(j, data_ptr);
+    const auto& data = encoded_images.at(j);
+    amdinfer::util::copy(data.data(), static_cast<std::byte*>(data_ptr),
+                         data.size());
+  }
+
+  new_batch->setBuffers(std::move(input_buffers), {});
+
+  return new_batch;
+}
+
+}  // extern "C"
diff --git a/src/amdinfer/models/echo.cpp b/src/amdinfer/models/echo.cpp
index dac59180b..70d661e30 100644
--- a/src/amdinfer/models/echo.cpp
+++ b/src/amdinfer/models/echo.cpp
@@ -49,16 +49,9 @@ void run(amdinfer::Batch* batch, amdinfer::Batch* new_batch) {
   const auto batch_size = batch->size();
   for (unsigned int j = 0; j < batch_size; j++) {
     const auto& req = batch->getRequest(j);
-#ifdef AMDINFER_ENABLE_TRACING
-    auto& trace = batch->getTrace(j);
-    trace->startSpan("echo");
-#endif
     const auto& new_request = new_batch->getRequest(j);
-    new_request->setCallback(req->getCallback());
 
-    // new_request->setID(req->getID());
     const auto& inputs = req->getInputs();
-    // auto outputs = req->getOutputs();
     const auto inputs_size = inputs.size();
 
     const auto& new_inputs = new_request->getInputs();
@@ -79,14 +72,7 @@ void run(amdinfer::Batch* batch, amdinfer::Batch* new_batch) {
                            static_cast<std::byte*>(new_inputs.at(i).getData()));
     }
 
-#ifdef AMDINFER_ENABLE_TRACING
-    trace->endSpan();
-    new_batch->addTrace(std::move(trace));
-#endif
-
-#ifdef AMDINFER_ENABLE_METRICS
-    new_batch->addTime(batch->getTime(j));
-#endif
+    new_batch->setModel(j, "echo");
   }
 }
 
diff --git a/src/amdinfer/models/echo_multi.cpp b/src/amdinfer/models/echo_multi.cpp
index 737d5dd5b..c2ccb255e 100644
--- a/src/amdinfer/models/echo_multi.cpp
+++ b/src/amdinfer/models/echo_multi.cpp
@@ -18,6 +18,7 @@
  */
 
 #include "amdinfer/batching/batch.hpp"
+#include "amdinfer/buffers/vector.hpp"
 #include "amdinfer/core/inference_request.hpp"
 #include "amdinfer/core/inference_response.hpp"
 #include "amdinfer/core/parameters.hpp"
@@ -47,32 +48,38 @@ std::vector<amdinfer::Tensor> getInputs() {
 }
 
 std::vector<amdinfer::Tensor> getOutputs() {
-  std::vector<amdinfer::Tensor> output_tensors;
-  output_tensors.reserve(kOutputTensors);
-  for (auto i = 0; i < kOutputTensors; ++i) {
-    std::vector<size_t> shape = {kOutputLengths.at(i)};
-    output_tensors.emplace_back("output" + std::to_string(i), shape,
-                                amdinfer::DataType::Uint32);
-  }
-  return output_tensors;
+  // intentionally return an empty shape, indicating an unknown output tensor
+  // size even though it is known in this case. This forces the worker to use
+  // different logic to invoke the run method.
+  return {};
 }
 
-void run(amdinfer::Batch* batch, amdinfer::Batch* new_batch) {
+amdinfer::BatchPtr run(amdinfer::Batch* batch) {
   amdinfer::Logger logger{amdinfer::Loggers::Server};
 
+  auto new_batch = batch->propagate();
   const auto batch_size = batch->size();
+  const auto data_size = amdinfer::DataType("Uint32").size();
+
+  std::vector<amdinfer::BufferPtr> input_buffers;
+  input_buffers.reserve(kOutputTensors);
+  for (auto i = 0; i < kOutputTensors; ++i) {
+    input_buffers.emplace_back(std::make_unique<amdinfer::VectorBuffer>(
+      kOutputLengths.at(i) * batch_size * data_size));
+  }
+
   for (unsigned int j = 0; j < batch_size; j++) {
     const auto& req = batch->getRequest(j);
-#ifdef AMDINFER_ENABLE_TRACING
-    auto& trace = batch->getTrace(j);
-    trace->startSpan("echoMulti");
-#endif
-    const auto& new_request = new_batch->getRequest(j);
-    new_request->setCallback(req->getCallback());
-
-    // new_request->setID(req->getID());
+    auto new_request = req->propagate();
+    for (auto i = 0; i < kOutputTensors; ++i) {
+      std::vector<size_t> shape = {kOutputLengths.at(i)};
+      auto* data_ptr =
+        input_buffers.at(i)->data(j * kOutputLengths.at(i) * data_size);
+      new_request->addInputTensor(data_ptr, shape, amdinfer::DataType::Uint32,
+                                  "output" + std::to_string(i));
+    }
+
     const auto& inputs = req->getInputs();
-    // auto outputs = req->getOutputs();
 
     std::vector<int> args;
     const auto input_num = amdinfer::util::containerSum(kInputLengths);
@@ -97,15 +104,13 @@ void run(amdinfer::Batch* batch, amdinfer::Batch* new_batch) {
       }
     }
 
-#ifdef AMDINFER_ENABLE_TRACING
-    trace->endSpan();
-    new_batch->addTrace(std::move(trace));
-#endif
-
-#ifdef AMDINFER_ENABLE_METRICS
-    new_batch->addTime(batch->getTime(j));
-#endif
+    new_batch->addRequest(new_request);
+    new_batch->setModel(j, "echo_multi");
   }
+
+  new_batch->setBuffers(std::move(input_buffers), {});
+
+  return new_batch;
 }
 
 }  // extern "C"
diff --git a/src/amdinfer/models/invert_image.cpp b/src/amdinfer/models/invert_image.cpp
new file mode 100644
index 000000000..87dd44397
--- /dev/null
+++ b/src/amdinfer/models/invert_image.cpp
@@ -0,0 +1,115 @@
+// Copyright 2023 Advanced Micro Devices, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/**
+ * @file
+ * @brief Implements the invert_image model
+ */
+
+#include <opencv2/core.hpp>       // for bitwise_not, Mat
+#include <opencv2/imgcodecs.hpp>  // for imdecode, imencode, IMRE...
+
+#include "amdinfer/batching/batch.hpp"
+#include "amdinfer/buffers/vector.hpp"
+#include "amdinfer/core/inference_request.hpp"
+#include "amdinfer/core/inference_response.hpp"
+#include "amdinfer/core/parameters.hpp"
+#include "amdinfer/observation/logging.hpp"
+#include "amdinfer/observation/metrics.hpp"
+#include "amdinfer/observation/tracing.hpp"
+#include "amdinfer/util/base64.hpp"  // for base64_decode, base64_e...
+#include "amdinfer/util/containers.hpp"
+#include "amdinfer/util/memory.hpp"
+#include "amdinfer/util/timer.hpp"
+
+/// Invert the pixel color. Assumes RGB in some order with optional alpha
+template <typename T, bool kAlphaPresent>
+void invert(void* input_buffer, void* output_buffer, uint64_t size) {
+  static_assert(std::is_pointer_v<T>, "T must be a pointer type");
+  auto* input_data = static_cast<T>(input_buffer);
+  auto* output_data = static_cast<T>(output_buffer);
+  static_assert(sizeof(input_data[0]) < sizeof(uint64_t), "T must be <8 bytes");
+
+  // mask to get the largest value. E.g. for uint8_t, mask will be 255.
+  const auto mask = (1ULL << (sizeof(input_data[0]) * CHAR_BIT)) - 1;
+  const uint64_t incr = kAlphaPresent ? 4 : 3;
+  for (uint64_t i = 0; i < size; i += incr) {
+    output_data[i] = mask - input_data[i];
+    output_data[i + 1] = mask - input_data[i + 1];
+    output_data[i + 2] = mask - input_data[i + 2];
+    if constexpr (kAlphaPresent) {
+      output_data[i + 3] = input_data[i + 3];
+    }
+  }
+}
+
+extern "C" {
+
+std::vector<amdinfer::Tensor> getInputs() {
+  // intentionally return an empty shape, indicating an unknown output tensor
+  // size even though it is known in this case. This forces the worker to use
+  // different logic to invoke the run method.
+  return {};
+}
+
+std::vector<amdinfer::Tensor> getOutputs() { return {}; }
+
+// Support up to Full HD
+const auto kMaxImageHeight = 1080;
+const auto kMaxImageWidth = 1920;
+const auto kMaxImageChannels = 3;
+const auto kMaxImageSize = kMaxImageHeight * kMaxImageWidth * kMaxImageChannels;
+
+amdinfer::BatchPtr run(amdinfer::Batch* batch) {
+  amdinfer::Logger logger{amdinfer::Loggers::Server};
+
+  auto new_batch = batch->propagate();
+  const auto batch_size = batch->size();
+  const auto data_size = amdinfer::DataType("Uint8").size();
+
+  std::vector<amdinfer::BufferPtr> input_buffers;
+  input_buffers.emplace_back(std::make_unique<amdinfer::VectorBuffer>(
+    kMaxImageSize * batch_size * data_size));
+
+  for (unsigned int j = 0; j < batch_size; j++) {
+    const auto& req = batch->getRequest(j);
+#ifdef AMDINFER_ENABLE_TRACING
+    const auto& trace = batch->getTrace(j);
+    trace->startSpan("invert_image");
+#endif
+    const auto& inputs = req->getInputs();
+    auto input_shape = inputs[0].getShape();
+
+    auto input_size = amdinfer::util::containerProduct(input_shape);
+
+    auto new_request = req->propagate();
+    auto* data_ptr = input_buffers.at(0)->data(j * kMaxImageSize * data_size);
+    new_request->addInputTensor(data_ptr, input_shape,
+                                amdinfer::DataType::Uint8, "output");
+    invert<uint8_t*, false>(inputs[0].getData(), data_ptr, input_size);
+
+    new_batch->addRequest(new_request);
+    new_batch->setModel(j, "invert_image");
+
+#ifdef AMDINFER_ENABLE_TRACING
+    trace->endSpan();
+#endif
+  }
+
+  new_batch->setBuffers(std::move(input_buffers), {});
+
+  return new_batch;
+}
+
+}  // extern "C"
diff --git a/src/amdinfer/servers/server.cpp b/src/amdinfer/servers/server.cpp
index 6646d3933..c2d82f119 100644
--- a/src/amdinfer/servers/server.cpp
+++ b/src/amdinfer/servers/server.cpp
@@ -62,7 +62,9 @@ void terminate() {
 
 Server::Server() {
   initializeServerLogging();
-  this->impl_ = std::make_unique<Server::ServerImpl>();
+  impl_ = std::make_unique<Server::ServerImpl>();
+
+  impl_->state.workerLoad("responder", {});
 
 #ifdef AMDINFER_ENABLE_TRACING
   startTracer();
@@ -82,6 +84,7 @@ Server::Server() {
 }
 
 Server::~Server() {
+  impl_->state.workerUnload("responder");
   stopHttp();
   stopGrpc();
   terminate();
diff --git a/src/amdinfer/util/containers.hpp b/src/amdinfer/util/containers.hpp
index 7354e8c7e..a769829e3 100644
--- a/src/amdinfer/util/containers.hpp
+++ b/src/amdinfer/util/containers.hpp
@@ -36,6 +36,10 @@ inline auto containerProduct(Iter begin, Iter end) {
   static_assert(std::is_arithmetic_v<value_type>,
                 "The container must contain an arithmetic type");
 
+  // if the range is empty
+  if (begin == end) {
+    return static_cast<value_type>(0);
+  }
   value_type init = 1;
   return std::accumulate(begin, end, init, std::multiplies<>());
 }
diff --git a/src/amdinfer/util/memory.hpp b/src/amdinfer/util/memory.hpp
index 0d1de5db8..edfde5220 100644
--- a/src/amdinfer/util/memory.hpp
+++ b/src/amdinfer/util/memory.hpp
@@ -34,8 +34,11 @@ std::byte *copy(const T &src, std::byte *dst,
     std::memcpy(dst, src, count);
     return dst + count;
   }
-  std::memcpy(dst, &src, sizeof(T));
-  return dst + sizeof(T);
+  if (count == 0) {
+    count = sizeof(T);
+  }
+  std::memcpy(dst, &src, count);
+  return dst + count;
 }
 
 }  // namespace amdinfer::util
diff --git a/src/amdinfer/util/string.hpp b/src/amdinfer/util/string.hpp
index 458c988ff..9b1a72aa9 100644
--- a/src/amdinfer/util/string.hpp
+++ b/src/amdinfer/util/string.hpp
@@ -46,6 +46,10 @@ inline bool startsWith(std::string_view str, std::string_view prefix) {
          0 == str.compare(0, prefix.size(), prefix);
 }
 
+inline bool contains(std::string_view str, std::string_view search) {
+  return str.find(search) != std::string::npos;
+}
+
 inline std::vector<std::string> split(std::string_view str,
                                       std::string_view delimiter) {
   auto last = 0UL;
diff --git a/src/amdinfer/workers/CMakeLists.txt b/src/amdinfer/workers/CMakeLists.txt
index 14df6f526..e50f8d027 100644
--- a/src/amdinfer/workers/CMakeLists.txt
+++ b/src/amdinfer/workers/CMakeLists.txt
@@ -15,7 +15,7 @@
 
 include(GNUInstallDirs)
 
-set(workers Echo EchoMulti InvertImage InvertVideo CPlusPlus)
+set(workers InvertVideo CPlusPlus Responder)
 
 if(${AMDINFER_ENABLE_VITIS})
   list(APPEND workers Xmodel)
@@ -61,9 +61,6 @@ foreach(worker ${workers})
   list(APPEND WORKER_TARGETS ${target})
 endforeach()
 
-target_link_libraries(
-  workerInvertimage PRIVATE base64 opencv_core opencv_imgcodecs
-)
 target_link_libraries(
   workerInvertvideo PRIVATE base64 opencv_core opencv_imgcodecs opencv_videoio
 )
@@ -120,13 +117,6 @@ if(${AMDINFER_ENABLE_AKS})
     workerResnet50stream PRIVATE opencv_imgproc opencv_videoio
   )
   target_link_libraries(workerAksdetectstream PRIVATE opencv_videoio)
-
-  # AKS Worker
-  add_library(workerAks SHARED aks.cpp)
-  target_link_libraries(workerAks PRIVATE buffers aks vart-runner xir timer)
-  target_include_directories(workerAks PRIVATE ${AMDINFER_INCLUDE_DIRS})
-  set_target_options(workerAks)
-  list(APPEND WORKER_TARGETS workerAks)
 else()
   message(STATUS "AKS not enabled, skipping AKS workers")
 endif()
diff --git a/src/amdinfer/workers/aks.cpp b/src/amdinfer/workers/aks.cpp
deleted file mode 100644
index 6fb74b67f..000000000
--- a/src/amdinfer/workers/aks.cpp
+++ /dev/null
@@ -1,217 +0,0 @@
-// Copyright 2021 Xilinx, Inc.
-// Copyright 2022 Advanced Micro Devices, Inc.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-/**
- * @file
- * @brief Implements the Aks worker
- */
-
-#include <aks/AksSysManagerExt.h>  // for SysManagerExt
-#include <aks/AksTensorBuffer.h>   // for AksTensorBuffer
-
-#include <algorithm>               // for max
-#include <cstddef>                 // for size_t, byte
-#include <cstdint>                 // for int32_t
-#include <cstring>                 // for memcpy
-#include <future>                  // for future
-#include <memory>                  // for unique_ptr, allocator
-#include <ratio>                   // for micro
-#include <string>                  // for string, operator+, char...
-#include <thread>                  // for thread
-#include <utility>                 // for move, pair
-#include <vart/tensor_buffer.hpp>  // for TensorBuffer
-#include <vector>                  // for vector
-#include <xir/tensor/tensor.hpp>   // for Tensor
-#include <xir/util/data_type.hpp>  // for create_data_type
-
-#include "amdinfer/batching/batcher.hpp"  // for BatchPtr, Batch, BatchP...
-#include "amdinfer/build_options.hpp"     // for AMDINFER_ENABLE_TRACING
-#include "amdinfer/core/data_types.hpp"   // for DataType, DataType::Fp32
-#include "amdinfer/core/exceptions.hpp"   // for external_error
-#include "amdinfer/core/inference_request.hpp"   // for InferenceRequest
-#include "amdinfer/core/inference_response.hpp"  // for InferenceResponse
-#include "amdinfer/core/parameters.hpp"          // for ParameterMap
-#include "amdinfer/declarations.hpp"         // for BufferPtrs, InferenceRe...
-#include "amdinfer/observation/logging.hpp"  // for Logger
-#include "amdinfer/observation/metrics.hpp"  // for Metrics, MetricSummaryIDs
-#include "amdinfer/observation/tracing.hpp"  // for Trace
-#include "amdinfer/util/parse_env.hpp"       // for autoExpandEnvironmentVa...
-#include "amdinfer/util/thread.hpp"          // for setThreadName
-#include "amdinfer/util/timer.hpp"           // for Timer
-#include "amdinfer/workers/worker.hpp"       // for Worker, kNumBufferAuto
-
-namespace AKS {  // NOLINT(readability-identifier-naming)
-class AIGraph;
-}  // namespace AKS
-
-namespace amdinfer::workers {
-
-/**
- * @brief The Aks worker is a simple worker that accepts a single uint32_t
- * argument and adds 1 to it and returns. It accepts multiple input tensors and
- * returns the corresponding number of output tensors.
- *
- */
-class Aks : public Worker {
- public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
-  [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
-
- private:
-  void doInit(ParameterMap* parameters) override;
-  void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
-  void doRelease() override;
-  void doDestroy() override;
-
-  /// the AKS system manager
-  AKS::SysManagerExt* sys_manager_ = nullptr;
-  /// the corresponding graph to the name
-  AKS::AIGraph* graph_ = nullptr;
-};
-
-std::thread Aks::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&Aks::run, this, input_queue);
-}
-
-std::vector<MemoryAllocators> Aks::getAllocators() const {
-  return {MemoryAllocators::Cpu};
-}
-
-void Aks::doInit(ParameterMap* parameters) {
-  // arbitrarily set the default batch size to 1
-  const int default_batch_size = 1;
-
-  /// Get AKS System Manager instance
-  this->sys_manager_ = AKS::SysManagerExt::getGlobal();
-
-  auto batch_size = default_batch_size;
-  if (parameters->has("batch_size")) {
-    batch_size = parameters->get<int32_t>("batch_size");
-  }
-  this->batch_size_ = batch_size;
-}
-
-void Aks::doAcquire(ParameterMap* parameters) {
-  std::string path{"${AKS_ROOT}/graph_zoo/graph_adder.json"};
-  if (parameters->has("aks_graph")) {
-    path = parameters->get<std::string>("aks_graph");
-  }
-  util::autoExpandEnvironmentVariables(path);
-  this->sys_manager_->loadGraphs(path);
-
-  std::string graph_name = "graph_adder";
-  this->graph_ = sys_manager_->getGraph(graph_name);
-  if (this->graph_ == nullptr) {
-    throw external_error("AKS graph " + graph_name + " not found");
-  }
-
-  this->metadata_.addInputTensor("input", {this->batch_size_, 1},
-                                 DataType::Fp32);
-  this->metadata_.addOutputTensor("output", {this->batch_size_, 1},
-                                  DataType::Fp32);
-  this->metadata_.setName(graph_name);
-}
-
-void Aks::doRun(BatchPtrQueue* input_queue) {
-  util::setThreadName("Aks");
-#ifdef AMDINFER_ENABLE_LOGGING
-  const auto& logger = this->getLogger();
-#endif
-
-  while (true) {
-    BatchPtr batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
-    AMDINFER_LOG_INFO(logger, "Got request in aks");
-    for (unsigned int j = 0; j < batch->size(); j++) {
-      const auto& req = batch->getRequest(static_cast<int>(j));
-#ifdef AMDINFER_ENABLE_TRACING
-      const auto& trace = batch->getTrace(static_cast<int>(j));
-      trace->startSpan("aks");
-#endif
-      InferenceResponse resp;
-      resp.setID(req->getID());
-      resp.setModel("aks");
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-
-      for (auto& input : inputs) {
-        auto* input_buffer = input.getData();
-
-        float value = 0.0F;
-        float* data = nullptr;
-        data = static_cast<float*>(input_buffer);
-        value = *data;
-
-        std::vector<std::unique_ptr<vart::TensorBuffer>> v;
-        v.emplace_back(std::make_unique<AKS::AksTensorBuffer>(
-          xir::Tensor::create("aks-echo", {1}, xir::create_data_type<int>())));
-
-        auto* data_in_ptr = reinterpret_cast<float*>(v[0]->data().first);
-        data_in_ptr[0] = value;
-
-        std::future<std::vector<std::unique_ptr<vart::TensorBuffer>>> future =
-          this->sys_manager_->enqueueJob(this->graph_, "", std::move(v),
-                                         nullptr);
-
-        auto out_data_descriptor = future.get();
-
-        value =
-          (reinterpret_cast<float*>(out_data_descriptor[0]->data().first))[0];
-
-        InferenceResponseOutput output;
-        output.setDatatype(DataType::Fp32);
-        output.setName("aks");
-        output.setShape({1});
-        std::vector<std::byte> buffer;
-        buffer.resize(sizeof(float));
-        memcpy(buffer.data(), &value, sizeof(float));
-        output.setData(std::move(buffer));
-        resp.addOutput(output);
-      }
-
-#ifdef AMDINFER_ENABLE_METRICS
-      util::Timer timer{batch->getTime(j)};
-      timer.stop();
-      auto duration = timer.count<std::micro>();
-      Metrics::getInstance().observeSummary(MetricSummaryIDs::RequestLatency,
-                                            duration);
-#endif
-#ifdef AMDINFER_ENABLE_TRACING
-      auto context = trace->propagate();
-      resp.setContext(std::move(context));
-#endif
-      req->runCallbackOnce(resp);
-    }
-  }
-  AMDINFER_LOG_INFO(logger, "Aks ending");
-}
-
-void Aks::doRelease() {}
-void Aks::doDestroy() {}
-
-}  // namespace amdinfer::workers
-
-extern "C" {
-// using smart pointer here may cause problems inside shared object so managing
-// manually
-amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::Aks("AKS", "AKS");
-}
-}  // extern C
diff --git a/src/amdinfer/workers/aks_detect.cpp b/src/amdinfer/workers/aks_detect.cpp
index 90ff6ec9b..5ab793efd 100644
--- a/src/amdinfer/workers/aks_detect.cpp
+++ b/src/amdinfer/workers/aks_detect.cpp
@@ -50,9 +50,9 @@
 #include "amdinfer/declarations.hpp"         // for BufferPtrs, InferenceRe...
 #include "amdinfer/observation/logging.hpp"  // for Logger
 #include "amdinfer/observation/metrics.hpp"  // for Metrics, MetricSummaryIDs
-#include "amdinfer/observation/tracing.hpp"  // for Trace
 #include "amdinfer/util/base64.hpp"          // for base64_decode
 #include "amdinfer/util/containers.hpp"      // for containerProduct
+#include "amdinfer/util/memory.hpp"          // for copy
 #include "amdinfer/util/parse_env.hpp"       // for autoExpandEnvironmentVa...
 #include "amdinfer/util/thread.hpp"          // for setThreadName
 #include "amdinfer/util/timer.hpp"           // for Timer
@@ -69,16 +69,15 @@ namespace amdinfer::workers {
  * classification IDs
  *
  */
-class AksDetect : public Worker {
+class AksDetect : public SingleThreadedWorker {
  public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
+  using SingleThreadedWorker::SingleThreadedWorker;
   [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
 
  private:
   void doInit(ParameterMap* parameters) override;
   void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
+  BatchPtr doRun(Batch* batch, const MemoryPool* pool) override;
   void doRelease() override;
   void doDestroy() override;
 
@@ -87,10 +86,6 @@ class AksDetect : public Worker {
   AKS::AIGraph* graph_ = nullptr;
 };
 
-std::thread AksDetect::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&AksDetect::run, this, input_queue);
-}
-
 std::vector<MemoryAllocators> AksDetect::getAllocators() const {
   return {MemoryAllocators::Cpu};
 }
@@ -98,7 +93,7 @@ std::vector<MemoryAllocators> AksDetect::getAllocators() const {
 void AksDetect::doInit(ParameterMap* parameters) {
   this->sys_manager_ = AKS::SysManagerExt::getGlobal();
 
-  // arbitrarily choose 4 as the default
+  // choose 4 as the default because some DPUs have it fixed to 4
   const auto default_batch_size = 4;
   auto batch_size = default_batch_size;
   if (parameters->has("batch_size")) {
@@ -106,9 +101,12 @@ void AksDetect::doInit(ParameterMap* parameters) {
   }
   this->batch_size_ = batch_size;
 
-  std::string graph_name{"yolov3"};
+  std::string graph_name;
   if (parameters->has("aks_graph_name")) {
     graph_name = parameters->get<std::string>("aks_graph_name");
+  } else {
+    throw invalid_argument(
+      "aks_graph_name must be specified in the parameters");
   }
   this->graph_name_ = graph_name;
 }
@@ -140,171 +138,107 @@ void AksDetect::doAcquire(ParameterMap* parameters) {
   this->metadata_.addInputTensor(
     "input", {this->batch_size_, kImageHeight, kImageWidth, kImageChannels},
     DataType::Int8);
-  // TODO(varunsh): what should we return here?
-  this->metadata_.addOutputTensor("output", {0}, DataType::Uint32);
+  this->metadata_.addOutputTensor("output", {0, 6}, DataType::FP32);
   this->metadata_.setName(this->graph_name_);
 }
 
-void AksDetect::doRun(BatchPtrQueue* input_queue) {
-  util::setThreadName("AksDetect");
+BatchPtr AksDetect::doRun(Batch* batch, const MemoryPool* pool) {
 #ifdef AMDINFER_ENABLE_LOGGING
-  const auto& logger = this->getLogger();
+  const auto logger = this->getLogger();
 #endif
+  const auto batch_size = batch->size();
 
-  while (true) {
-    BatchPtr batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
-    AMDINFER_LOG_INFO(logger, "Got request in AksDetect");
-    std::vector<InferenceResponse> responses;
-    responses.reserve(batch->getInputSize());
-
-    std::vector<std::unique_ptr<vart::TensorBuffer>> v;
-    v.reserve(batch->size());
-
-    size_t tensor_count = 0;
-    for (unsigned int j = 0; j < batch->size(); j++) {
-      const auto& req = batch->getRequest(static_cast<int>(j));
-#ifdef AMDINFER_ENABLE_TRACING
-      const auto& trace = batch->getTrace(static_cast<int>(j));
-      trace->startSpan("AksDetect");
-#endif
-      auto& resp = responses.emplace_back();
-      resp.setID(req->getID());
-      resp.setModel(this->graph_name_);
-
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-
-      uint64_t input_size = 0;
-      for (auto& input : inputs) {
-        auto* input_buffer = input.getData();
-
-        auto input_shape = input.getShape();
-
-        input_size = util::containerProduct(input_shape);
-        auto input_dtype = input.getDatatype();
-
-        std::vector<int> tensor_shape = {static_cast<int>(this->batch_size_)};
-        if (input_dtype == DataType::Uint8) {
-          if (tensor_count == 0) {
-            tensor_shape.insert(tensor_shape.end(), input_shape.begin(),
-                                input_shape.end());
-            v.emplace_back(std::make_unique<AKS::AksTensorBuffer>(
-              xir::Tensor::create(this->graph_name_, tensor_shape,
-                                  xir::create_data_type<unsigned char>())));
-          }
-          /// Copy input to AKS Buffers: Find a better way to share buffers
-          memcpy(reinterpret_cast<uint8_t*>(v[0]->data().first) +
-                   ((tensor_count % this->batch_size_) * input_size),
-                 input_buffer, input_size);
-        } else if (input_dtype == DataType::String) {
-          auto* idata = static_cast<char*>(input_buffer);
-          auto decoded_str = util::base64Decode(idata, input_size);
-          std::vector<char> data(decoded_str.begin(), decoded_str.end());
-          cv::Mat img = cv::imdecode(data, cv::IMREAD_UNCHANGED);
-          if (img.empty()) {
-            const char* error = "Decoded image is empty";
-            AMDINFER_LOG_ERROR(logger, error);
-            req->runCallbackError(error);
-            continue;
-          }
-          // set size to actual size of image instead of size of base64 str
-          input_size = img.step[0] * img.rows;
-
-          if (tensor_count == 0) {
-            tensor_shape.insert(
-              tensor_shape.end(),
-              {img.size().height, img.size().width, kImageChannels});
-            v.emplace_back(std::make_unique<AKS::AksTensorBuffer>(
-              xir::Tensor::create(this->graph_name_, tensor_shape,
-                                  xir::create_data_type<unsigned char>())));
-          }
-
-          // TODO(varunsh): assuming decoded image will be right size
-          memcpy(reinterpret_cast<uint8_t*>(
-                   v[0]->data().first +
-                   ((tensor_count % this->batch_size_) * input_size)),
-                 img.data, input_size);
-        }
-        tensor_count = (tensor_count + 1) % this->batch_size_;
-      }
-    }
+  std::vector<std::unique_ptr<vart::TensorBuffer>> v;
+  v.reserve(batch_size);
 
-    std::future<std::vector<std::unique_ptr<vart::TensorBuffer>>> future =
-      this->sys_manager_->enqueueJob(this->graph_, "", std::move(v), nullptr);
+  size_t tensor_count = 0;
+  for (unsigned int j = 0; j < batch_size; j++) {
+    const auto& req = batch->getRequest(static_cast<int>(j));
 
-    auto out_data_descriptor = future.get();
+    auto inputs = req->getInputs();
 
-    auto shape = out_data_descriptor[0]->get_tensor()->get_shape();
-    // shape[0] is number of boxes, shape[1] is the number of values per box
-    int size = shape.size() > 1 ? shape[0] * shape[1] : 0;
+    uint64_t input_size = 0;
+    for (auto& input : inputs) {
+      auto* input_buffer = input.getData();
 
-    auto* top_k_data =
-      reinterpret_cast<float*>(out_data_descriptor[0]->data().first);
-    auto my_data_2 = std::vector<std::vector<std::byte>>();
-    my_data_2.resize(this->batch_size_);
-    for (int i = 0; i < size; i += kAkdDetectResponseSize) {
-      auto batch_id = static_cast<int>(top_k_data[i]);
-      auto len = my_data_2[batch_id].size();
-      my_data_2[batch_id].resize(len + sizeof(DetectResponse));
+      auto input_shape = input.getShape();
 
-      memcpy(my_data_2[batch_id].data() + len, &(top_k_data[i + 1]),
-             sizeof(DetectResponse));
-    }
+      input_size = util::containerProduct(input_shape);
 
-    tensor_count = 0;
-    for (unsigned int k = 0; k < batch->size(); k++) {
-      const auto& req = batch->getRequest(static_cast<int>(k));
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-      auto& resp = responses[k];
-
-      // int offset = 0;
-      for (unsigned int i = 0; i < inputs.size(); i++) {
-        InferenceResponseOutput output;
-
-        output.setDatatype(DataType::Fp32);
-
-        std::string output_name;
-        if (i < outputs.size()) {
-          output_name = outputs[i].getName();
-        }
-
-        if (output_name.empty()) {
-          output.setName(inputs[0].getName());
-        } else {
-          output.setName(output_name);
-        }
-
-        output.setShape(
-          {kAkdDetectResponseSize - 1,
-           my_data_2[tensor_count].size() / sizeof(DetectResponse)});
-        output.setData(std::move(my_data_2[tensor_count]));
-        resp.addOutput(output);
-        tensor_count++;
+      std::vector<int> tensor_shape = {static_cast<int>(this->batch_size_)};
+      if (tensor_count == 0) {
+        tensor_shape.insert(tensor_shape.end(), input_shape.begin(),
+                            input_shape.end());
+        v.emplace_back(std::make_unique<AKS::AksTensorBuffer>(
+          xir::Tensor::create(this->graph_name_, tensor_shape,
+                              xir::create_data_type<unsigned char>())));
       }
+      /// Copy input to AKS Buffers: Find a better way to share buffers
+      memcpy(reinterpret_cast<uint8_t*>(v[0]->data().first) +
+               ((tensor_count % this->batch_size_) * input_size),
+             input_buffer, input_size);
+      tensor_count = (tensor_count + 1) % this->batch_size_;
+    }
+  }
 
-#ifdef AMDINFER_ENABLE_METRICS
-      util::Timer timer{batch->getTime(k)};
-      timer.stop();
-      auto duration = timer.count<std::micro>();
-      Metrics::getInstance().observeSummary(MetricSummaryIDs::RequestLatency,
-                                            duration);
-#endif
+  std::future<std::vector<std::unique_ptr<vart::TensorBuffer>>> future =
+    this->sys_manager_->enqueueJob(this->graph_, "", std::move(v), nullptr);
 
-#ifdef AMDINFER_ENABLE_TRACING
-      const auto& trace = batch->getTrace(k);
-      auto context = trace->propagate();
-      resp.setContext(std::move(context));
-#endif
-      req->runCallbackOnce(resp);
+  auto aks_output = future.get();
+
+  assert(aks_output.size() == 1);
+  const auto& aks_tensor_buffer = aks_output.at(0);
+
+  auto aks_tensor_shape = aks_tensor_buffer->get_tensor()->get_shape();
+  assert(aks_tensor_shape.empty() || aks_tensor_shape.size() == 2);
+  // shape[0] is number of boxes, shape[1] is the number of values per box
+  const int size = util::containerProduct(aks_tensor_shape);
+  const auto* aks_data =
+    reinterpret_cast<float*>(aks_tensor_buffer->data().first);
+
+  // get the max number of boxes for any of the images in the batch and use
+  // that to size the next batch
+  std::vector<size_t> boxes;
+  boxes.resize(batch_size);
+  for (int i = 0; i < size; i += kAksDetectResponseSize) {
+    auto batch_id = static_cast<int>(aks_data[i]);
+    boxes.at(batch_id)++;
+  }
+  const auto max_boxes = *std::max_element(boxes.begin(), boxes.end());
+
+  auto new_batch = batch->propagate();
+  std::vector<BufferPtr> input_buffers;
+  input_buffers.push_back(
+    pool->get(next_allocators_,
+              Tensor{"name", {max_boxes, kDetectResponseSize}, DataType::Fp32},
+              batch_size));
+
+  for (auto i = 0U; i < batch_size; ++i) {
+    const auto& req = batch->getRequest(i);
+    auto new_request = req->propagate();
+    new_request->addInputTensor(InferenceRequestInput{
+      nullptr, {boxes.at(i), kDetectResponseSize}, DataType::Fp32, ""});
+    auto* data_ptr = input_buffers.at(0)->data(
+      i * max_boxes * kDetectResponseSize * DataType("Fp32").size());
+    new_request->setInputTensorData(0, data_ptr);
+
+    for (auto j = 0; j < size; j += kAksDetectResponseSize) {
+      auto batch_id = static_cast<size_t>(aks_data[j]);
+      if (batch_id == i) {
+        data_ptr =
+          util::copy(&(aks_data[j + 1]), static_cast<std::byte*>(data_ptr),
+                     sizeof(DetectResponse));
+      }
     }
-    this->returnInputBuffers(std::move(batch));
+
+    new_batch->addRequest(new_request);
+
+    new_batch->setModel(i, graph_name_);
   }
-  AMDINFER_LOG_INFO(logger, "AksDetect ending");
+
+  new_batch->setBuffers(std::move(input_buffers), {});
+
+  return new_batch;
 }
 
 void AksDetect::doRelease() {}
@@ -316,6 +250,6 @@ extern "C" {
 // using smart pointer here may cause problems inside shared object so managing
 // manually
 amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::AksDetect("AksDetect", "AKS");
+  return new amdinfer::workers::AksDetect("AksDetect", "AKS", true);
 }
 }  // extern C
diff --git a/src/amdinfer/workers/aks_detect.hpp b/src/amdinfer/workers/aks_detect.hpp
index 44252b184..5848c029a 100644
--- a/src/amdinfer/workers/aks_detect.hpp
+++ b/src/amdinfer/workers/aks_detect.hpp
@@ -31,8 +31,11 @@ struct DetectResponse {
   float h;
 };
 
-// the number of float values per response
-const int kAkdDetectResponseSize = 7;
+// the number of float values in DetectResponse
+const auto kDetectResponseSize = 6;
+
+// the number of float values per response from AKS. +1 for the batch ID
+const int kAksDetectResponseSize = kDetectResponseSize + 1;
 
 }  // namespace amdinfer::workers
 
diff --git a/src/amdinfer/workers/aks_detect_stream.cpp b/src/amdinfer/workers/aks_detect_stream.cpp
index 475561d98..1fb41a120 100644
--- a/src/amdinfer/workers/aks_detect_stream.cpp
+++ b/src/amdinfer/workers/aks_detect_stream.cpp
@@ -70,20 +70,19 @@ std::string constructMessage(const std::string& key, const std::string& data,
 namespace workers {
 
 /**
- * @brief The AksDetectStream worker is a simple worker that accepts an path to
- * a video and sends the inverted frames back to the client over a websocket.
+ * @brief The AksDetectStream worker is a streaming (over WebSocket) worker that
+ * runs a detection-based model on a video with AKS.
  *
  */
-class AksDetectStream : public Worker {
+class AksDetectStream : public SingleThreadedWorker {
  public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
+  using SingleThreadedWorker::SingleThreadedWorker;
   [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
 
  private:
   void doInit(ParameterMap* parameters) override;
   void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
+  BatchPtr doRun(Batch* batch, const MemoryPool* pool) override;
   void doRelease() override;
   void doDestroy() override;
 
@@ -91,16 +90,11 @@ class AksDetectStream : public Worker {
   AKS::AIGraph* graph_ = nullptr;
 };
 
-std::thread AksDetectStream::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&AksDetectStream::run, this, input_queue);
-}
-
 std::vector<MemoryAllocators> AksDetectStream::getAllocators() const {
   return {MemoryAllocators::Cpu};
 }
 
-void AksDetectStream::doInit(ParameterMap* parameters) {
-  (void)parameters;  // suppress unused variable warning
+void AksDetectStream::doInit([[maybe_unused]] ParameterMap* parameters) {
   constexpr auto kBatchSize = 4;
 
   /// Get AKS System Manager instance
@@ -114,17 +108,21 @@ constexpr auto kImageHeight = 1080;
 constexpr auto kImageChannels = 3;
 
 void AksDetectStream::doAcquire(ParameterMap* parameters) {
-  std::string path{
-    "${AKS_ROOT}/graph_zoo/graph_yolov3_u200_u250_amdinfer.json"};
+  std::string path;
   if (parameters->has("aks_graph")) {
     path = parameters->get<std::string>("aks_graph");
+  } else {
+    throw invalid_argument("aks_graph must be specified in the parameters");
   }
   util::autoExpandEnvironmentVariables(path);
   this->sys_manager_->loadGraphs(path);
 
-  std::string graph_name{"yolov3"};
+  std::string graph_name;
   if (parameters->has("aks_graph_name")) {
     graph_name = parameters->get<std::string>("aks_graph_name");
+  } else {
+    throw invalid_argument(
+      "aks_graph_name must be specified in the parameters");
   }
   this->graph_ = this->sys_manager_->getGraph(graph_name);
 
@@ -135,179 +133,114 @@ void AksDetectStream::doAcquire(ParameterMap* parameters) {
   this->metadata_.addOutputTensor("output", {0}, DataType::Uint32);
   this->metadata_.setName(graph_name);
 }
-
-void AksDetectStream::doRun(BatchPtrQueue* input_queue) {
-  util::setThreadName("AksDetectStream");
+BatchPtr AksDetectStream::doRun(Batch* batch,
+                                [[maybe_unused]] const MemoryPool* pool) {
 #ifdef AMDINFER_ENABLE_LOGGING
   const auto& logger = this->getLogger();
 #endif
 
-  while (true) {
-    BatchPtr batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
-
-    AMDINFER_LOG_INFO(logger, "Got request in AksDetectStream");
-    for (unsigned int k = 0; k < batch->size(); k++) {
-      const auto& req = batch->getRequest(static_cast<int>(k));
-#ifdef AMDINFER_ENABLE_TRACING
-      const auto& trace = batch->getTrace(static_cast<int>(k));
-      trace->startSpan("aks_detect_stream");
-#endif
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-      auto key = req->getParameters().get<std::string>("key");
-      for (auto& input : inputs) {
-        auto* input_buffer = input.getData();
-
-        const auto* idata = static_cast<char*>(input_buffer);
-        std::string data{idata, input.getSize()};
-
-        cv::VideoCapture cap(data);  // open the video file
-        if (!cap.isOpened()) {       // check if we succeeded
-          const char* error = "Cannot open video file";
-          AMDINFER_LOG_ERROR(logger, error);
-          req->runCallbackError(error);
-          continue;
-        }
+  for (unsigned int k = 0; k < batch->size(); k++) {
+    const auto& req = batch->getRequest(static_cast<int>(k));
+    auto inputs = req->getInputs();
+    auto outputs = req->getOutputs();
+    auto key = req->getParameters().get<std::string>("key");
+    for (auto& input : inputs) {
+      auto* input_buffer = input.getData();
+
+      const auto* idata = static_cast<char*>(input_buffer);
+      std::string data{idata, input.getSize()};
+
+      cv::VideoCapture cap(data);  // open the video file
+      if (!cap.isOpened()) {       // check if we succeeded
+        const char* error = "Cannot open video file";
+        AMDINFER_LOG_ERROR(logger, error);
+        req->runCallbackError(error);
+        continue;
+      }
 
-        InferenceResponse resp;
-        resp.setID(req->getID());
-        resp.setModel("aks_detect_stream");
+      InferenceResponse resp;
+      resp.setID(req->getID());
+      resp.setModel("aks_detect_stream");
 
-        // contains the number of frames in the video;
-        auto count = static_cast<size_t>(
-          cap.get(cv::VideoCaptureProperties::CAP_PROP_FRAME_COUNT));
-        if (input.getParameters().has("count")) {
-          auto requested_count = input.getParameters().get<int32_t>("count");
-          count = std::min(count, static_cast<size_t>(requested_count));
-        }
-        double fps = cap.get(cv::VideoCaptureProperties::CAP_PROP_FPS);
-        auto video_width =
-          cap.get(cv::VideoCaptureProperties::CAP_PROP_FRAME_WIDTH);
-        auto video_height =
-          cap.get(cv::VideoCaptureProperties::CAP_PROP_FRAME_HEIGHT);
-
-        InferenceResponseOutput output;
-        output.setName("key");
-        output.setDatatype(DataType::String);
-        std::string metadata = "[" + std::to_string(video_width) + "," +
-                               std::to_string(video_height) + "]";
-        auto message = constructMessage(key, std::to_string(fps), metadata);
-        std::vector<std::byte> buffer;
-        buffer.resize(message.size());
-        memcpy(buffer.data(), message.data(), message.size());
-        output.setData(std::move(buffer));
-        output.setShape({message.size()});
-        resp.addOutput(output);
-        req->runCallback(resp);
-
-        // round to nearest multiple of batch size
-        auto count_adjusted = count - (count % this->batch_size_);
-        std::queue<
-          std::future<std::vector<std::unique_ptr<vart::TensorBuffer>>>>
-          futures;
-        std::queue<std::string> frames;
-        for (unsigned int frame_num = 0; frame_num < count_adjusted;
-             frame_num += this->batch_size_) {
-          std::vector<std::unique_ptr<vart::TensorBuffer>> v;
-          v.reserve(1);
+      // contains the number of frames in the video;
+      auto count = static_cast<size_t>(
+        cap.get(cv::VideoCaptureProperties::CAP_PROP_FRAME_COUNT));
+      if (input.getParameters().has("count")) {
+        auto requested_count = input.getParameters().get<int32_t>("count");
+        count = std::min(count, static_cast<size_t>(requested_count));
+      }
+      double fps = cap.get(cv::VideoCaptureProperties::CAP_PROP_FPS);
+      auto video_width =
+        cap.get(cv::VideoCaptureProperties::CAP_PROP_FRAME_WIDTH);
+      auto video_height =
+        cap.get(cv::VideoCaptureProperties::CAP_PROP_FRAME_HEIGHT);
+
+      InferenceResponseOutput output;
+      output.setName("key");
+      output.setDatatype(DataType::String);
+      std::string metadata = "[" + std::to_string(video_width) + "," +
+                             std::to_string(video_height) + "]";
+      auto message = constructMessage(key, std::to_string(fps), metadata);
+      std::vector<std::byte> buffer;
+      buffer.resize(message.size());
+      memcpy(buffer.data(), message.data(), message.size());
+      output.setData(std::move(buffer));
+      output.setShape({message.size()});
+      resp.addOutput(output);
+      req->runCallback(resp);
+
+      // round to nearest multiple of batch size
+      auto count_adjusted = count - (count % this->batch_size_);
+      std::queue<std::future<std::vector<std::unique_ptr<vart::TensorBuffer>>>>
+        futures;
+      std::queue<std::string> frames;
+      for (unsigned int frame_num = 0; frame_num < count_adjusted;
+           frame_num += this->batch_size_) {
+        std::vector<std::unique_ptr<vart::TensorBuffer>> v;
+        v.reserve(1);
 
 #ifdef AMDINFER_ENABLE_TRACING
-          trace->startSpan("enqueue_batch");
+        const auto& trace = batch->getTrace(k);
+        trace->startSpan("enqueue_batch");
 #endif
 
-          for (size_t i = 0; i < this->batch_size_; i++) {
-            cv::Mat frame;
-            cap >> frame;  // get the next frame from video
-            if (frame.empty()) {
-              i--;
-              continue;
-            }
-            if (i == 0) {
-              v.emplace_back(
-                std::make_unique<AKS::AksTensorBuffer>(xir::Tensor::create(
-                  "AksDetect-stream",
-                  {static_cast<int>(this->batch_size_), frame.size().height,
-                   frame.size().width, kImageChannels},
-                  xir::create_data_type<unsigned char>())));
-            }
-            // cv::resize(frame, frame, cv::Size(kImageWidth, kImageHeight));
-            auto input_size = frame.step[0] * frame.rows;
-            memcpy(
-              reinterpret_cast<uint8_t*>(v[0]->data().first) + (i * input_size),
-              frame.data, input_size);
-
-            std::vector<unsigned char> buf;
-            cv::imencode(".jpg", frame, buf);
-            const auto* enc_msg = reinterpret_cast<const char*>(buf.data());
-            std::string encoded = util::base64Encode(enc_msg, buf.size());
-            frames.push("data:image/jpg;base64," + encoded);
+        for (size_t i = 0; i < this->batch_size_; i++) {
+          cv::Mat frame;
+          cap >> frame;  // get the next frame from video
+          if (frame.empty()) {
+            i--;
+            continue;
           }
-          AMDINFER_LOG_INFO(logger, "Enqueuing in " + key);
-          futures.push(this->sys_manager_->enqueueJob(this->graph_, "",
-                                                      std::move(v), nullptr));
-#ifdef AMDINFER_ENABLE_TRACING
-          trace->endSpan();
-#endif
-          auto status = futures.front().wait_for(std::chrono::seconds(0));
-          if (status == std::future_status::ready) {
-            std::vector<std::unique_ptr<vart::TensorBuffer>>
-              out_data_descriptor = futures.front().get();
-            futures.pop();
-            auto* top_k_data =
-              reinterpret_cast<float*>(out_data_descriptor[0]->data().first);
-            auto shape = out_data_descriptor[0]->get_tensor()->get_shape();
-            std::vector<std::string> labels;
-            labels.reserve(this->batch_size_);
-            for (unsigned int j = 0; j < this->batch_size_; j++) {
-              labels.emplace_back("[");
-            }
-            for (int i = 0; i < shape[0] * shape[1];
-                 i += kAkdDetectResponseSize) {
-              auto batch_id = static_cast<int>(top_k_data[i]);
-              const auto* detect_response =
-                reinterpret_cast<DetectResponse*>(&(top_k_data[i + 1]));
-
-              labels[batch_id].append(R"({"fill": false, "box": [)");
-              labels[batch_id].append(std::to_string(detect_response->x) + ",");
-              labels[batch_id].append(std::to_string(detect_response->y) + ",");
-              labels[batch_id].append(std::to_string(detect_response->w) + ",");
-              labels[batch_id].append(std::to_string(detect_response->h));
-              labels[batch_id].append(R"(], "label": ")");
-              labels[batch_id].append(
-                std::to_string(detect_response->class_id) + "\"},");
-            }
-            for (unsigned int j = 0; j < this->batch_size_; j++) {
-              if (labels[j].size() > 1) {
-                labels[j].pop_back();  // trim trailing comma
-              }
-              labels[j] += "]";
-              InferenceResponse resp;
-              resp.setID(req->getID());
-              resp.setModel("invert_video");
-
-              InferenceResponseOutput output;
-              output.setName("image");
-              output.setDatatype(DataType::String);
-              auto message = constructMessage(key, frames.front(), labels[j]);
-              std::vector<std::byte> buffer;
-              buffer.resize(message.size());
-              memcpy(buffer.data(), message.data(), message.size());
-              output.setData(std::move(buffer));
-              output.setShape({message.size()});
-              resp.addOutput(output);
-              req->runCallback(resp);
-              frames.pop();
-            }
+          if (i == 0) {
+            v.emplace_back(
+              std::make_unique<AKS::AksTensorBuffer>(xir::Tensor::create(
+                "AksDetect-stream",
+                {static_cast<int>(this->batch_size_), frame.size().height,
+                 frame.size().width, kImageChannels},
+                xir::create_data_type<unsigned char>())));
           }
+          // cv::resize(frame, frame, cv::Size(kImageWidth, kImageHeight));
+          auto input_size = frame.step[0] * frame.rows;
+          memcpy(
+            reinterpret_cast<uint8_t*>(v[0]->data().first) + (i * input_size),
+            frame.data, input_size);
+
+          std::vector<unsigned char> buf;
+          cv::imencode(".jpg", frame, buf);
+          const auto* enc_msg = reinterpret_cast<const char*>(buf.data());
+          std::string encoded = util::base64Encode(enc_msg, buf.size());
+          frames.push("data:image/jpg;base64," + encoded);
         }
-        while (!futures.empty()) {
+        AMDINFER_LOG_INFO(logger, "Enqueuing in " + key);
+        futures.push(this->sys_manager_->enqueueJob(this->graph_, "",
+                                                    std::move(v), nullptr));
+#ifdef AMDINFER_ENABLE_TRACING
+        trace->endSpan();
+#endif
+        auto status = futures.front().wait_for(std::chrono::seconds(0));
+        if (status == std::future_status::ready) {
           std::vector<std::unique_ptr<vart::TensorBuffer>> out_data_descriptor =
             futures.front().get();
-          AMDINFER_LOG_INFO(logger, "Got future with key " + key);
           futures.pop();
           auto* top_k_data =
             reinterpret_cast<float*>(out_data_descriptor[0]->data().first);
@@ -318,7 +251,7 @@ void AksDetectStream::doRun(BatchPtrQueue* input_queue) {
             labels.emplace_back("[");
           }
           for (int i = 0; i < shape[0] * shape[1];
-               i += kAkdDetectResponseSize) {
+               i += kAksDetectResponseSize) {
             auto batch_id = static_cast<int>(top_k_data[i]);
             const auto* detect_response =
               reinterpret_cast<DetectResponse*>(&(top_k_data[i + 1]));
@@ -345,6 +278,7 @@ void AksDetectStream::doRun(BatchPtrQueue* input_queue) {
             output.setName("image");
             output.setDatatype(DataType::String);
             auto message = constructMessage(key, frames.front(), labels[j]);
+            std::vector<std::byte> buffer;
             buffer.resize(message.size());
             memcpy(buffer.data(), message.data(), message.size());
             output.setData(std::move(buffer));
@@ -355,9 +289,60 @@ void AksDetectStream::doRun(BatchPtrQueue* input_queue) {
           }
         }
       }
+      while (!futures.empty()) {
+        std::vector<std::unique_ptr<vart::TensorBuffer>> out_data_descriptor =
+          futures.front().get();
+        AMDINFER_LOG_INFO(logger, "Got future with key " + key);
+        futures.pop();
+        auto* top_k_data =
+          reinterpret_cast<float*>(out_data_descriptor[0]->data().first);
+        auto shape = out_data_descriptor[0]->get_tensor()->get_shape();
+        std::vector<std::string> labels;
+        labels.reserve(this->batch_size_);
+        for (unsigned int j = 0; j < this->batch_size_; j++) {
+          labels.emplace_back("[");
+        }
+        for (int i = 0; i < shape[0] * shape[1]; i += kAksDetectResponseSize) {
+          auto batch_id = static_cast<int>(top_k_data[i]);
+          const auto* detect_response =
+            reinterpret_cast<DetectResponse*>(&(top_k_data[i + 1]));
+
+          labels[batch_id].append(R"({"fill": false, "box": [)");
+          labels[batch_id].append(std::to_string(detect_response->x) + ",");
+          labels[batch_id].append(std::to_string(detect_response->y) + ",");
+          labels[batch_id].append(std::to_string(detect_response->w) + ",");
+          labels[batch_id].append(std::to_string(detect_response->h));
+          labels[batch_id].append(R"(], "label": ")");
+          labels[batch_id].append(std::to_string(detect_response->class_id) +
+                                  "\"},");
+        }
+        for (unsigned int j = 0; j < this->batch_size_; j++) {
+          if (labels[j].size() > 1) {
+            labels[j].pop_back();  // trim trailing comma
+          }
+          labels[j] += "]";
+          InferenceResponse resp;
+          resp.setID(req->getID());
+          resp.setModel("invert_video");
+
+          InferenceResponseOutput output;
+          output.setName("image");
+          output.setDatatype(DataType::String);
+          auto message = constructMessage(key, frames.front(), labels[j]);
+          buffer.resize(message.size());
+          memcpy(buffer.data(), message.data(), message.size());
+          output.setData(std::move(buffer));
+          output.setShape({message.size()});
+          resp.addOutput(output);
+          req->runCallback(resp);
+          frames.pop();
+        }
+      }
     }
   }
-  AMDINFER_LOG_INFO(logger, "AksDetectStream ending");
+
+  // okay because ensembles disabled for this worker
+  return nullptr;
 }
 
 void AksDetectStream::doRelease() {}
@@ -371,6 +356,7 @@ extern "C" {
 // using smart pointer here may cause problems inside shared object so managing
 // manually
 amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::AksDetectStream("AksDetectStream", "AKS");
+  return new amdinfer::workers::AksDetectStream("AksDetectStream", "AKS",
+                                                false);
 }
 }  // extern C
diff --git a/src/amdinfer/workers/c_plus_plus.cpp b/src/amdinfer/workers/c_plus_plus.cpp
index c82738cd0..b55b84cc7 100644
--- a/src/amdinfer/workers/c_plus_plus.cpp
+++ b/src/amdinfer/workers/c_plus_plus.cpp
@@ -103,16 +103,15 @@ namespace workers {
  * @brief The CPlusPlus worker can run a compiled C++ "model".
  *
  */
-class CPlusPlus : public Worker {
+class CPlusPlus : public SingleThreadedWorker {
  public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
+  using SingleThreadedWorker::SingleThreadedWorker;
   [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
 
  private:
   void doInit(ParameterMap* parameters) override;
   void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
+  BatchPtr doRun(Batch* batch, const MemoryPool* pool) override;
   void doRelease() override;
   void doDestroy() override;
 
@@ -130,71 +129,10 @@ class CPlusPlus : public Worker {
   };
 };
 
-std::thread CPlusPlus::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&CPlusPlus::run, this, input_queue);
-}
-
 std::vector<MemoryAllocators> CPlusPlus::getAllocators() const {
   return {MemoryAllocators::Cpu};
 }
 
-void respond(Batch* batch) {
-  const auto batch_size = batch->size();
-  for (unsigned int j = 0; j < batch_size; j++) {
-    const auto& req = batch->getRequest(j);
-#ifdef AMDINFER_ENABLE_TRACING
-    const auto& trace = batch->getTrace(j);
-    trace->startSpan("response");
-#endif
-    InferenceResponse resp;
-    resp.setID(req->getID());
-    resp.setModel("CPlusPlus");
-    auto inputs = req->getInputs();
-    auto outputs = req->getOutputs();
-    for (unsigned int i = 0; i < inputs.size(); i++) {
-      const auto& input = inputs[i];
-      const auto* input_buffer = input.getData();
-
-      InferenceResponseOutput output;
-      output.setDatatype(DataType::Uint32);
-      std::string output_name;
-      if (i < outputs.size()) {
-        output_name = outputs[i].getName();
-      }
-
-      if (output_name.empty()) {
-        output.setName(input.getName());
-      } else {
-        output.setName(output_name);
-      }
-      output.setShape(input.getShape());
-      std::vector<std::byte> buffer;
-      const auto size = input.getSize() * input.getDatatype().size();
-      buffer.resize(size);
-      memcpy(buffer.data(), input_buffer, size);
-      output.setData(std::move(buffer));
-      resp.addOutput(output);
-    }
-
-#ifdef AMDINFER_ENABLE_TRACING
-    auto context = trace->propagate();
-    resp.setContext(std::move(context));
-#endif
-
-    // respond back to the client
-    req->runCallbackOnce(resp);
-#ifdef AMDINFER_ENABLE_METRICS
-    Metrics::getInstance().incrementCounter(
-      MetricCounterIDs::PipelineEgressWorker);
-    util::Timer timer{batch->getTime(j)};
-    timer.stop();
-    auto duration = timer.count<std::micro>();
-    Metrics::getInstance().observeSummary(MetricSummaryIDs::RequestLatency,
-                                          duration);
-#endif
-  }
-}
-
 void CPlusPlus::doInit(ParameterMap* parameters) {
   constexpr auto kBatchSize = 1;
 
@@ -232,64 +170,45 @@ void CPlusPlus::doAcquire([[maybe_unused]] ParameterMap* parameters) {
   }
 }
 
-void CPlusPlus::doRun(BatchPtrQueue* input_queue) {
-  util::setThreadName("CPlusPlus");
-#ifdef AMDINFER_ENABLE_LOGGING
-  const auto& logger = this->getLogger();
-#endif
-
-  while (true) {
-    BatchPtr batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
+BatchPtr CPlusPlus::doRun(Batch* batch, const MemoryPool* pool) {
+  BatchPtr new_batch;
+  if (!(input_tensors_.empty() || output_tensors_.empty())) {
+    new_batch = batch->propagate();
+    std::vector<BufferPtr> input_buffers;
+    input_buffers.reserve(output_tensors_.size());
+    const auto batch_size = batch->size();
+    for (const auto& tensor : output_tensors_) {
+      input_buffers.push_back(pool->get(next_allocators_, tensor, batch_size));
     }
-    AMDINFER_LOG_INFO(logger, "Got request in CPlusPlus");
-#ifdef AMDINFER_ENABLE_METRICS
-    Metrics::getInstance().incrementCounter(
-      MetricCounterIDs::PipelineIngressWorker);
-#endif
-
-    auto new_batch = std::make_unique<Batch>();
-    if (!(input_tensors_.empty() || output_tensors_.empty())) {
-      std::vector<BufferPtr> input_buffers;
-      input_buffers.reserve(output_tensors_.size());
-      const auto batch_size = batch->size();
+    for (auto i = 0U; i < batch_size; ++i) {
+      const auto& req = batch->getRequest(i);
+      auto new_request = req->propagate();
+      int index = 0;
       for (const auto& tensor : output_tensors_) {
-        input_buffers.push_back(
-          pool_->get({MemoryAllocators::Cpu}, tensor, batch_size));
+        new_request->addInputTensor(InferenceRequestInput{tensor});
+        new_request->setInputTensorData(
+          index, input_buffers.at(index)->data(i * tensor.getSize() *
+                                               tensor.getDatatype().size()));
+        index++;
       }
-      for (auto i = 0U; i < batch_size; ++i) {
-        auto new_request = std::make_shared<InferenceRequest>();
-        int index = 0;
-        for (const auto& tensor : output_tensors_) {
-          new_request->addInputTensor(InferenceRequestInput{tensor});
-          new_request->setInputTensorData(
-            index, input_buffers.at(index)->data(i * tensor.getSize() *
-                                                 tensor.getDatatype().size()));
-          index++;
-        }
-        new_batch->addRequest(new_request);
-      }
-      new_batch->setBuffers(std::move(input_buffers), {});
-
-      auto* run_ptr = getFunction(handle_, "run");
-      auto* runModel =
-        reinterpret_cast<void (*)(amdinfer::Batch*, amdinfer::Batch*)>(run_ptr);
-
-      runModel(batch.get(), new_batch.get());
-    } else {
-      // TODO(varunsh): support this use case
-      throw invalid_argument(
-        "Cannot run a model with an unknown tensor shapes");
+      new_batch->addRequest(new_request);
     }
+    new_batch->setBuffers(std::move(input_buffers), {});
+
+    auto* run_ptr = getFunction(handle_, "run");
+    auto* runModel =
+      reinterpret_cast<void (*)(amdinfer::Batch*, amdinfer::Batch*)>(run_ptr);
 
-    respond(new_batch.get());
+    runModel(batch, new_batch.get());
+  } else {
+    auto* run_ptr = getFunction(handle_, "run");
+    auto* runModel =
+      reinterpret_cast<amdinfer::BatchPtr (*)(amdinfer::Batch*)>(run_ptr);
 
-    this->returnInputBuffers(std::move(batch));
-    this->returnInputBuffers(std::move(new_batch));
+    new_batch = runModel(batch);
   }
-  AMDINFER_LOG_INFO(logger, "CPlusPlus ending");
+
+  return new_batch;
 }
 
 void CPlusPlus::doRelease() {}
@@ -303,6 +222,6 @@ extern "C" {
 // using smart pointer here may cause problems inside shared object so managing
 // manually
 amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::CPlusPlus("cPlusPlus", "cpu");
+  return new amdinfer::workers::CPlusPlus("CPlusPlus", "CPU", true);
 }
 }  // extern C
diff --git a/src/amdinfer/workers/echo.cpp b/src/amdinfer/workers/echo.cpp
deleted file mode 100644
index ffda447b0..000000000
--- a/src/amdinfer/workers/echo.cpp
+++ /dev/null
@@ -1,203 +0,0 @@
-// Copyright 2021 Xilinx, Inc.
-// Copyright 2022 Advanced Micro Devices, Inc.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-/**
- * @file
- * @brief Implements the Echo worker
- */
-
-#include <cstddef>    // for size_t, byte
-#include <cstdint>    // for uint32_t, int32_t
-#include <cstring>    // for memcpy
-#include <exception>  // for exception
-#include <memory>     // for unique_ptr, allocator
-#include <ratio>      // for micro
-#include <string>     // for string
-#include <thread>     // for thread
-#include <utility>    // for move
-#include <vector>     // for vector
-
-#include "amdinfer/batching/hard.hpp"    // for HardBatcher
-#include "amdinfer/build_options.hpp"    // for AMDINFER_ENABLE_TRACING
-#include "amdinfer/core/data_types.hpp"  // for DataType, DataType::Uint32
-#include "amdinfer/core/inference_request.hpp"   // for InferenceRequest
-#include "amdinfer/core/inference_response.hpp"  // for InferenceResponse
-#include "amdinfer/core/parameters.hpp"          // for ParameterMap
-#include "amdinfer/declarations.hpp"         // for BufferPtr, InferenceRes...
-#include "amdinfer/observation/logging.hpp"  // for Logger
-#include "amdinfer/observation/metrics.hpp"  // for Metrics
-#include "amdinfer/observation/tracing.hpp"  // for startFollowSpan, SpanPtr
-#include "amdinfer/util/thread.hpp"          // for setThreadName
-#include "amdinfer/util/timer.hpp"           // for Timer
-#include "amdinfer/workers/worker.hpp"       // for Worker
-
-namespace amdinfer::workers {
-
-/**
- * @brief The Echo worker is a simple worker that accepts a single uint32_t
- * argument and adds 1 to it and returns. It accepts multiple input tensors and
- * returns the corresponding number of output tensors.
- *
- */
-class Echo : public Worker {
- public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
-  [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
-
- private:
-  void doInit(ParameterMap* parameters) override;
-  void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
-  void doRelease() override;
-  void doDestroy() override;
-
-  // workers define what batcher implementation should be used for them.
-  // if not explicitly defined here, a default value is used from worker.hpp.
-  using Worker::makeBatcher;
-  std::vector<std::unique_ptr<Batcher>> makeBatcher(int num,
-                                                    ParameterMap* parameters,
-                                                    MemoryPool* pool) override {
-    return this->makeBatcher<HardBatcher>(num, parameters, pool);
-  };
-};
-
-std::thread Echo::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&Echo::run, this, input_queue);
-}
-
-std::vector<MemoryAllocators> Echo::getAllocators() const {
-  return {MemoryAllocators::Cpu};
-}
-
-void Echo::doInit(ParameterMap* parameters) {
-  constexpr auto kBatchSize = 1;
-
-  auto batch_size = kBatchSize;
-  if (parameters->has("batch_size")) {
-    batch_size = parameters->get<int32_t>("batch_size");
-  }
-  this->batch_size_ = batch_size;
-}
-
-void Echo::doAcquire(ParameterMap* parameters) {
-  (void)parameters;  // suppress unused variable warning
-
-  this->metadata_.addInputTensor("input", {1}, DataType::Uint32);
-  this->metadata_.addOutputTensor("output", {1}, DataType::Uint32);
-}
-
-void Echo::doRun(BatchPtrQueue* input_queue) {
-  util::setThreadName("Echo");
-#ifdef AMDINFER_ENABLE_LOGGING
-  const auto& logger = this->getLogger();
-#endif
-
-  while (true) {
-    BatchPtr batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
-    AMDINFER_LOG_INFO(logger, "Got request in echo");
-#ifdef AMDINFER_ENABLE_METRICS
-    Metrics::getInstance().incrementCounter(
-      MetricCounterIDs::PipelineIngressWorker);
-#endif
-    for (unsigned int j = 0; j < batch->size(); j++) {
-      const auto& req = batch->getRequest(j);
-#ifdef AMDINFER_ENABLE_TRACING
-      const auto& trace = batch->getTrace(j);
-      trace->startSpan("echo");
-#endif
-      InferenceResponse resp;
-      resp.setID(req->getID());
-      resp.setModel("echo");
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-      for (unsigned int i = 0; i < inputs.size(); i++) {
-        auto* input_buffer = inputs[i].getData();
-        // std::byte* output_buffer = outputs[i].getData();
-        // auto* input_buffer = dynamic_cast<VectorBuffer*>(input_ptr);
-        // auto* output_buffer = dynamic_cast<VectorBuffer*>(output_ptr);
-
-        uint32_t value = *static_cast<uint32_t*>(input_buffer);
-
-        // this is my operation: add one to the read argument. While this can't
-        // raise an exception, if exceptions can happen, they should be handled
-        try {
-          value++;
-        } catch (const std::exception& e) {
-          AMDINFER_LOG_ERROR(logger, e.what());
-          req->runCallbackError("Something went wrong");
-          continue;
-        }
-
-        // output_buffer->write(value);
-
-        InferenceResponseOutput output;
-        output.setDatatype(DataType::Uint32);
-        std::string output_name;
-        if (i < outputs.size()) {
-          output_name = outputs[i].getName();
-        }
-
-        if (output_name.empty()) {
-          output.setName(inputs[0].getName());
-        } else {
-          output.setName(output_name);
-        }
-        output.setShape({1});
-        std::vector<std::byte> buffer;
-        buffer.resize(sizeof(uint32_t));
-        memcpy(buffer.data(), &value, sizeof(uint32_t));
-        output.setData(std::move(buffer));
-        resp.addOutput(output);
-      }
-
-#ifdef AMDINFER_ENABLE_TRACING
-      auto context = trace->propagate();
-      resp.setContext(std::move(context));
-#endif
-
-      // respond back to the client
-      req->runCallbackOnce(resp);
-#ifdef AMDINFER_ENABLE_METRICS
-      Metrics::getInstance().incrementCounter(
-        MetricCounterIDs::PipelineEgressWorker);
-      util::Timer timer{batch->getTime(j)};
-      timer.stop();
-      auto duration = timer.count<std::micro>();
-      Metrics::getInstance().observeSummary(MetricSummaryIDs::RequestLatency,
-                                            duration);
-#endif
-    }
-    this->returnInputBuffers(std::move(batch));
-  }
-  AMDINFER_LOG_INFO(logger, "Echo ending");
-}
-
-void Echo::doRelease() {}
-void Echo::doDestroy() {}
-
-}  // namespace amdinfer::workers
-
-extern "C" {
-// using smart pointer here may cause problems inside shared object so managing
-// manually
-amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::Echo("echo", "cpu");
-}
-}  // extern C
diff --git a/src/amdinfer/workers/echo_multi.cpp b/src/amdinfer/workers/echo_multi.cpp
deleted file mode 100644
index 11aac3c6e..000000000
--- a/src/amdinfer/workers/echo_multi.cpp
+++ /dev/null
@@ -1,224 +0,0 @@
-// Copyright 2022 Advanced Micro Devices, Inc.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-/**
- * @file
- * @brief Implements the EchoMulti worker
- */
-
-#include <array>    // for array
-#include <cassert>  // for assert
-#include <cstddef>  // for size_t, byte
-#include <cstdint>  // for uint32_t, int32_t
-#include <cstring>  // for memcpy
-#include <memory>   // for unique_ptr, allocator
-#include <ratio>    // for micro
-#include <string>   // for string
-#include <thread>   // for thread
-#include <utility>  // for move
-#include <vector>   // for vector
-
-#include "amdinfer/batching/hard.hpp"    // for HardBatcher
-#include "amdinfer/build_options.hpp"    // for AMDINFER_ENABLE_TRACING
-#include "amdinfer/core/data_types.hpp"  // for DataType, DataType::Uint32
-#include "amdinfer/core/inference_request.hpp"   // for InferenceRequest
-#include "amdinfer/core/inference_response.hpp"  // for InferenceResponse
-#include "amdinfer/core/parameters.hpp"          // for ParameterMap
-#include "amdinfer/declarations.hpp"         // for BufferPtr, InferenceRes...
-#include "amdinfer/observation/logging.hpp"  // for Logger
-#include "amdinfer/observation/metrics.hpp"  // for Metrics
-#include "amdinfer/observation/tracing.hpp"  // for startFollowSpan, SpanPtr
-#include "amdinfer/util/containers.hpp"      // for containerSum
-#include "amdinfer/util/queue.hpp"           // for BufferPtrsQueue
-#include "amdinfer/util/thread.hpp"          // for setThreadName
-#include "amdinfer/util/timer.hpp"           // for Timer
-#include "amdinfer/workers/worker.hpp"       // for Worker
-
-namespace amdinfer {
-
-const int kInputTensors = 2;
-const std::array<int, kInputTensors> kInputLengths = {1, 2};
-const int kOutputTensors = 3;
-const std::array<int, kOutputTensors> kOutputLengths = {1, 4, 3};
-
-namespace workers {
-
-/**
- * @brief The EchoMulti worker is a simple worker that accepts two input tensors
- * and produces 3 output tensors as a test case for multi-input/output models.
- *
- */
-class EchoMulti : public Worker {
- public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
-  [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
-
- private:
-  void doInit(ParameterMap* parameters) override;
-  void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
-  void doRelease() override;
-  void doDestroy() override;
-
-  // workers define what batcher implementation should be used for them.
-  // if not explicitly defined here, a default value is used from worker.hpp.
-  using Worker::makeBatcher;
-  std::vector<std::unique_ptr<Batcher>> makeBatcher(int num,
-                                                    ParameterMap* parameters,
-                                                    MemoryPool* pool) override {
-    return this->makeBatcher<HardBatcher>(num, parameters, pool);
-  };
-};
-
-std::thread EchoMulti::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&EchoMulti::run, this, input_queue);
-}
-
-std::vector<MemoryAllocators> EchoMulti::getAllocators() const {
-  return {MemoryAllocators::Cpu};
-}
-
-void EchoMulti::doInit(ParameterMap* parameters) {
-  constexpr auto kBatchSize = 1;
-
-  auto batch_size = kBatchSize;
-  if (parameters->has("batch_size")) {
-    batch_size = parameters->get<int32_t>("batch_size");
-  }
-  this->batch_size_ = batch_size;
-}
-
-void EchoMulti::doAcquire([[maybe_unused]] ParameterMap* parameters) {
-  for (auto i = 0; i < kInputTensors; ++i) {
-    this->metadata_.addInputTensor("input" + std::to_string(i),
-                                   {static_cast<uint64_t>(kInputLengths.at(i))},
-                                   DataType::Uint32);
-  }
-  for (auto i = 0; i < kOutputTensors; ++i) {
-    this->metadata_.addOutputTensor(
-      "input" + std::to_string(i),
-      {static_cast<uint64_t>(kOutputLengths.at(i))}, DataType::Uint32);
-  }
-}
-
-void EchoMulti::doRun(BatchPtrQueue* input_queue) {
-  util::setThreadName("EchoMulti");
-#ifdef AMDINFER_ENABLE_LOGGING
-  const auto& logger = this->getLogger();
-#endif
-
-  while (true) {
-    BatchPtr batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
-    AMDINFER_LOG_INFO(logger, "Got request in echoMulti");
-#ifdef AMDINFER_ENABLE_METRICS
-    Metrics::getInstance().incrementCounter(
-      MetricCounterIDs::PipelineIngressWorker);
-#endif
-    const auto batch_size = batch->size();
-    for (unsigned int j = 0; j < batch_size; j++) {
-      const auto& req = batch->getRequest(static_cast<int>(j));
-#ifdef AMDINFER_ENABLE_TRACING
-      const auto& trace = batch->getTrace(static_cast<int>(j));
-      trace->startSpan("echoMulti");
-#endif
-      InferenceResponse resp;
-      resp.setID(req->getID());
-      resp.setModel("echoMulti");
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-
-      std::vector<int> args;
-      const auto input_num = util::containerSum(kInputLengths);
-      assert(input_num != 0);
-      args.reserve(input_num);
-      for (auto i = 0; i < kInputTensors; ++i) {
-        const auto* input_buffer = static_cast<uint32_t*>(inputs[i].getData());
-        for (auto k = 0; k < kInputLengths.at(i); ++k) {
-          args.push_back(input_buffer[k]);
-        }
-      }
-
-      std::vector<int> output_args;
-      auto input_index = 0;
-      auto offset = 0;
-      output_args.reserve(util::containerSum(kOutputLengths));
-      for (auto i = 0; i < kOutputTensors; ++i) {
-        for (auto k = 0; k < kOutputLengths.at(i); ++k) {
-          output_args.push_back(args[input_index]);
-          input_index = (input_index + 1) % input_num;
-        }
-
-        InferenceResponseOutput output;
-        output.setDatatype(DataType::Uint32);
-        std::string output_name;
-        if (static_cast<size_t>(i) < outputs.size()) {
-          output_name = outputs[i].getName();
-        }
-
-        if (output_name.empty()) {
-          output.setName(inputs[0].getName());
-        } else {
-          output.setName(output_name);
-        }
-        output.setShape({static_cast<uint64_t>(kOutputLengths.at(i))});
-        std::vector<std::byte> buffer;
-        buffer.resize(kOutputLengths.at(i) * sizeof(uint32_t));
-        memcpy(buffer.data(), &(output_args[offset]),
-               kOutputLengths.at(i) * sizeof(uint32_t));
-        output.setData(std::move(buffer));
-        resp.addOutput(output);
-        offset += kOutputLengths.at(i);
-      }
-
-#ifdef AMDINFER_ENABLE_TRACING
-      auto context = trace->propagate();
-      resp.setContext(std::move(context));
-#endif
-
-      // respond back to the client
-      req->runCallbackOnce(resp);
-#ifdef AMDINFER_ENABLE_METRICS
-      Metrics::getInstance().incrementCounter(
-        MetricCounterIDs::PipelineEgressWorker);
-      util::Timer timer{batch->getTime(j)};
-      timer.stop();
-      auto duration = timer.count<std::micro>();
-      Metrics::getInstance().observeSummary(MetricSummaryIDs::RequestLatency,
-                                            duration);
-#endif
-    }
-    this->returnInputBuffers(std::move(batch));
-  }
-  AMDINFER_LOG_INFO(logger, "EchoMulti ending");
-}
-
-void EchoMulti::doRelease() {}
-void EchoMulti::doDestroy() {}
-
-}  // namespace workers
-
-}  // namespace amdinfer
-
-extern "C" {
-// using smart pointer here may cause problems inside shared object so managing
-// manually
-amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::EchoMulti("echoMulti", "cpu");
-}
-}  // extern C
diff --git a/src/amdinfer/workers/invert_image.cpp b/src/amdinfer/workers/invert_image.cpp
deleted file mode 100644
index 883498cec..000000000
--- a/src/amdinfer/workers/invert_image.cpp
+++ /dev/null
@@ -1,266 +0,0 @@
-// Copyright 2021 Xilinx, Inc.
-// Copyright 2022 Advanced Micro Devices, Inc.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-/**
- * @file
- * @brief Implements the InvertImage worker
- */
-
-#include <algorithm>              // for max
-#include <climits>                // for CHAR_BIT
-#include <cstddef>                // for size_t, byte
-#include <cstdint>                // for uint8_t, uint64_t, int32_t
-#include <cstring>                // for memcpy
-#include <memory>                 // for unique_ptr, allocator
-#include <opencv2/core.hpp>       // for bitwise_not, Mat
-#include <opencv2/imgcodecs.hpp>  // for imdecode, imencode, IMRE...
-#include <ratio>                  // for micro
-#include <string>                 // for string, basic_string
-#include <thread>                 // for thread
-#include <utility>                // for move
-#include <vector>                 // for vector
-
-#include "amdinfer/batching/batcher.hpp"        // for Batch, BatchPtrQueue
-#include "amdinfer/build_options.hpp"           // for AMDINFER_ENABLE_TRACING
-#include "amdinfer/core/data_types.hpp"         // for DataType, DataType::Uint8
-#include "amdinfer/core/inference_request.hpp"  // for InferenceRequest
-#include "amdinfer/core/inference_response.hpp"  // for InferenceResponse
-#include "amdinfer/core/parameters.hpp"          // for ParameterMap
-#include "amdinfer/declarations.hpp"         // for BufferPtr, InferenceRes...
-#include "amdinfer/observation/logging.hpp"  // for Logger
-#include "amdinfer/observation/metrics.hpp"  // for Metrics
-#include "amdinfer/observation/tracing.hpp"  // for startFollowSpan, SpanPtr
-#include "amdinfer/util/base64.hpp"          // for base64_decode, base64_e...
-#include "amdinfer/util/containers.hpp"      // for containerProduct
-#include "amdinfer/util/thread.hpp"          // for setThreadName
-#include "amdinfer/util/timer.hpp"           // for Timer
-#include "amdinfer/workers/worker.hpp"       // for Worker
-
-namespace {
-
-/// Invert the pixel color. Assumes RGB in some order with optional alpha
-template <typename T, bool kAlphaPresent>
-void invert(void* ibuf, void* obuf, uint64_t size) {
-  static_assert(std::is_pointer_v<T>, "T must be a pointer type");
-  auto* idata = static_cast<T>(ibuf);
-  auto* odata = static_cast<T>(obuf);
-  static_assert(sizeof(idata[0]) < sizeof(uint64_t), "T must be <8 bytes");
-
-  // mask to get the largest value. E.g. for uint8_t, mask will be 255.
-  const auto mask = (1ULL << (sizeof(idata[0]) * CHAR_BIT)) - 1;
-  const uint64_t incr = kAlphaPresent ? 4 : 3;
-  for (uint64_t i = 0; i < size; i += incr) {
-    odata[i] = mask - idata[i];
-    odata[i + 1] = mask - idata[i + 1];
-    odata[i + 2] = mask - idata[i + 2];
-    if constexpr (kAlphaPresent) {
-      odata[i + 3] = idata[i + 3];
-    }
-  }
-}
-
-}  // namespace
-
-namespace amdinfer::workers {
-
-/**
- * @brief The InvertImage worker is a simple worker that accepts an array
- * representing an image and adds one to each pixel. It accepts multiple input
- * tensors and returns the corresponding number of output tensors.
- *
- */
-class InvertImage : public Worker {
- public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
-  [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
-
- private:
-  void doInit(ParameterMap* parameters) override;
-  void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
-  void doRelease() override;
-  void doDestroy() override;
-};
-
-std::thread InvertImage::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&InvertImage::run, this, input_queue);
-}
-
-std::vector<MemoryAllocators> InvertImage::getAllocators() const {
-  return {MemoryAllocators::Cpu};
-}
-
-void InvertImage::doInit(ParameterMap* parameters) {
-  constexpr auto kBatchSize = 1;
-
-  auto batch_size = kBatchSize;
-  if (parameters->has("batch_size")) {
-    batch_size = parameters->get<int32_t>("batch_size");
-  }
-  this->batch_size_ = batch_size;
-}
-
-// Support up to Full HD
-const auto kMaxImageHeight = 1080;
-const auto kMaxImageWidth = 1920;
-const auto kMaxImageChannels = 3;
-
-void InvertImage::doAcquire(ParameterMap* parameters) {
-  (void)parameters;  // suppress unused variable warning
-
-  this->metadata_.addInputTensor(
-    "input",
-    {this->batch_size_, kMaxImageHeight, kMaxImageWidth, kMaxImageChannels},
-    DataType::Uint8);
-  this->metadata_.addOutputTensor(
-    "output",
-    {this->batch_size_, kMaxImageHeight, kMaxImageWidth, kMaxImageChannels},
-    DataType::Uint32);
-}
-
-void InvertImage::doRun(BatchPtrQueue* input_queue) {
-  util::setThreadName("InvertImage");
-#ifdef AMDINFER_ENABLE_LOGGING
-  const auto& logger = this->getLogger();
-#endif
-
-  std::vector<std::byte> output;
-  output.resize(kMaxImageHeight * kMaxImageWidth * kMaxImageChannels);
-
-  while (true) {
-    BatchPtr batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
-
-    AMDINFER_LOG_INFO(logger, "Got request in InvertImage");
-    for (unsigned int j = 0; j < batch->size(); j++) {
-      const auto& req = batch->getRequest(j);
-#ifdef AMDINFER_ENABLE_TRACING
-      const auto& trace = batch->getTrace(j);
-      trace->startSpan("InvertImage");
-#endif
-      InferenceResponse resp;
-      resp.setID(req->getID());
-      resp.setModel("invert_image");
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-      for (unsigned int i = 0; i < inputs.size(); i++) {
-        auto* input_buffer = inputs[i].getData();
-        auto* output_buffer = output.data();
-
-        auto input_shape = inputs[i].getShape();
-
-        auto input_size = util::containerProduct(input_shape);
-        auto input_dtype = inputs[i].getDatatype();
-
-        // invert image, store in output
-        InferenceResponseOutput output;
-        if (input_dtype == DataType::Uint8) {
-          // Output will have the same shape as input
-          output.setShape(input_shape);
-
-          if (input_shape.size() == 3 && input_shape[2] == 3) {
-            invert<uint8_t*, false>(input_buffer, output_buffer, input_size);
-          } else {
-            invert<uint8_t*, true>(input_buffer, output_buffer, input_size);
-          }
-
-          auto* output_data = reinterpret_cast<uint8_t*>(output_buffer);
-
-          std::vector<std::byte> buffer;
-          buffer.resize(input_size);
-          memcpy(buffer.data(), output_data, input_size);
-          output.setData(std::move(buffer));
-          output.setDatatype(DataType::Uint8);
-        } else if (input_dtype == DataType::String) {
-          auto* idata = static_cast<char*>(input_buffer);
-          auto decoded_str = util::base64Decode(idata, input_size);
-          std::vector<char> data(decoded_str.begin(), decoded_str.end());
-          cv::Mat img;
-          try {
-            img = cv::imdecode(data, cv::IMREAD_UNCHANGED);
-          } catch (const cv::Exception& e) {
-            AMDINFER_LOG_ERROR(logger, e.what());
-            req->runCallbackError("Failed to decode base64 image data");
-            continue;
-          }
-
-          if (img.empty()) {
-            const char* error = "Decoded image is empty";
-            AMDINFER_LOG_ERROR(logger, error);
-            req->runCallbackError(error);
-            continue;
-          }
-          cv::bitwise_not(img, img);
-          std::vector<unsigned char> buf;
-          cv::imencode(".jpg", img, buf);
-          const auto* enc_msg = reinterpret_cast<const char*>(buf.data());
-          auto encoded = util::base64Encode(enc_msg, buf.size());
-          std::vector<std::byte> buffer;
-          buffer.resize(encoded.size());
-          memcpy(buffer.data(), encoded.data(), encoded.length());
-          output.setData(std::move(buffer));
-          output.setDatatype(DataType::String);
-          output.setShape({encoded.size()});
-        }
-
-        // if our output is explicitly named, use that name in response, or use
-        // the input tensor's name if it's not defined.
-        std::string output_name;
-        if (i < outputs.size()) {
-          output_name = outputs[i].getName();
-        }
-
-        if (output_name.empty()) {
-          output.setName(inputs[0].getName());
-        } else {
-          output.setName(output_name);
-        }
-
-        resp.addOutput(output);
-      }
-#ifdef AMDINFER_ENABLE_METRICS
-      util::Timer timer{batch->getTime(j)};
-      timer.stop();
-      auto duration = timer.count<std::micro>();
-      Metrics::getInstance().observeSummary(MetricSummaryIDs::RequestLatency,
-                                            duration);
-#endif
-#ifdef AMDINFER_ENABLE_TRACING
-      auto context = trace->propagate();
-      resp.setContext(std::move(context));
-#endif
-      req->runCallbackOnce(resp);
-    }
-    this->returnInputBuffers(std::move(batch));
-  }
-  AMDINFER_LOG_INFO(logger, "InvertImage ending");
-}
-
-void InvertImage::doRelease() {}
-void InvertImage::doDestroy() {}
-
-}  // namespace amdinfer::workers
-
-extern "C" {
-// using smart pointer here may cause problems inside shared object so managing
-// manually
-amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::InvertImage("InvertImage", "CPU");
-}
-}  // extern C
diff --git a/src/amdinfer/workers/invert_video.cpp b/src/amdinfer/workers/invert_video.cpp
index 966a01f07..178b29f8d 100644
--- a/src/amdinfer/workers/invert_video.cpp
+++ b/src/amdinfer/workers/invert_video.cpp
@@ -56,24 +56,19 @@ namespace workers {
  * video and sends the inverted frames back to the client over a websocket.
  *
  */
-class InvertVideo : public Worker {
+class InvertVideo : public SingleThreadedWorker {
  public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
+  using SingleThreadedWorker::SingleThreadedWorker;
   [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
 
  private:
   void doInit(ParameterMap* parameters) override;
   void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
+  BatchPtr doRun(Batch* batch, const MemoryPool* pool) override;
   void doRelease() override;
   void doDestroy() override;
 };
 
-std::thread InvertVideo::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&InvertVideo::run, this, input_queue);
-}
-
 std::vector<MemoryAllocators> InvertVideo::getAllocators() const {
   return {MemoryAllocators::Cpu};
 }
@@ -102,98 +97,89 @@ void InvertVideo::doAcquire(ParameterMap* parameters) {
     DataType::Int8);
 }
 
-void InvertVideo::doRun(BatchPtrQueue* input_queue) {
-  util::setThreadName("InvertVideo");
+BatchPtr InvertVideo::doRun(Batch* batch,
+                            [[maybe_unused]] const MemoryPool* pool) {
 #ifdef AMDINFER_ENABLE_LOGGING
   const auto& logger = this->getLogger();
 #endif
 
-  while (true) {
-    BatchPtr batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
+  for (unsigned int j = 0; j < batch->size(); j++) {
+    const auto& req = batch->getRequest(j);
+    auto inputs = req->getInputs();
+    auto outputs = req->getOutputs();
+    auto key = req->getParameters().get<std::string>("key");
+    for (auto& input : inputs) {
+      auto* input_buffer = input.getData();
+
+      // TODO(varunsh): should strings have null terminators embedded?
+      const auto* idata = static_cast<char*>(input_buffer);
+      std::string data{idata, input.getSize()};
+
+      cv::VideoCapture cap(data);  // open the video file
+      if (!cap.isOpened()) {       // check if we succeeded
+        const char* error = "Cannot open video file";
+        AMDINFER_LOG_ERROR(logger, error);
+        req->runCallbackError(error);
+        continue;
+      }
 
-    AMDINFER_LOG_INFO(logger, "Got request in InvertVideo");
-    for (unsigned int j = 0; j < batch->size(); j++) {
-      const auto& req = batch->getRequest(j);
-#ifdef AMDINFER_ENABLE_TRACING
-      const auto& trace = batch->getTrace(j);
-      trace->startSpan("InvertVideo");
-#endif
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-      auto key = req->getParameters().get<std::string>("key");
-      for (auto& input : inputs) {
-        auto* input_buffer = input.getData();
-
-        auto* idata = static_cast<char*>(input_buffer);
-
-        cv::VideoCapture cap(idata);  // open the video file
-        if (!cap.isOpened()) {        // check if we succeeded
-          const char* error = "Cannot open video file";
-          AMDINFER_LOG_ERROR(logger, error);
-          req->runCallbackError(error);
+      InferenceResponse resp;
+      resp.setID(req->getID());
+      resp.setModel("invert_video");
+
+      // contains the number of frames in the video;
+      auto count = static_cast<int32_t>(
+        cap.get(cv::VideoCaptureProperties::CAP_PROP_FRAME_COUNT));
+      if (input.getParameters().has("count")) {
+        count = input.getParameters().get<int32_t>("count");
+      }
+      double fps = cap.get(cv::VideoCaptureProperties::CAP_PROP_FPS);
+
+      InferenceResponseOutput output;
+      output.setName("key");
+      output.setDatatype(DataType::String);
+      auto message = constructMessage(key, std::to_string(fps));
+      std::vector<std::byte> buffer;
+      buffer.resize(message.size());
+      memcpy(buffer.data(), message.data(), message.size());
+      output.setData(std::move(buffer));
+      output.setShape({message.size()});
+      resp.addOutput(output);
+      req->runCallback(resp);
+      for (int num_frames = 0; num_frames < count; num_frames++) {
+        cv::Mat frame;
+        cap >> frame;  // get the next frame from video
+        if (frame.empty()) {
+          num_frames--;
           continue;
         }
+        cv::bitwise_not(frame, frame);
+        std::vector<unsigned char> buf;
+        cv::imencode(".jpg", frame, buf);
+        const auto* enc_msg = reinterpret_cast<const char*>(buf.data());
+        std::string encoded =
+          "data:image/jpg;base64," + util::base64Encode(enc_msg, buf.size());
 
         InferenceResponse resp;
         resp.setID(req->getID());
         resp.setModel("invert_video");
 
-        // contains the number of frames in the video;
-        auto count = static_cast<int32_t>(
-          cap.get(cv::VideoCaptureProperties::CAP_PROP_FRAME_COUNT));
-        if (input.getParameters().has("count")) {
-          count = input.getParameters().get<int32_t>("count");
-        }
-        double fps = cap.get(cv::VideoCaptureProperties::CAP_PROP_FPS);
-
         InferenceResponseOutput output;
-        output.setName("key");
+        output.setName("image");
         output.setDatatype(DataType::String);
-        auto message = constructMessage(key, std::to_string(fps));
-        std::vector<std::byte> buffer;
+        message = constructMessage(key, encoded);
         buffer.resize(message.size());
         memcpy(buffer.data(), message.data(), message.size());
         output.setData(std::move(buffer));
         output.setShape({message.size()});
         resp.addOutput(output);
         req->runCallback(resp);
-        for (int num_frames = 0; num_frames < count; num_frames++) {
-          cv::Mat frame;
-          cap >> frame;  // get the next frame from video
-          if (frame.empty()) {
-            num_frames--;
-            continue;
-          }
-          cv::bitwise_not(frame, frame);
-          std::vector<unsigned char> buf;
-          cv::imencode(".jpg", frame, buf);
-          const auto* enc_msg = reinterpret_cast<const char*>(buf.data());
-          std::string encoded =
-            "data:image/jpg;base64," + util::base64Encode(enc_msg, buf.size());
-
-          InferenceResponse resp;
-          resp.setID(req->getID());
-          resp.setModel("invert_video");
-
-          InferenceResponseOutput output;
-          output.setName("image");
-          output.setDatatype(DataType::String);
-          message = constructMessage(key, encoded);
-          buffer.resize(message.size());
-          memcpy(buffer.data(), message.data(), message.size());
-          output.setData(std::move(buffer));
-          output.setShape({message.size()});
-          resp.addOutput(output);
-          req->runCallback(resp);
-        }
       }
     }
   }
-  AMDINFER_LOG_INFO(logger, "InvertVideo ending");
+
+  // okay because ensembles disabled for this worker
+  return nullptr;
 }
 
 void InvertVideo::doRelease() {}
@@ -207,6 +193,6 @@ extern "C" {
 // using smart pointer here may cause problems inside shared object so managing
 // manually
 amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::InvertVideo("InvertVideo", "CPU");
+  return new amdinfer::workers::InvertVideo("InvertVideo", "CPU", false);
 }
 }  // extern C
diff --git a/src/amdinfer/workers/migraphx.cpp b/src/amdinfer/workers/migraphx.cpp
index ea5c69db4..a80f0941c 100644
--- a/src/amdinfer/workers/migraphx.cpp
+++ b/src/amdinfer/workers/migraphx.cpp
@@ -46,7 +46,9 @@
 #include "amdinfer/observation/logging.hpp"  // for AMDINFER_LOG_INFO, AMD...
 #include "amdinfer/observation/metrics.hpp"  // for Metrics, MetricCounterIDs
 #include "amdinfer/util/containers.hpp"      // for containerProduct
+#include "amdinfer/util/memory.hpp"          // for copy
 #include "amdinfer/util/queue.hpp"           // for BufferPtrsQueue
+#include "amdinfer/util/string.hpp"          // for contains
 #include "amdinfer/util/thread.hpp"          // for setThreadName
 #include "amdinfer/util/timer.hpp"           // for Timer
 #include "amdinfer/workers/worker.hpp"       // for Worker, kNumBufferAuto
@@ -58,19 +60,20 @@ namespace amdinfer::workers {
  * argument and compiles and evaluates it.
  *
  */
-class MIGraphXWorker : public Worker {
+class MIGraphXWorker : public SingleThreadedWorker {
  public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
+  using SingleThreadedWorker::SingleThreadedWorker;
   [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
 
  private:
   void doInit(ParameterMap* parameters) override;
   void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
+  BatchPtr doRun(Batch* batch, const MemoryPool* pool) override;
   void doRelease() override;
   void doDestroy() override;
 
+  void compile(const std::string& onnx_path, const std::string& compiled_path);
+
   // the model file to be loaded.  Supported types are *.onnx and *.mxr
   std::filesystem::path input_file_;
   // The prog_ is populated by reading the model file and contains most of
@@ -79,25 +82,25 @@ class MIGraphXWorker : public Worker {
   migraphx::program prog_;
 
   // flag to pad out a batch with dummy data.  Sending a batch of requests
-  // with uninitialized data may crash migraphx, for certain models.
+  // with uninitialized data may crash MIGraphX, for certain models.
   // If pad_batch_ is true, this worker will pad any unused request slots
   // in a batch with dummy copies of the first request.
   bool pad_batch_ = true;
   // Calculated sizes in bytes for each input tensor, by input name
-  std::map<std::string, size_t> input_sizes_;
+  std::map<std::string, size_t, std::less<>> input_sizes_;
 };
 
-std::thread MIGraphXWorker::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&MIGraphXWorker::run, this, input_queue);
-}
-
 std::vector<MemoryAllocators> MIGraphXWorker::getAllocators() const {
   return {MemoryAllocators::Cpu};
 }
 
-// Enum-to-enum conversion to let us read data type from migraphx model.
-// The definitions are taken from the MIGraphX macro
-// MIGRAPHX_SHAPE_VISIT_TYPES
+/**
+ * @brief Enum-to-enum conversion to let us read data type from MIGraphX model.
+ * The definitions are taken from the MIGraphX macro MIGRAPHX_SHAPE_VISIT_TYPES
+ *
+ * @param in MIGraphX datatype
+ * @return DataType
+ */
 DataType toDataType(migraphx_shape_datatype_t in) {
   switch (in) {
     // case 0 is tuple_type which we don't support here
@@ -130,6 +133,75 @@ DataType toDataType(migraphx_shape_datatype_t in) {
   }
 }
 
+void MIGraphXWorker::compile(const std::string& onnx_path,
+                             const std::string& compiled_path) {
+#ifdef AMDINFER_ENABLE_LOGGING
+  const auto& logger = this->getLogger();
+#endif
+
+  // Look for onnx file.  ifstream tests that the file can be opened
+  auto f = std::ifstream(onnx_path.c_str());
+  if (!f.good()) {
+    // Not finding the model file makes it impossible to finish initializing
+    // this worker
+    AMDINFER_LOG_ERROR(
+      logger, std::string("migraphx worker cannot open the model file ") +
+                onnx_path + ".  Does this path exist?");
+    throw invalid_argument(std::string("model file ") + onnx_path +
+                           " not found or can't be opened");
+  }
+  // Load the onnx file
+  // Using parse_onnx() instead of load() because there's a bug at the
+  // time of writing
+  AMDINFER_LOG_INFO(logger,
+                    std::string("migraphx worker loading ONNX model file ") +
+                      onnx_path.c_str());
+
+  migraphx::onnx_options onnx_opts;
+  onnx_opts.set_default_dim_value(static_cast<unsigned int>(batch_size_));
+  this->prog_ = migraphx::parse_onnx(onnx_path.c_str(), onnx_opts);
+
+  auto param_shapes =
+    prog_.get_parameter_shapes();  // program_parameter_shapes struct
+
+  AMDINFER_LOG_INFO(
+    logger,
+    std::string("migraphx worker loaded ONNX model file ") + onnx_path.c_str());
+
+  // Compile the model.  Hard-coded choices of offload_copy and gpu target
+  migraphx::compile_options comp_opts;
+  comp_opts.set_offload_copy();
+
+  // migraphx can support a reference (cpu) target as a fallback if GPU is
+  // not found; not implemented here
+  const bool use_gpu = true;
+  std::string target_str = use_gpu ? "gpu" : "ref";
+  migraphx::target targ{target_str.c_str()};
+  // The hip library will throw a cryptic error if unable to connect with
+  // a GPU at this point.
+  try {
+    prog_.compile(migraphx::target("gpu"), comp_opts);
+  } catch (const std::exception& e) {
+    std::string error = e.what();
+    if (util::contains(error, "Failed to call function")) {
+      error = error + ".  Server could not connect to a GPU.";
+    }
+    AMDINFER_LOG_ERROR(logger, error);
+    throw external_error(error);
+  }
+
+  // Save the compiled program as a MessagePack (*.mxr) file
+  f = std::ifstream(compiled_path.c_str());
+  if (!f.good()) {
+    migraphx::file_options options;
+    options.set_file_format("msgpack");
+
+    migraphx::save(this->prog_, compiled_path.c_str(), options);
+    AMDINFER_LOG_INFO(logger, std::string(" Saved compiled model file ") +
+                                compiled_path.c_str());
+  }
+}
+
 void MIGraphXWorker::doInit(ParameterMap* parameters) {
   // default batch size; client may request a change. Arbitrarily set to 64
   const int default_batch_size = 64;
@@ -208,72 +280,7 @@ void MIGraphXWorker::doInit(ParameterMap* parameters) {
       // prog_ does not need to be compiled.
     }
   } else {
-    // Look for onnx file.  ifstream tests that the file can be opened
-    f = std::ifstream(onnx_path.c_str());
-    if (f.good()) {
-      // Load the onnx file
-      // Using parse_onnx() instead of load() because there's a bug at the
-      // time of writing
-      AMDINFER_LOG_INFO(
-        logger, std::string("migraphx worker loading ONNX model file ") +
-                  onnx_path.c_str());
-
-      migraphx::onnx_options onnx_opts;
-      onnx_opts.set_default_dim_value(batch_size_);
-      this->prog_ = migraphx::parse_onnx(onnx_path.c_str(), onnx_opts);
-
-      auto param_shapes =
-        prog_.get_parameter_shapes();  // program_parameter_shapes struct
-
-      AMDINFER_LOG_INFO(logger,
-                        std::string("migraphx worker loaded ONNX model file ") +
-                          onnx_path.c_str());
-
-      // Compile the model.  Hard-coded choices of offload_copy and gpu
-      // target.
-      migraphx::compile_options comp_opts;
-      comp_opts.set_offload_copy();
-
-      // migraphx can support a reference (cpu) target as a fallback if GPU is
-      // not found; not implemented here
-      const bool use_gpu = true;
-      std::string target_str = use_gpu ? "gpu" : "ref";
-      migraphx::target targ{target_str.c_str()};
-      // The hip library will throw a cryptic error if unable to connect with
-      // a GPU at this point.
-      try {
-        prog_.compile(migraphx::target("gpu"), comp_opts);
-      } catch (const std::exception& e) {
-        std::string emsg = e.what();
-        if (emsg.find("Failed to call function") != std::string::npos) {
-          emsg = emsg + ".  Server could not connect to a GPU.";
-        }
-        AMDINFER_LOG_ERROR(logger, emsg);
-        throw std::runtime_error(emsg);
-      }
-
-      // Save the compiled program as a MessagePack (*.mxr) file
-      f = std::ifstream(compiled_path.c_str());
-      if (!f.good()) {
-        migraphx::file_options options;
-        options.set_file_format("msgpack");
-
-        migraphx::save(this->prog_, compiled_path.c_str(), options);
-        AMDINFER_LOG_INFO(logger, std::string(" Saved compiled model file ") +
-                                    compiled_path.c_str());
-      }
-
-    } else {
-      // Not finding the model file makes it impossible to finish initializing
-      // this worker
-      AMDINFER_LOG_INFO(
-        logger, std::string("migraphx worker cannot open the model file ") +
-                  onnx_path.c_str() + " or " + compiled_path.c_str() +
-                  ".  Does this path exist?");
-      throw std::invalid_argument(std::string("model file ") +
-                                  onnx_path.c_str() +
-                                  " not found or can't be opened");
-    }
+    compile(onnx_path, compiled_path);
   }
   //
   // Fetch the expected dimensions of the input from the parsed model.
@@ -297,13 +304,10 @@ void MIGraphXWorker::doInit(ParameterMap* parameters) {
 
 void MIGraphXWorker::doAcquire(ParameterMap* parameters) { (void)parameters; }
 
-void MIGraphXWorker::doRun(BatchPtrQueue* input_queue) {
+BatchPtr MIGraphXWorker::doRun(Batch* batch, const MemoryPool* pool) {
 #ifdef AMDINFER_ENABLE_LOGGING
   const auto& logger = this->getLogger();
 #endif
-  AMDINFER_LOG_INFO(logger, "beginning of MIGraphXWorker::doRun");
-
-  util::setThreadName("Migraphx");
 
   // stringstream used for formatting logger messages
   std::string msg;
@@ -316,59 +320,50 @@ void MIGraphXWorker::doRun(BatchPtrQueue* input_queue) {
   // client and continue waiting for requests.
   //
 
-  while (true) {
-    BatchPtr batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
-    AMDINFER_LOG_INFO(logger, "New batch request in migraphx");
-    util::Timer timer;
-    timer.add("batch_start");
-#ifdef AMDINFER_ENABLE_METRICS
-    Metrics::getInstance().incrementCounter(
-      MetricCounterIDs::PipelineIngressWorker);
-#endif
+  util::Timer timer;
+  timer.add("batch_start");
 
-    // The MIGraphX operation: run the migraphx eval() method.
-    // If migraphx exceptions happen, they will be handled
+  // The MIGraphX operation: run the migraphx eval() method.
+  // If migraphx exceptions happen, they will be handled
 
-    // We only need to look at the 0'th request to set up evaluation, because
-    // its input pointers (one for each input) are the base addresses of the
-    // data for the entire batch. The different input tensors are not required
-    // to be contiguous with each other.
-    const auto& req0 = batch->getRequest(0);
-    auto inputs0 =
-      req0->getInputs();  // const std::vector<InferenceRequestInput>
+  // We only need to look at the 0'th request to set up evaluation, because
+  // its input pointers (one for each input) are the base addresses of the
+  // data for the entire batch. The different input tensors are not required
+  // to be contiguous with each other.
+  const auto& req0 = batch->getRequest(0);
+  auto inputs0 = req0->getInputs();  // const std::vector<InferenceRequestInput>
 
-    try {
-      migraphx::program_parameters params;
+  BatchPtr new_batch;
+  std::vector<amdinfer::BufferPtr> input_buffers;
 
-      // populate the migraphx parameters with shape read from the onnx
-      // model.
-      auto param_shapes = prog_.get_parameter_shapes();
+  try {
+    migraphx::program_parameters params;
 
-      for (const auto& aninput : inputs0) {  // InferenceRequestInput
-        auto aname = aninput.getName();
+    // populate the migraphx parameters with shape read from the onnx
+    // model.
+    auto param_shapes = prog_.get_parameter_shapes();
 
-        // Look up the shape by name in the model, but if there's only 1 input
-        // then the name in the request isn't required to match.
-        if (inputs0.size() == 1) {
-          aname = param_shapes.names().front();
-        }
-        migraphx::shape modelshape = param_shapes[aname.c_str()];
-
-        if (toDataType(modelshape.type()) != aninput.getDatatype()) {
-          smsg.str("");
-          smsg << "Migraph worker model and input data types don't match:   "
-               << toDataType(modelshape.type()) << " vs "
-               << aninput.getDatatype();
-          throw(invalid_argument(smsg.str()));
-        }
+    for (const auto& aninput : inputs0) {  // InferenceRequestInput
+      auto aname = aninput.getName();
 
-        // check that lengths() and type match
-        auto llen = modelshape.lengths();
-        // clang-format off
+      // Look up the shape by name in the model, but if there's only 1 input
+      // then the name in the request isn't required to match.
+      if (inputs0.size() == 1) {
+        aname = param_shapes.names().front();
+      }
+      migraphx::shape modelshape = param_shapes[aname.c_str()];
+
+      if (toDataType(modelshape.type()) != aninput.getDatatype()) {
+        smsg.str("");
+        smsg << "Migraph worker model and input data types don't match:   "
+             << toDataType(modelshape.type()) << " vs "
+             << aninput.getDatatype();
+        throw(invalid_argument(smsg.str()));
+      }
+
+      // check that lengths() and type match
+      auto llen = modelshape.lengths();
+      // clang-format off
         //    compare each dimension of shapes except the 0'th (batch size)
         //  TODO(bpickrel): the following check works inconsistently between different example client scripts.
         // It accepts inputs from the yolo script but rejects hello_migraphx.py inputs
@@ -386,179 +381,134 @@ void MIGraphXWorker::doRun(BatchPtrQueue* input_queue) {
         //     throw invalid_argument(smsg.str());
         //   }
         // }
-        // clang-format on
+      // clang-format on
 
-        auto* a_data = aninput.getData();  //  void *
-        params.add(aname.c_str(), migraphx::argument(modelshape, a_data));
-      }
-      // If there were fewer requests in the batch than the stated batch size,
-      // pad the various input tensors with copies of the 0'th request's data.
-
-      if (pad_batch_) {
-        // for each named input channel
-        for (const auto& aninput : inputs0) {
-          auto aname = aninput.getName();
-          // Look up the shape by name in the model, but if there's only 1 input
-          // then the name in the request isn't required to match.
-          if (inputs0.size() == 1) {
-            aname = param_shapes.names().front();
-          }
-          auto* a_data = static_cast<char*>(aninput.getData());
-          // For each empty slot in buffer, i.e. from end of real requests up to
-          // batch size
-          for (size_t req_idx = batch->getRequests().size();
-               req_idx < batch_size_; req_idx++) {
-            memcpy(a_data + req_idx * input_sizes_[aname], a_data,
-                   input_sizes_[aname]);
-          }
+      auto* a_data = aninput.getData();  //  void *
+      params.add(aname.c_str(), migraphx::argument(modelshape, a_data));
+    }
+    // If there were fewer requests in the batch than the stated batch size,
+    // pad the various input tensors with copies of the 0'th request's data.
+
+    if (pad_batch_) {
+      // for each named input channel
+      for (const auto& aninput : inputs0) {
+        auto aname = aninput.getName();
+        // Look up the shape by name in the model, but if there's only 1 input
+        // then the name in the request isn't required to match.
+        if (inputs0.size() == 1) {
+          aname = param_shapes.names().front();
+        }
+        auto* a_data = static_cast<char*>(aninput.getData());
+        // For each empty slot in buffer, i.e. from end of real requests up to
+        // batch size
+        for (size_t req_idx = batch->getRequests().size();
+             req_idx < batch_size_; req_idx++) {
+          memcpy(a_data + req_idx * input_sizes_[aname], a_data,
+                 input_sizes_[aname]);
         }
       }
+    }
 
-      //
-      // Run the inference
-      //
-
-      AMDINFER_LOG_INFO(logger, "Beginning migraphx eval");
-      timer.add("eval_start");
-      migraphx::api::arguments migraphx_output = this->prog_.eval(params);
-      timer.add("eval_end");
-      auto eval_duration_us = timer.count<std::micro>("eval_start", "eval_end");
-      [[maybe_unused]] auto eval_duration_s = eval_duration_us / std::mega::num;
-      AMDINFER_LOG_INFO(
-        logger, std::string("Finished migraphx eval; batch size: ") +
-                  std::to_string(batch_size_) + "  elapsed time: " +
-                  std::to_string(eval_duration_us) + " us.  Images/sec: " +
-                  std::to_string(batch_size_ / (eval_duration_s)));
-
-      //
-      //           Fetch the results and populate response to each request in
-      //           the batch
-      //
-
-      // for each request in the batch
-      for (unsigned int j = 0; j < batch->size(); j++) {
-        const auto& req = batch->getRequest(j);
-        try {
-          InferenceResponse resp;
-          resp.setID(req->getID());
-          resp.setModel("migraphx");
-
-          // We don't use the outputs portion of the request currently.  It is
-          // part of the kserve format specification, which the Inference Server
-          // is intended to follow. "The $request_output JSON is used to request
-          // which output tensors should be returned from the model."
-          // https://github.com/kserve/kserve/blob/master/docs/predict-api/v2/required_api.md
-          //
-          // Selecting the request output is only relevant to models that have
-          // more than one output tensor.
-          //
-
-          // Fetch the vector shape, data, etc. for output from the
-          // parsed/compiled model
-          migraphx::api::shapes output_shapes = prog_.get_output_shapes();
-
-          //
-          // Transfer the migraphx results to output
-          //
-          size_t result_size =
-            migraphx_output.size();  //   Resnet models have 1 output; yolo and
-                                     //   bert models have 3
-
-          // For each output channel in result:
-          //
-          for (size_t i = 0; i < result_size; i++) {
-            // the buffer to populate for return
-            InferenceResponseOutput output;
-
-            migraphx_shape_datatype_t output_type = output_shapes[i].type();
-            amdinfer::DataType output_dt = toDataType(output_type);
-            output.setDatatype(output_dt);
-
-            auto this_output = migraphx_output[i];
-            migraphx::api::shape shape = this_output.get_shape();
-            auto lengths = shape.lengths();
-
-            auto num_results =
-              util::containerProduct(lengths.begin() + 1, lengths.end());
-
-            // remove the 0'th dimension (batch size) from lengths
-            lengths.erase(lengths.begin());
-            // size of each result array, bytes
-            size_t size_of_result = num_results * output_dt.size();
-
-            // pointer to offset in data blob
-            char* results = this_output.data() + j * size_of_result;
-
-            // the kserve specification for response output is at
-            // https://github.com/kserve/kserve/blob/master/docs/predict-api/v2/required_api.md#response-output
-            //
-            // The outputs buffer in the InferenceRequest is not used or
-            // enforced at the time of writing this, but here it is. Give the
-            // output a default name if necessary.
-            auto outputs =
-              req->getOutputs();  // one result vector for each request
-
-            std::string output_name;
-            if (i < outputs.size()) {
-              output_name = outputs[i].getName();
-            }
-
-            if (output_name.empty()) {
-              output.setName(inputs0[0].getName());
-            } else {
-              output.setName(output_name);
-            }
-            output.setShape(lengths);
-
-            // Copy migraphx results to a buffer and add to output
-            std::vector<std::byte> buffer;
-            buffer.resize(size_of_result);
-            memcpy(buffer.data(), results, size_of_result);
-            output.setData(std::move(buffer));
-            resp.addOutput(output);
-          }
-          // respond back to the client
-          req->runCallbackOnce(resp);
-#ifdef AMDINFER_ENABLE_METRICS
-          Metrics::getInstance().incrementCounter(
-            MetricCounterIDs::PipelineEgressWorker);
-          timer.add("batch_time", batch->getTime(j));
-          timer.add("request_latency");
-          auto duration =
-            timer.count<std::micro>("batch_time", "request_latency");
-          Metrics::getInstance().observeSummary(
-            MetricSummaryIDs::RequestLatency, duration);
-#endif
-        } catch (const std::exception& e) {
-          AMDINFER_LOG_ERROR(logger, e.what());
-          // Pass error message back as reply to request; continue processing
-          // more inference requests
+    //
+    // Run the inference
+    //
 
-          req->runCallbackError(
-            std::string("Error processing Migraphx request: ") + e.what());
-        }
-      }  // end j, request
-    } catch (const std::exception& e) {
-      // This outer catch block catches exceptions in evaluation of the batch.
-      AMDINFER_LOG_ERROR(logger, e.what());
-      // Pass error message back as reply for each request in the batch
-      const auto& requests = batch->getRequests();
-      for (const auto& req_e : requests) {
-        req_e->runCallbackError(std::string("Migraphx inference error: ") +
-                                e.what());
+    AMDINFER_LOG_INFO(logger, "Beginning migraphx eval");
+    timer.add("eval_start");
+    migraphx::api::arguments migraphx_output = this->prog_.eval(params);
+    timer.add("eval_end");
+    auto eval_duration_us = timer.count<std::micro>("eval_start", "eval_end");
+    [[maybe_unused]] auto eval_duration_s = eval_duration_us / std::mega::num;
+    AMDINFER_LOG_INFO(
+      logger,
+      std::string("Finished migraphx eval; batch size: ") +
+        std::to_string(batch_size_) +
+        "  elapsed time: " + std::to_string(eval_duration_us) +
+        " us.  Images/sec: " + std::to_string(batch_size_ / (eval_duration_s)));
+
+    //
+    //           Fetch the results and populate response to each request in
+    //           the batch
+    //
+
+    new_batch = batch->propagate();
+    const auto batch_size = batch->size();
+
+    // Fetch the vector shape, data, etc. for output from the
+    // parsed/compiled model
+    migraphx::api::shapes output_shapes = prog_.get_output_shapes();
+    std::vector<DataType> datatypes;
+    datatypes.reserve(output_shapes.size());
+
+    size_t num_output_tensors = migraphx_output.size();
+    assert(output_shapes.size() == num_output_tensors);
+    input_buffers.reserve(num_output_tensors);
+    for (auto i = 0U; i < num_output_tensors; ++i) {
+      auto migraphx_shape = migraphx_output[i].get_shape();
+      auto shape = migraphx_shape.lengths();
+      // erase the leading batch size to get the tensor size
+      shape.erase(shape.begin());
+
+      datatypes.push_back(toDataType(output_shapes[i].type()));
+
+      Tensor tensor{"", shape, datatypes.back()};
+      input_buffers.emplace_back(
+        pool->get(next_allocators_, tensor, batch_size));
+    }
+
+    for (unsigned int j = 0; j < batch_size; j++) {
+      const auto& req = batch->getRequest(j);
+      auto new_request = req->propagate();
+
+      for (auto i = 0U; i < num_output_tensors; ++i) {
+        auto migraphx_shape = migraphx_output[i].get_shape();
+        auto shape = migraphx_shape.lengths();
+        // erase the leading batch size to get the tensor size
+        shape.erase(shape.begin());
+
+        const auto& datatype = datatypes.at(i);
+
+        auto size = util::containerProduct(shape) * datatype.size();
+        // TODO(bpickrel): the BERT example's last tensor produces a shape of
+        // [10], which becomes zero after erasing the batch size, resulting in
+        // a tensor of size zero. What does this mean?
+        // assert(size > 0);
+        auto* data_ptr = input_buffers.at(i)->data(size * j);
+        const char* results = migraphx_output[i].data() + (size * j);
+
+        std::memcpy(data_ptr, results, size);
+        InferenceRequestInput input{data_ptr, shape, datatype, ""};
+
+        new_request->addInputTensor(std::move(input));
       }
+
+      new_batch->addRequest(new_request);
+
+      new_batch->setModel(j, "migraphx");
+
+    }  // end j, request
+  } catch (const std::exception& e) {
+    // This outer catch block catches exceptions in evaluation of the batch.
+    AMDINFER_LOG_ERROR(logger, e.what());
+    // Pass error message back as reply for each request in the batch
+    const auto& requests = batch->getRequests();
+    for (const auto& req_e : requests) {
+      req_e->runCallbackError(std::string("Migraphx inference error: ") +
+                              e.what());
     }
+  }
 
-    timer.add("batch_stop");
-    this->returnInputBuffers(std::move(batch));
-    [[maybe_unused]] auto duration =
-      timer.count<std::micro>("batch_start", "batch_stop");
-    AMDINFER_LOG_INFO(
-      logger, std::string("Finished migraphx batch processing; batch size: ") +
-                std::to_string(batch_size_) +
-                "  elapsed time: " + std::to_string(duration) + " us");
-  }  // end while (batch)
-  AMDINFER_LOG_INFO(logger, "Migraphx::doRun ending");
+  new_batch->setBuffers(std::move(input_buffers), {});
+
+  timer.add("batch_stop");
+  [[maybe_unused]] auto duration =
+    timer.count<std::micro>("batch_start", "batch_stop");
+  AMDINFER_LOG_INFO(
+    logger, std::string("Finished migraphx batch processing; batch size: ") +
+              std::to_string(batch_size_) +
+              "  elapsed time: " + std::to_string(duration) + " us");
+
+  return new_batch;
 }
 
 void MIGraphXWorker::doRelease() {}
@@ -570,6 +520,6 @@ extern "C" {
 // using smart pointer here may cause problems inside shared object so managing
 // manually
 amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::MIGraphXWorker("MIGraphX", "gpu");
+  return new amdinfer::workers::MIGraphXWorker("MIGraphX", "GPU", true);
 }
 }  // extern C
diff --git a/src/amdinfer/workers/pt_zendnn.cpp b/src/amdinfer/workers/pt_zendnn.cpp
index cfd756d8a..f32ac32cd 100644
--- a/src/amdinfer/workers/pt_zendnn.cpp
+++ b/src/amdinfer/workers/pt_zendnn.cpp
@@ -40,8 +40,8 @@
 #include "amdinfer/declarations.hpp"             // for InferenceResponseOutput
 #include "amdinfer/observation/logging.hpp"  // for Logger, AMDINFER_LOG_INFO
 #include "amdinfer/observation/metrics.hpp"  // for Metrics, MetricCounterIDs
-#include "amdinfer/observation/tracing.hpp"  // for Trace
 #include "amdinfer/util/containers.hpp"      // for containerProduct
+#include "amdinfer/util/memory.hpp"          // for copy
 #include "amdinfer/util/thread.hpp"          // for setThreadName
 #include "amdinfer/util/timer.hpp"           // for Timer
 #include "amdinfer/workers/worker.hpp"       // for Worker, kNumBufferAuto
@@ -61,16 +61,15 @@ const int kResNetOutputClasses = 1000;
  * returns the corresponding number of output tensors.
  *
  */
-class PtZendnn : public Worker {
+class PtZendnn : public SingleThreadedWorker {
  public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
+  using SingleThreadedWorker::SingleThreadedWorker;
   [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
 
  private:
   void doInit(ParameterMap* parameters) override;
   void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
+  BatchPtr doRun(Batch* batch, const MemoryPool* pool) override;
   void doRelease() override;
   void doDestroy() override;
 
@@ -95,10 +94,6 @@ class PtZendnn : public Worker {
   DataType input_dt_ = DataType::FP32;
 };
 
-std::thread PtZendnn::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&PtZendnn::run, this, input_queue);
-}
-
 std::vector<MemoryAllocators> PtZendnn::getAllocators() const {
   return {MemoryAllocators::Cpu};
 }
@@ -176,156 +171,110 @@ void PtZendnn::doAcquire(ParameterMap* parameters) {
   this->metadata_.setName("PtZendnn");
 }
 
-void PtZendnn::doRun(BatchPtrQueue* input_queue) {
-  std::unique_ptr<Batch> batch;
-  util::setThreadName("PtZendnn");
+BatchPtr PtZendnn::doRun(Batch* batch, const MemoryPool* pool) {
 #ifdef AMDINFER_ENABLE_LOGGING
   const auto& logger = this->getLogger();
 #endif
 
-  while (true) {
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
-    AMDINFER_LOG_DEBUG(logger, "Got request in PtZendnn. Size: " +
-                                 std::to_string(batch->size()));
-
-    std::vector<InferenceResponse> responses;
-    responses.reserve(batch->size());
-
-    // This ensures no gradient is calculated and provide performance boost
-    torch::NoGradGuard no_grad;
-    c10::InferenceMode guard;
+  // This ensures no gradient is calculated and provide performance boost
+  torch::NoGradGuard no_grad;
+  c10::InferenceMode guard;
 
-    size_t input_size = 0;
-    std::vector<torch::jit::IValue> input_vec;
-    auto tensors = static_cast<int>(batch->size());
+  size_t input_size = 0;
+  std::vector<torch::jit::IValue> input_vec;
+  auto tensors = static_cast<int>(batch->size());
 
-    // Initialize a PT tensor with required shape
-    torch::Tensor input_tensor = torch::empty(
-      {tensors, image_channels_, image_height_, image_width_}, torch::kF32);
+  // Initialize a PT tensor with required shape
+  torch::Tensor input_tensor = torch::empty(
+    {tensors, image_channels_, image_height_, image_width_}, torch::kF32);
 
 #ifdef AMDINFER_ENABLE_METRICS
-    Metrics::getInstance().incrementCounter(
-      MetricCounterIDs::PipelineIngressWorker);
-#endif
-    size_t vec_size = 0;
-    util::Timer timer{true};
-    for (unsigned int j = 0; j < batch->size(); j++) {
-      const auto& req = batch->getRequest(j);
-
-#ifdef AMDINFER_ENABLE_TRACING
-      const auto& trace = batch->getTrace(j);
-      trace->startSpan("ptzendnn");
+  Metrics::getInstance().incrementCounter(
+    MetricCounterIDs::PipelineIngressWorker);
 #endif
-      auto& resp = responses.emplace_back();
-      resp.setID(req->getID());
-      resp.setModel("PTModel");
-
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-      AMDINFER_LOG_DEBUG(logger,
-                         "Size of input: " + std::to_string(inputs.size()));
-
-      // Get all the inputs from the requests and copy to the PT tensor
-      for (const auto& input : inputs) {
-        auto* input_buffer = input.getData();
-        const auto& input_shape = input.getShape();
-        input_size = util::containerProduct(input_shape);
-
-        auto* float_buffer = static_cast<float*>(input_buffer);
-        std::copy(float_buffer, float_buffer + input_size,
-                  input_tensor.data_ptr<float>() + vec_size);
-        vec_size = vec_size + input_size;
-      }
+  size_t vec_size = 0;
+  util::Timer timer{true};
+  for (unsigned int j = 0; j < batch->size(); j++) {
+    const auto& req = batch->getRequest(j);
+
+    auto inputs = req->getInputs();
+    auto outputs = req->getOutputs();
+    AMDINFER_LOG_DEBUG(logger,
+                       "Size of input: " + std::to_string(inputs.size()));
+
+    // Get all the inputs from the requests and copy to the PT tensor
+    for (const auto& input : inputs) {
+      auto* input_buffer = input.getData();
+      const auto& input_shape = input.getShape();
+      input_size = util::containerProduct(input_shape);
+
+      auto* float_buffer = static_cast<float*>(input_buffer);
+      std::copy(float_buffer, float_buffer + input_size,
+                input_tensor.data_ptr<float>() + vec_size);
+      vec_size = vec_size + input_size;
     }
+  }
 
-    // Create the inputs and output tensor
-    input_vec.emplace_back(input_tensor);
-    c10::IValue prediction;
-
-    // Run through the model to get the predictions
-    timer.add("infer_start");
-    try {
-      prediction = this->model_.forward(input_vec);
-    } catch (const c10::Error& e) {
-      AMDINFER_LOG_ERROR(logger, "Model not suported/Issue with the model");
-      for (const auto& req : *batch) {
-        req->runCallbackError("Something went wrong");
-      }
-    }
-    timer.add("infer_stop");
-    {
-      [[maybe_unused]] auto duration =
-        timer.count<std::milli>("infer_start", "infer_stop");
-      AMDINFER_LOG_INFO(logger, "Time (ms) taken for " +
-                                  std::to_string(batch->size()) +
-                                  " images: " + std::to_string(duration));
-    }
-    at::Tensor output_tensor;
-    if (!prediction.isTuple()) {
-      output_tensor = prediction.toTensor();
-    } else {
-      // For some models like InceptionV3 and GoogleNet which returns Tuple
-      output_tensor = prediction.toTuple()->elements()[0].toTensor();
+  // Create the inputs and output tensor
+  input_vec.emplace_back(input_tensor);
+  c10::IValue prediction;
+
+  // Run through the model to get the predictions
+  timer.add("infer_start");
+  try {
+    prediction = this->model_.forward(input_vec);
+  } catch (const c10::Error& e) {
+    AMDINFER_LOG_ERROR(logger, "Model not suported/Issue with the model");
+    for (const auto& req : *batch) {
+      req->runCallbackError("Something went wrong");
     }
+  }
+  timer.add("infer_stop");
+  {
+    [[maybe_unused]] auto duration =
+      timer.count<std::milli>("infer_start", "infer_stop");
+    AMDINFER_LOG_INFO(logger, "Time (ms) taken for " +
+                                std::to_string(batch->size()) +
+                                " images: " + std::to_string(duration));
+  }
+  at::Tensor output_tensor;
+  if (!prediction.isTuple()) {
+    output_tensor = prediction.toTensor();
+  } else {
+    // For some models like InceptionV3 and GoogleNet which returns Tuple
+    output_tensor = prediction.toTuple()->elements()[0].toTensor();
+  }
 
-    // Copy the output from the model to the response object
-    size_t response_size = output_classes_;
-    std::vector<size_t> new_shape = {response_size};
-    for (unsigned int k = 0; k < batch->size(); k++) {
-      const auto& req = batch->getRequest(k);
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-      auto& resp = responses[k];
-
-      for (unsigned int i = 0; i < inputs.size(); i++) {
-        InferenceResponseOutput output;
-        std::vector<std::byte> buffer;
-        buffer.resize(response_size * sizeof(float));
-
-        memcpy(buffer.data(), output_tensor[i].data_ptr<float>(),
+  // Copy the output from the model to the response object
+  size_t response_size = output_classes_;
+  std::vector<size_t> new_shape = {response_size};
+
+  auto new_batch = batch->propagate();
+  std::vector<BufferPtr> input_buffers;
+  input_buffers.push_back(pool->get(next_allocators_,
+                                    Tensor{"name", new_shape, DataType::Fp32},
+                                    batch->size()));
+
+  for (unsigned int k = 0; k < batch->size(); k++) {
+    const auto& req = batch->getRequest(k);
+
+    auto new_request = req->propagate();
+    auto* data_ptr =
+      input_buffers.at(0)->data(k * response_size * DataType("Fp32").size());
+    new_request->addInputTensor(
+      InferenceRequestInput{data_ptr, new_shape, DataType::Fp32, ""});
+    util::copy(output_tensor[0].data_ptr<float>() + (k * response_size),
+               static_cast<std::byte*>(data_ptr),
                response_size * sizeof(float));
-        output.setData(std::move(buffer));
-
-        std::string output_name;
-        if (i < outputs.size()) {
-          output_name = outputs[i].getName();
-        }
-
-        if (output_name.empty()) {
-          output.setName(inputs[0].getName());
-        } else {
-          output.setName(output_name);
-        }
-
-        output.setShape(new_shape);
-        output.setDatatype(DataType::FP32);
-        resp.addOutput(output);
-      }
-
-#ifdef AMDINFER_ENABLE_TRACING
-      auto context = batch->getTrace(k)->propagate();
-      resp.setContext(std::move(context));
-#endif
-      timer.stop();
-      [[maybe_unused]] auto duration = timer.count<std::milli>();
-      AMDINFER_LOG_DEBUG(logger,
-                         "Total time taken: " + std::to_string(duration));
 
-      req->runCallbackOnce(resp);
+    new_batch->addRequest(new_request);
 
-#ifdef AMDINFER_ENABLE_METRICS
-      timer.add("metrics", batch->getTime(k));
-      duration = timer.count<std::micro>("metrics", "stop");
-      Metrics::getInstance().observeSummary(MetricSummaryIDs::RequestLatency,
-                                            duration);
-#endif
-    }
-    this->returnInputBuffers(std::move(batch));
+    new_batch->setModel(k, "PTModel");
   }
-  AMDINFER_LOG_INFO(logger, "PtZendnn ending");
+
+  new_batch->setBuffers(std::move(input_buffers), {});
+
+  return new_batch;
 }
 
 void PtZendnn::doRelease() {}
@@ -337,6 +286,6 @@ extern "C" {
 // using smart pointer here may cause problems inside shared object so managing
 // manually
 amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::PtZendnn("PtZendnn", "cpu");
+  return new amdinfer::workers::PtZendnn("PtZendnn", "CPU", true);
 }
 }  // extern C
diff --git a/src/amdinfer/workers/resnet50.cpp b/src/amdinfer/workers/resnet50.cpp
index e42da397d..1f4f27fb9 100644
--- a/src/amdinfer/workers/resnet50.cpp
+++ b/src/amdinfer/workers/resnet50.cpp
@@ -47,9 +47,9 @@
 #include "amdinfer/declarations.hpp"         // for BufferPtrs, InferenceRe...
 #include "amdinfer/observation/logging.hpp"  // for Logger
 #include "amdinfer/observation/metrics.hpp"  // for Metrics, MetricSummaryIDs
-#include "amdinfer/observation/tracing.hpp"  // for Trace
 #include "amdinfer/util/base64.hpp"          // for base64_decode
 #include "amdinfer/util/containers.hpp"      // for containerProduct
+#include "amdinfer/util/memory.hpp"          // for copy
 #include "amdinfer/util/parse_env.hpp"       // for autoExpandEnvironmentVa...
 #include "amdinfer/util/thread.hpp"          // for setThreadName
 #include "amdinfer/util/timer.hpp"           // for Timer
@@ -66,16 +66,15 @@ namespace amdinfer::workers {
  * classification IDs
  *
  */
-class ResNet50 : public Worker {
+class ResNet50 : public SingleThreadedWorker {
  public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
+  using SingleThreadedWorker::SingleThreadedWorker;
   [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
 
  private:
   void doInit(ParameterMap* parameters) override;
   void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
+  BatchPtr doRun(Batch* batch, const MemoryPool* pool) override;
   void doRelease() override;
   void doDestroy() override;
 
@@ -84,10 +83,6 @@ class ResNet50 : public Worker {
   AKS::AIGraph* graph_ = nullptr;
 };
 
-std::thread ResNet50::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&ResNet50::run, this, input_queue);
-}
-
 std::vector<MemoryAllocators> ResNet50::getAllocators() const {
   return {MemoryAllocators::Cpu};
 }
@@ -134,163 +129,86 @@ void ResNet50::doAcquire(ParameterMap* parameters) {
   this->metadata_.setName(this->graph_name_);
 }
 
-void ResNet50::doRun(BatchPtrQueue* input_queue) {
-  std::shared_ptr<InferenceRequest> req;
-  util::setThreadName("ResNet50");
-#ifdef AMDINFER_ENABLE_LOGGING
-  const auto& logger = this->getLogger();
-#endif
-
-  while (true) {
-    BatchPtr batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
-    AMDINFER_LOG_INFO(logger, "Got request in Resnet50");
-    std::vector<InferenceResponse> responses;
-    responses.reserve(batch->size());
-
-    std::vector<std::unique_ptr<vart::TensorBuffer>> v;
-    v.reserve(batch->getInputSize());
-
-    size_t tensor_count = 0;
-    for (unsigned int j = 0; j < batch->size(); j++) {
-      const auto& req = batch->getRequest(j);
-#ifdef AMDINFER_ENABLE_TRACING
-      auto& trace = batch->getTrace(j);
-      trace->startSpan("Resnet50");
-#endif
-      auto& resp = responses.emplace_back();
-      resp.setID(req->getID());
-      resp.setModel(this->graph_name_);
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-
-      uint64_t input_size = 0;
-      for (auto& input : inputs) {
-        auto* input_buffer = input.getData();
-
-        auto input_shape = input.getShape();
-
-        input_size = util::containerProduct(input_shape);
-        auto input_dtype = input.getDatatype();
-
-        std::vector<int> tensor_shape = {static_cast<int>(this->batch_size_)};
-        if (input_dtype == DataType::Uint8) {
-          if (tensor_count == 0) {
-            tensor_shape.insert(tensor_shape.end(), input_shape.begin(),
-                                input_shape.end());
-            v.emplace_back(std::make_unique<AKS::AksTensorBuffer>(
-              xir::Tensor::create(this->graph_name_, tensor_shape,
-                                  xir::create_data_type<unsigned char>())));
-          }
-          /// Copy input to AKS Buffers: Find a better way to share buffers
-          memcpy(reinterpret_cast<uint8_t*>(v[0]->data().first) +
-                   (tensor_count * input_size),
-                 input_buffer, input_size);
-        } else if (input_dtype == DataType::String) {
-          auto* idata = static_cast<char*>(input_buffer);
-          auto decoded_str = util::base64Decode(idata, input_size);
-          std::vector<char> data(decoded_str.begin(), decoded_str.end());
-          cv::Mat img = cv::imdecode(data, cv::IMREAD_UNCHANGED);
-          if (img.empty()) {
-            const char* error = "Decoded image is empty";
-            AMDINFER_LOG_ERROR(logger, error);
-            req->runCallbackError(error);
-            continue;
-          }
-          // set size to actual size of image instead of size of base64 str
-          input_size = img.step[0] * img.rows;
-
-          if (tensor_count == 0) {
-            tensor_shape.insert(
-              tensor_shape.end(),
-              {img.size().height, img.size().width, kImageChannels});
-            v.emplace_back(std::make_unique<AKS::AksTensorBuffer>(
-              xir::Tensor::create(this->graph_name_, tensor_shape,
-                                  xir::create_data_type<unsigned char>())));
-          }
-
-          // TODO(varunsh): assuming decoded image will be right size
-          memcpy(reinterpret_cast<uint8_t*>(v[0]->data().first) +
-                   (tensor_count * input_size),
-                 img.data, input_size);
-        }
-        tensor_count = (tensor_count + 1) % this->batch_size_;
-      }
-    }
+BatchPtr ResNet50::doRun(Batch* batch, const MemoryPool* pool) {
+  std::vector<std::unique_ptr<vart::TensorBuffer>> v;
+  v.reserve(batch->getInputSize());
+
+  size_t tensor_count = 0;
+  for (unsigned int j = 0; j < batch->size(); j++) {
+    const auto& req = batch->getRequest(j);
+    auto inputs = req->getInputs();
 
-    std::future<std::vector<std::unique_ptr<vart::TensorBuffer>>> future =
-      this->sys_manager_->enqueueJob(this->graph_, "", std::move(v), nullptr);
+    uint64_t input_size = 0;
+    for (auto& input : inputs) {
+      auto* input_buffer = input.getData();
 
-    auto out_data_descriptor = future.get();
+      auto input_shape = input.getShape();
 
-    int* top_k_data =
-      reinterpret_cast<int*>(out_data_descriptor[0]->data().first);
-    auto shape = out_data_descriptor[0]->get_tensor()->get_shape();
+      input_size = util::containerProduct(input_shape);
 
-    int response_size = 0;
-    std::vector<uint64_t> new_shape;
-    if (shape.size() > 1) {  // [batch, a, b, c]
-      response_size = util::containerProduct(shape.begin() + 1, shape.end());
-      // We exclude the batch size from the returned shape
-      new_shape.reserve(shape.size() - 1);
-      for (size_t i = 0; i < shape.size() - 1; i++) {
-        new_shape.push_back(shape[i + 1]);
+      std::vector<int> tensor_shape = {static_cast<int>(this->batch_size_)};
+      if (tensor_count == 0) {
+        tensor_shape.insert(tensor_shape.end(), input_shape.begin(),
+                            input_shape.end());
+        v.emplace_back(std::make_unique<AKS::AksTensorBuffer>(
+          xir::Tensor::create(this->graph_name_, tensor_shape,
+                              xir::create_data_type<unsigned char>())));
       }
-    } else {  // [a]
-      new_shape.push_back(shape[0]);
-      response_size = shape[0];
+      /// Copy input to AKS Buffers: Find a better way to share buffers
+      memcpy(reinterpret_cast<uint8_t*>(v[0]->data().first) +
+               (tensor_count * input_size),
+             input_buffer, input_size);
+      tensor_count = (tensor_count + 1) % this->batch_size_;
     }
+  }
 
-    for (unsigned int k = 0; k < batch->size(); k++) {
-      const auto& req = batch->getRequest(k);
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-      auto& resp = responses[k];
-
-      for (unsigned int i = 0; i < inputs.size(); i++) {
-        InferenceResponseOutput output;
-        std::vector<std::byte> buffer;
-        buffer.resize(response_size * sizeof(int));
-        memcpy(buffer.data(), top_k_data + (i * response_size),
-               response_size * sizeof(int));
-        output.setData(std::move(buffer));
-
-        std::string output_name;
-        if (i < outputs.size()) {
-          output_name = outputs[i].getName();
-        }
-
-        if (output_name.empty()) {
-          output.setName(inputs[0].getName());
-        } else {
-          output.setName(output_name);
-        }
-
-        output.setShape(new_shape);
-
-        output.setDatatype(DataType::Uint32);
-        resp.addOutput(output);
-      }
+  std::future<std::vector<std::unique_ptr<vart::TensorBuffer>>> future =
+    this->sys_manager_->enqueueJob(this->graph_, "", std::move(v), nullptr);
+
+  auto out_data_descriptor = future.get();
 
-#ifdef AMDINFER_ENABLE_METRICS
-      util::Timer timer{batch->getTime(k)};
-      timer.stop();
-      auto duration = timer.count<std::micro>();
-      Metrics::getInstance().observeSummary(MetricSummaryIDs::RequestLatency,
-                                            duration);
-#endif
-#ifdef AMDINFER_ENABLE_TRACING
-      auto context = batch->getTrace(k)->propagate();
-      resp.setContext(std::move(context));
-#endif
-      req->runCallbackOnce(resp);
+  int* top_k_data =
+    reinterpret_cast<int*>(out_data_descriptor[0]->data().first);
+  auto shape = out_data_descriptor[0]->get_tensor()->get_shape();
+
+  int response_size = 0;
+  std::vector<uint64_t> new_shape;
+  if (shape.size() > 1) {  // [batch, a, b, c]
+    response_size = util::containerProduct(shape.begin() + 1, shape.end());
+    // We exclude the batch size from the returned shape
+    new_shape.reserve(shape.size() - 1);
+    for (size_t i = 0; i < shape.size() - 1; i++) {
+      new_shape.push_back(shape[i + 1]);
     }
-    this->returnInputBuffers(std::move(batch));
+  } else {  // [a]
+    new_shape.push_back(shape[0]);
+    response_size = shape[0];
   }
-  AMDINFER_LOG_INFO(logger, "ResNet50 ending");
+
+  auto new_batch = batch->propagate();
+  std::vector<BufferPtr> input_buffers;
+  input_buffers.push_back(pool->get(next_allocators_,
+                                    Tensor{"name", new_shape, DataType::Uint32},
+                                    batch->size()));
+
+  for (unsigned int k = 0; k < batch->size(); k++) {
+    const auto& req = batch->getRequest(k);
+    auto new_request = req->propagate();
+    auto* data_ptr =
+      input_buffers.at(0)->data(k * response_size * DataType("Uint32").size());
+    new_request->addInputTensor(
+      InferenceRequestInput{data_ptr, new_shape, DataType::Uint32, ""});
+    util::copy(top_k_data + (k * response_size),
+               static_cast<std::byte*>(data_ptr), response_size * sizeof(int));
+
+    new_batch->addRequest(new_request);
+
+    new_batch->setModel(k, graph_name_);
+  }
+
+  new_batch->setBuffers(std::move(input_buffers), {});
+
+  return new_batch;
 }
 
 void ResNet50::doRelease() {}
@@ -302,6 +220,6 @@ extern "C" {
 // using smart pointer here may cause problems inside shared object so managing
 // manually
 amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::ResNet50("ResNet50", "AKS");
+  return new amdinfer::workers::ResNet50("ResNet50", "AKS", true);
 }
 }  // extern C
diff --git a/src/amdinfer/workers/resnet50_stream.cpp b/src/amdinfer/workers/resnet50_stream.cpp
index a81625f5e..200053ebf 100644
--- a/src/amdinfer/workers/resnet50_stream.cpp
+++ b/src/amdinfer/workers/resnet50_stream.cpp
@@ -75,16 +75,15 @@ namespace workers {
  * video and sends the inverted frames back to the client over a websocket.
  *
  */
-class ResNet50Stream : public Worker {
+class ResNet50Stream : public SingleThreadedWorker {
  public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
+  using SingleThreadedWorker::SingleThreadedWorker;
   [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
 
  private:
   void doInit(ParameterMap* parameters) override;
   void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
+  BatchPtr doRun(Batch* batch, const MemoryPool* pool) override;
   void doRelease() override;
   void doDestroy() override;
 
@@ -93,10 +92,6 @@ class ResNet50Stream : public Worker {
   AKS::AIGraph* graph_ = nullptr;
 };
 
-std::thread ResNet50Stream::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&ResNet50Stream::run, this, input_queue);
-}
-
 std::vector<MemoryAllocators> ResNet50Stream::getAllocators() const {
   return {MemoryAllocators::Cpu};
 }
@@ -146,144 +141,95 @@ void ResNet50Stream::doAcquire(ParameterMap* parameters) {
   this->metadata_.setName(this->graph_name_);
 }
 
-void ResNet50Stream::doRun(BatchPtrQueue* input_queue) {
+BatchPtr ResNet50Stream::doRun(Batch* batch,
+                               [[maybe_unused]] const MemoryPool* pool) {
   std::shared_ptr<InferenceRequest> req;
-  util::setThreadName("ResNet50Stream");
 #ifdef AMDINFER_ENABLE_LOGGING
   const auto& logger = this->getLogger();
 #endif
 
-  while (true) {
-    BatchPtr batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
-
-    AMDINFER_LOG_INFO(logger, "Got request in ResNet50Stream");
-    for (const auto& req : *batch) {
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-      auto key = req->getParameters().get<std::string>("key");
-      for (auto& input : inputs) {
-        auto* input_buffer = input.getData();
-
-        const auto* idata = static_cast<char*>(input_buffer);
-        std::string data{idata, input.getSize()};
-
-        cv::VideoCapture cap(data);  // open the video file
-        if (!cap.isOpened()) {       // check if we succeeded
-          const char* error = "Cannot open video file";
-          AMDINFER_LOG_ERROR(logger, error);
-          req->runCallbackError(error);
-          continue;
-        }
+  for (const auto& req : *batch) {
+    auto inputs = req->getInputs();
+    auto outputs = req->getOutputs();
+    auto key = req->getParameters().get<std::string>("key");
+    for (auto& input : inputs) {
+      auto* input_buffer = input.getData();
+
+      const auto* idata = static_cast<char*>(input_buffer);
+      std::string data{idata, input.getSize()};
+
+      cv::VideoCapture cap(data);  // open the video file
+      if (!cap.isOpened()) {       // check if we succeeded
+        const char* error = "Cannot open video file";
+        AMDINFER_LOG_ERROR(logger, error);
+        req->runCallbackError(error);
+        continue;
+      }
 
-        InferenceResponse resp;
-        resp.setID(req->getID());
-        resp.setModel("aks_detect_stream");
+      InferenceResponse resp;
+      resp.setID(req->getID());
+      resp.setModel("aks_detect_stream");
 
-        // contains the number of frames in the video;
-        auto count =
-          static_cast<size_t>(cap.get(VidProps::CAP_PROP_FRAME_COUNT));
-        if (input.getParameters().has("count")) {
-          auto requested_count = input.getParameters().get<int32_t>("count");
-          count = std::min(count, static_cast<size_t>(requested_count));
-        }
-        double fps = cap.get(VidProps::CAP_PROP_FPS);
-        auto video_width = cap.get(VidProps::CAP_PROP_FRAME_WIDTH);
-        auto video_height = cap.get(VidProps::CAP_PROP_FRAME_HEIGHT);
-
-        InferenceResponseOutput output;
-        output.setName("key");
-        output.setDatatype(DataType::String);
-        std::string metadata = "[" + std::to_string(video_width) + "," +
-                               std::to_string(video_height) + "]";
-        auto message = constructMessage(key, std::to_string(fps), metadata);
-        std::vector<std::byte> buffer;
-        buffer.resize(message.size());
-        memcpy(buffer.data(), message.data(), message.size());
-        output.setData(std::move(buffer));
-        output.setShape({message.size()});
-        resp.addOutput(output);
-        req->runCallback(resp);
-        // round to nearest multiple of batch size
-        auto count_adjusted = count - (count % this->batch_size_);
-        std::queue<
-          std::future<std::vector<std::unique_ptr<vart::TensorBuffer>>>>
-          futures;
-        std::queue<std::string> frames;
-        for (unsigned int num_frames = 0; num_frames < count_adjusted;
-             num_frames += this->batch_size_) {
-          std::vector<std::unique_ptr<vart::TensorBuffer>> v;
-          v.reserve(1);
-          v.emplace_back(std::make_unique<AKS::AksTensorBuffer>(
-            xir::Tensor::create("resnet-stream",
-                                {static_cast<int>(this->batch_size_),
-                                 kImageHeight, kImageWidth, kImageChannels},
-                                xir::create_data_type<unsigned char>())));
-
-          for (size_t i = 0; i < this->batch_size_; i++) {
-            cv::Mat frame;
-            cap >> frame;  // get the next frame from video
-            // if (frame.empty()) {
-            //   continue;
-            // }
-            cv::resize(frame, frame, cv::Size(kImageWidth, kImageHeight));
-
-            memcpy(
-              reinterpret_cast<uint8_t*>(v[0]->data().first) + (i * kImageSize),
-              frame.data, kImageSize);
-
-            std::vector<unsigned char> buf;
-            cv::imencode(".jpg", frame, buf);
-            const auto* enc_msg = reinterpret_cast<const char*>(buf.data());
-            std::string encoded = util::base64Encode(enc_msg, buf.size());
-            frames.push("data:image/jpg;base64," + encoded);
-          }
-          futures.push(this->sys_manager_->enqueueJob(this->graph_, "",
-                                                      std::move(v), nullptr));
-          auto status = futures.front().wait_for(std::chrono::seconds(0));
-          if (status == std::future_status::ready) {
-            std::vector<std::unique_ptr<vart::TensorBuffer>>
-              out_data_descriptor = futures.front().get();
-            futures.pop();
-            int* top_k_data =
-              reinterpret_cast<int*>(out_data_descriptor[0]->data().first);
-            for (unsigned int i = 0; i < this->batch_size_; i++) {
-              std::string labels = "[";
-              for (unsigned int j = 0; j < kResnetClassifications; j++) {
-                auto y = std::to_string(j * kBoxHeight);
-                auto label =
-                  std::to_string(top_k_data[(i * kResnetClassifications) + j]);
-                labels.append(R"({"fill": true, "box": [0,)");
-                labels.append(y + ",");
-                labels.append(kImageWidthStr + ",");
-                labels.append(kBoxHeightStr + R"(], "label": ")");
-                labels.append(label + "\"},");
-              }
-              labels.pop_back();  // trim trailing comma
-              labels += "]";
-              InferenceResponse resp;
-              resp.setID(req->getID());
-              resp.setModel("invert_video");
-
-              InferenceResponseOutput output;
-              output.setName("image");
-              output.setDatatype(DataType::String);
-              auto message = constructMessage(key, frames.front(), labels);
-              std::vector<std::byte> buffer;
-              buffer.resize(message.size());
-              memcpy(buffer.data(), message.data(), message.size());
-              output.setData(std::move(buffer));
-              output.setShape({message.size()});
-              resp.addOutput(output);
-              req->runCallback(resp);
-              frames.pop();
-            }
-          }
+      // contains the number of frames in the video;
+      auto count = static_cast<size_t>(cap.get(VidProps::CAP_PROP_FRAME_COUNT));
+      if (input.getParameters().has("count")) {
+        auto requested_count = input.getParameters().get<int32_t>("count");
+        count = std::min(count, static_cast<size_t>(requested_count));
+      }
+      double fps = cap.get(VidProps::CAP_PROP_FPS);
+      auto video_width = cap.get(VidProps::CAP_PROP_FRAME_WIDTH);
+      auto video_height = cap.get(VidProps::CAP_PROP_FRAME_HEIGHT);
+
+      InferenceResponseOutput output;
+      output.setName("key");
+      output.setDatatype(DataType::String);
+      std::string metadata = "[" + std::to_string(video_width) + "," +
+                             std::to_string(video_height) + "]";
+      auto message = constructMessage(key, std::to_string(fps), metadata);
+      std::vector<std::byte> buffer;
+      buffer.resize(message.size());
+      memcpy(buffer.data(), message.data(), message.size());
+      output.setData(std::move(buffer));
+      output.setShape({message.size()});
+      resp.addOutput(output);
+      req->runCallback(resp);
+      // round to nearest multiple of batch size
+      auto count_adjusted = count - (count % this->batch_size_);
+      std::queue<std::future<std::vector<std::unique_ptr<vart::TensorBuffer>>>>
+        futures;
+      std::queue<std::string> frames;
+      for (unsigned int num_frames = 0; num_frames < count_adjusted;
+           num_frames += this->batch_size_) {
+        std::vector<std::unique_ptr<vart::TensorBuffer>> v;
+        v.reserve(1);
+        v.emplace_back(std::make_unique<AKS::AksTensorBuffer>(
+          xir::Tensor::create("resnet-stream",
+                              {static_cast<int>(this->batch_size_),
+                               kImageHeight, kImageWidth, kImageChannels},
+                              xir::create_data_type<unsigned char>())));
+
+        for (size_t i = 0; i < this->batch_size_; i++) {
+          cv::Mat frame;
+          cap >> frame;  // get the next frame from video
+          // if (frame.empty()) {
+          //   continue;
+          // }
+          cv::resize(frame, frame, cv::Size(kImageWidth, kImageHeight));
+
+          memcpy(
+            reinterpret_cast<uint8_t*>(v[0]->data().first) + (i * kImageSize),
+            frame.data, kImageSize);
+
+          std::vector<unsigned char> buf;
+          cv::imencode(".jpg", frame, buf);
+          const auto* enc_msg = reinterpret_cast<const char*>(buf.data());
+          std::string encoded = util::base64Encode(enc_msg, buf.size());
+          frames.push("data:image/jpg;base64," + encoded);
         }
-        while (!futures.empty()) {
+        futures.push(this->sys_manager_->enqueueJob(this->graph_, "",
+                                                    std::move(v), nullptr));
+        auto status = futures.front().wait_for(std::chrono::seconds(0));
+        if (status == std::future_status::ready) {
           std::vector<std::unique_ptr<vart::TensorBuffer>> out_data_descriptor =
             futures.front().get();
           futures.pop();
@@ -322,9 +268,49 @@ void ResNet50Stream::doRun(BatchPtrQueue* input_queue) {
           }
         }
       }
+      while (!futures.empty()) {
+        std::vector<std::unique_ptr<vart::TensorBuffer>> out_data_descriptor =
+          futures.front().get();
+        futures.pop();
+        int* top_k_data =
+          reinterpret_cast<int*>(out_data_descriptor[0]->data().first);
+        for (unsigned int i = 0; i < this->batch_size_; i++) {
+          std::string labels = "[";
+          for (unsigned int j = 0; j < kResnetClassifications; j++) {
+            auto y = std::to_string(j * kBoxHeight);
+            auto label =
+              std::to_string(top_k_data[(i * kResnetClassifications) + j]);
+            labels.append(R"({"fill": true, "box": [0,)");
+            labels.append(y + ",");
+            labels.append(kImageWidthStr + ",");
+            labels.append(kBoxHeightStr + R"(], "label": ")");
+            labels.append(label + "\"},");
+          }
+          labels.pop_back();  // trim trailing comma
+          labels += "]";
+          InferenceResponse resp;
+          resp.setID(req->getID());
+          resp.setModel("invert_video");
+
+          InferenceResponseOutput output;
+          output.setName("image");
+          output.setDatatype(DataType::String);
+          auto message = constructMessage(key, frames.front(), labels);
+          std::vector<std::byte> buffer;
+          buffer.resize(message.size());
+          memcpy(buffer.data(), message.data(), message.size());
+          output.setData(std::move(buffer));
+          output.setShape({message.size()});
+          resp.addOutput(output);
+          req->runCallback(resp);
+          frames.pop();
+        }
+      }
     }
   }
-  AMDINFER_LOG_INFO(logger, "ResNet50Stream ending");
+
+  // okay because ensembles disabled for this worker
+  return nullptr;
 }
 
 void ResNet50Stream::doRelease() {}
@@ -338,6 +324,6 @@ extern "C" {
 // using smart pointer here may cause problems inside shared object so managing
 // manually
 amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::ResNet50Stream("ResNet50Stream", "AKS");
+  return new amdinfer::workers::ResNet50Stream("ResNet50Stream", "AKS", false);
 }
 }  // extern C
diff --git a/src/amdinfer/workers/responder.cpp b/src/amdinfer/workers/responder.cpp
new file mode 100644
index 000000000..d8d1a5e43
--- /dev/null
+++ b/src/amdinfer/workers/responder.cpp
@@ -0,0 +1,175 @@
+// Copyright 2023 Advanced Micro Devices, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/**
+ * @file
+ * @brief Implements the Responder worker
+ */
+
+#include <array>    // for array
+#include <cassert>  // for assert
+#include <cstddef>  // for size_t, byte
+#include <cstdint>  // for uint32_t, int32_t
+#include <cstring>  // for memcpy
+#include <memory>   // for unique_ptr, allocator
+#include <ratio>    // for micro
+#include <string>   // for string
+#include <thread>   // for thread
+#include <utility>  // for move
+#include <vector>   // for vector
+
+#include "amdinfer/batching/hard.hpp"    // for HardBatcher
+#include "amdinfer/build_options.hpp"    // for AMDINFER_ENABLE_TRACING
+#include "amdinfer/core/data_types.hpp"  // for DataType, DataType::Uint32
+#include "amdinfer/core/inference_request.hpp"  // for InferenceRequest, Infe...
+#include "amdinfer/core/inference_response.hpp"  // for InferenceResponse
+#include "amdinfer/core/parameters.hpp"          // for ParameterMap
+#include "amdinfer/declarations.hpp"         // for BufferPtr, InferenceRes...
+#include "amdinfer/observation/logging.hpp"  // for Logger
+#include "amdinfer/observation/metrics.hpp"  // for Metrics
+#include "amdinfer/observation/tracing.hpp"  // for startFollowSpan, SpanPtr
+#include "amdinfer/util/containers.hpp"      // for containerSum
+#include "amdinfer/util/queue.hpp"           // for BufferPtrsQueue
+#include "amdinfer/util/thread.hpp"          // for setThreadName
+#include "amdinfer/util/timer.hpp"           // for Timer
+#include "amdinfer/workers/worker.hpp"       // for Worker
+
+namespace amdinfer {
+
+const int kInputTensors = 2;
+const std::array<int, kInputTensors> kInputLengths = {1, 2};
+const int kOutputTensors = 3;
+const std::array<int, kOutputTensors> kOutputLengths = {1, 4, 3};
+
+namespace workers {
+
+/**
+ * @brief The Responder worker can run a compiled C++ "model".
+ *
+ */
+class Responder : public SingleThreadedWorker {
+ public:
+  using SingleThreadedWorker::SingleThreadedWorker;
+  [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
+
+ private:
+  void doInit(ParameterMap* parameters) override;
+  void doAcquire(ParameterMap* parameters) override;
+  BatchPtr doRun(Batch* batch, const MemoryPool* pool) override;
+  void doRelease() override;
+  void doDestroy() override;
+
+  std::vector<Tensor> input_tensors_;
+  std::vector<Tensor> output_tensors_;
+
+  // workers define what batcher implementation should be used for them.
+  // if not explicitly defined here, a default value is used from worker.hpp.
+  using Worker::makeBatcher;
+  std::vector<std::unique_ptr<Batcher>> makeBatcher(int num,
+                                                    ParameterMap* parameters,
+                                                    MemoryPool* pool) override {
+    return this->makeBatcher<HardBatcher>(num, parameters, pool);
+  };
+};
+
+std::vector<MemoryAllocators> Responder::getAllocators() const {
+  return {MemoryAllocators::Cpu};
+}
+
+void Responder::doInit(ParameterMap* parameters) {
+  constexpr auto kBatchSize = 1;
+
+  auto batch_size = kBatchSize;
+  if (parameters->has("batch_size")) {
+    batch_size = parameters->get<int32_t>("batch_size");
+  }
+  this->batch_size_ = batch_size;
+}
+
+void Responder::doAcquire([[maybe_unused]] ParameterMap* parameters) {
+  // TODO(varunsh): what should we do for metadata_?
+}
+
+BatchPtr Responder::doRun(Batch* batch,
+                          [[maybe_unused]] const MemoryPool* pool) {
+  const auto batch_size = batch->size();
+  for (unsigned int j = 0; j < batch_size; j++) {
+    const auto& req = batch->getRequest(j);
+
+    InferenceResponse resp;
+    resp.setID(req->getID());
+    resp.setModel(batch->getModel(j));
+    auto inputs = req->getInputs();
+    auto outputs = req->getOutputs();
+    for (unsigned int i = 0; i < inputs.size(); i++) {
+      const auto& input = inputs[i];
+      const auto* input_buffer = input.getData();
+
+      InferenceResponseOutput output;
+      output.setDatatype(input.getDatatype());
+      std::string output_name;
+      if (i < outputs.size()) {
+        output_name = outputs[i].getName();
+      }
+
+      if (output_name.empty()) {
+        output.setName(input.getName());
+      } else {
+        output.setName(output_name);
+      }
+      output.setShape(input.getShape());
+      std::vector<std::byte> buffer;
+      const auto size = input.getSize() * input.getDatatype().size();
+      buffer.resize(size);
+      memcpy(buffer.data(), input_buffer, size);
+      output.setData(std::move(buffer));
+      resp.addOutput(output);
+    }
+
+#ifdef AMDINFER_ENABLE_TRACING
+    const auto& trace = batch->getTrace(j);
+    auto context = trace->propagate();
+    resp.setContext(std::move(context));
+#endif
+
+    // respond back to the client
+    req->runCallbackOnce(resp);
+#ifdef AMDINFER_ENABLE_METRICS
+    Metrics::getInstance().incrementCounter(
+      MetricCounterIDs::PipelineEgressWorker);
+    util::Timer timer{batch->getTime(j)};
+    timer.stop();
+    auto duration = timer.count<std::micro>();
+    Metrics::getInstance().observeSummary(MetricSummaryIDs::RequestLatency,
+                                          duration);
+#endif
+  }
+  // okay because ensembles disabled for this worker
+  return nullptr;
+}
+
+void Responder::doRelease() {}
+void Responder::doDestroy() {}
+
+}  // namespace workers
+
+}  // namespace amdinfer
+
+extern "C" {
+// using smart pointer here may cause problems inside shared object so managing
+// manually
+amdinfer::workers::Worker* getWorker() {
+  return new amdinfer::workers::Responder("responder", "CPU", false);
+}
+}  // extern C
diff --git a/src/amdinfer/workers/tf_zendnn.cpp b/src/amdinfer/workers/tf_zendnn.cpp
index 0bcc9f3de..513704358 100644
--- a/src/amdinfer/workers/tf_zendnn.cpp
+++ b/src/amdinfer/workers/tf_zendnn.cpp
@@ -53,7 +53,7 @@
 #include "amdinfer/declarations.hpp"             // for InferenceResp...
 #include "amdinfer/observation/logging.hpp"      // for Logger, PROTE...
 #include "amdinfer/observation/metrics.hpp"      // for Metrics, Metr...
-#include "amdinfer/observation/tracing.hpp"      // for Trace
+#include "amdinfer/util/memory.hpp"              // for copy
 #include "amdinfer/util/thread.hpp"              // for setThreadName
 #include "amdinfer/util/timer.hpp"               // for Timer
 #include "amdinfer/workers/worker.hpp"           // for Worker, kNumB...
@@ -72,16 +72,15 @@ const int kResNetOutputClasses = 1000;
  * returns the corresponding number of output tensors.
  *
  */
-class TfZendnn : public Worker {
+class TfZendnn : public SingleThreadedWorker {
  public:
-  using Worker::Worker;
-  std::thread spawn(BatchPtrQueue* input_queue) override;
+  using SingleThreadedWorker::SingleThreadedWorker;
   [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
 
  private:
   void doInit(ParameterMap* parameters) override;
   void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
+  BatchPtr doRun(Batch* batch, const MemoryPool* pool) override;
   void doRelease() override;
   void doDestroy() override;
 
@@ -103,10 +102,6 @@ class TfZendnn : public Worker {
   DataType input_dt_ = DataType::Fp32;
 };
 
-std::thread TfZendnn::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&TfZendnn::run, this, input_queue);
-}
-
 std::vector<MemoryAllocators> TfZendnn::getAllocators() const {
   return {MemoryAllocators::Cpu};
 }
@@ -211,150 +206,95 @@ void TfZendnn::doAcquire(ParameterMap* parameters) {
   this->metadata_.setName("TfZendnn");
 }
 
-void TfZendnn::doRun(BatchPtrQueue* input_queue) {
-  util::setThreadName("TfZendnn");
+BatchPtr TfZendnn::doRun(Batch* batch, const MemoryPool* pool) {
 #ifdef AMDINFER_ENABLE_LOGGING
   const auto& logger = this->getLogger();
 #endif
 
-  while (true) {
-    std::unique_ptr<Batch> batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
-    AMDINFER_LOG_DEBUG(logger, "Got request in TfZendnn. Size: " +
-                                 std::to_string(batch->size()));
+  util::Timer timer{true};
 
-    std::vector<InferenceResponse> responses;
-    responses.reserve(batch->size());
+  // Find total number of images to initialize for TF tensor
+  auto tensor_count = static_cast<int>(batch->size());
 
-#ifdef AMDINFER_ENABLE_METRICS
-    Metrics::getInstance().incrementCounter(
-      MetricCounterIDs::PipelineIngressWorker);
-#endif
+  // Initialize a TensorFlow tensor with required shape
+  tf::Tensor input_tensor(
+    tf::DT_FLOAT, tf::TensorShape({tensor_count, image_height_, image_width_,
+                                   image_channels_}));
 
-    util::Timer timer{true};
+  uint64_t input_size = image_height_ * image_width_ * image_channels_;
+  size_t vec_size = 0;
 
-    // Find total number of images to initialize for TF tensor
-    auto tensor_count = static_cast<int>(batch->size());
+  for (unsigned int j = 0; j < batch->size(); j++) {
+    const auto& req = batch->getRequest(j);
 
-    // Initialize a TensorFlow tensor with required shape
-    tf::Tensor input_tensor(
-      tf::DT_FLOAT, tf::TensorShape({tensor_count, image_height_, image_width_,
-                                     image_channels_}));
+    auto inputs = req->getInputs();
+    auto outputs = req->getOutputs();
+    AMDINFER_LOG_DEBUG(logger,
+                       "Size of input: " + std::to_string(inputs.size()));
 
-    uint64_t input_size = image_height_ * image_width_ * image_channels_;
-    size_t vec_size = 0;
+    // Get all the inputs from the requests and copy to the TensorFlow tensor
+    for (auto& input : inputs) {
+      auto* input_buffer = input.getData();
+      const auto* float_buffer = static_cast<float*>(input_buffer);
+      std::copy(float_buffer, float_buffer + input_size,
+                input_tensor.flat<float>().data() + vec_size);
+      vec_size = vec_size + input_size;
+    }
+  }
 
-    for (unsigned int j = 0; j < batch->size(); j++) {
-      const auto& req = batch->getRequest(j);
+  AMDINFER_LOG_DEBUG(logger, input_tensor.DebugString());
 
-#ifdef AMDINFER_ENABLE_TRACING
-      const auto& trace = batch->getTrace(j);
-      trace->startSpan("tfzendnn");
-#endif
-      auto& resp = responses.emplace_back();
-      resp.setID(req->getID());
-      resp.setModel("TFModel");
-
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-      AMDINFER_LOG_DEBUG(logger,
-                         "Size of input: " + std::to_string(inputs.size()));
-
-      // Get all the inputs from the requests and copy to the TensorFlow tensor
-      for (auto& input : inputs) {
-        auto* input_buffer = input.getData();
-        const auto* float_buffer = static_cast<float*>(input_buffer);
-        std::copy(float_buffer, float_buffer + input_size,
-                  input_tensor.flat<float>().data() + vec_size);
-        vec_size = vec_size + input_size;
-      }
-    }
+  // Create the inputs and output tensor
+  std::vector<std::pair<std::string, tf::Tensor>> input_pair = {
+    {input_node_, input_tensor}};
+  std::vector<tensorflow::Tensor> output_tensor;
 
-    AMDINFER_LOG_DEBUG(logger, input_tensor.DebugString());
-
-    // Create the inputs and output tensor
-    std::vector<std::pair<std::string, tf::Tensor>> input_pair = {
-      {input_node_, input_tensor}};
-    std::vector<tensorflow::Tensor> output_tensor;
-
-    // Run the session to get the predictions
-    timer.add("infer_start");
-    status_ =
-      this->session_->Run(input_pair, {output_node_}, {}, &output_tensor);
-    timer.add("infer_stop");
-    [[maybe_unused]] auto duration =
-      timer.count<std::milli>("infer_start", "infer_stop");
-    AMDINFER_LOG_INFO(logger, "Time taken for " + std::to_string(tensor_count) +
-                                " images: " + std::to_string(duration));
-
-    if (!status_.ok()) {
-      AMDINFER_LOG_ERROR(logger, status_.ToString());
-      for (const auto& req : *batch) {
-        req->runCallbackError("Issue with prediction");
-      }
+  // Run the session to get the predictions
+  timer.add("infer_start");
+  status_ = this->session_->Run(input_pair, {output_node_}, {}, &output_tensor);
+  timer.add("infer_stop");
+  [[maybe_unused]] auto duration =
+    timer.count<std::milli>("infer_start", "infer_stop");
+  AMDINFER_LOG_INFO(logger, "Time taken for " + std::to_string(tensor_count) +
+                              " images: " + std::to_string(duration));
+
+  if (!status_.ok()) {
+    AMDINFER_LOG_ERROR(logger, status_.ToString());
+    for (const auto& req : *batch) {
+      req->runCallbackError("Issue with prediction");
     }
-    AMDINFER_LOG_DEBUG(logger, output_tensor[0].DebugString());
-
-    // Copy the output from the model to the response object
-    size_t response_size = output_classes_;
-    std::vector<size_t> new_shape = {response_size};
-    for (unsigned int k = 0; k < batch->size(); k++) {
-      const auto& req = batch->getRequest(k);
-      auto inputs = req->getInputs();
-      auto outputs = req->getOutputs();
-      auto& resp = responses[k];
-
-      for (unsigned int i = 0; i < inputs.size(); i++) {
-        InferenceResponseOutput output;
-        std::vector<std::byte> buffer;
-        buffer.resize(response_size * sizeof(float));
-
-        memcpy(buffer.data(),
-               output_tensor[0].flat<float>().data() + (i * response_size),
+  }
+  AMDINFER_LOG_DEBUG(logger, output_tensor[0].DebugString());
+
+  // Copy the output from the model to the response object
+  size_t response_size = output_classes_;
+  std::vector<size_t> new_shape = {response_size};
+
+  auto new_batch = batch->propagate();
+  std::vector<BufferPtr> input_buffers;
+  input_buffers.push_back(pool->get(next_allocators_,
+                                    Tensor{"name", new_shape, DataType::Fp32},
+                                    batch->size()));
+
+  for (unsigned int k = 0; k < batch->size(); k++) {
+    const auto& req = batch->getRequest(k);
+
+    auto new_request = req->propagate();
+    auto* data_ptr =
+      input_buffers.at(0)->data(k * response_size * DataType("Fp32").size());
+    new_request->addInputTensor(
+      InferenceRequestInput{data_ptr, new_shape, DataType::Fp32, ""});
+    util::copy(output_tensor[0].flat<float>().data() + (k * response_size),
+               static_cast<std::byte*>(data_ptr),
                response_size * sizeof(float));
-        output.setData(std::move(buffer));
-
-        std::string output_name;
-        if (i < outputs.size()) {
-          output_name = outputs[i].getName();
-        }
-
-        if (output_name.empty()) {
-          output.setName(inputs[0].getName());
-        } else {
-          output.setName(output_name);
-        }
-
-        output.setShape(new_shape);
-        output.setDatatype(DataType::Fp32);
-        resp.addOutput(output);
-      }
-
-#ifdef AMDINFER_ENABLE_TRACING
-      auto context = batch->getTrace(k)->propagate();
-      resp.setContext(std::move(context));
-#endif
 
-      timer.stop();
-      duration = timer.count<std::milli>();
-      AMDINFER_LOG_DEBUG(logger,
-                         "Total time taken: " + std::to_string(duration));
+    new_batch->addRequest(new_request);
 
-      req->runCallbackOnce(resp);
-
-#ifdef AMDINFER_ENABLE_METRICS
-      timer.add("metrics", batch->getTime(k));
-      duration = timer.count<std::micro>("metrics", "stop");
-      Metrics::getInstance().observeSummary(MetricSummaryIDs::RequestLatency,
-                                            duration);
-#endif
-    }
-    this->returnInputBuffers(std::move(batch));
+    new_batch->setModel(k, "TFModel");
   }
-  AMDINFER_LOG_INFO(logger, "TfZendnn ending");
+  new_batch->setBuffers(std::move(input_buffers), {});
+
+  return new_batch;
 }
 
 void TfZendnn::doRelease() {
@@ -392,6 +332,6 @@ amdinfer::workers::Worker* getWorker() {
   // symbols before the global scope.
   openLibrary("libtensorflow_cc.so", RTLD_GLOBAL | RTLD_LAZY | RTLD_DEEPBIND);
 
-  return new amdinfer::workers::TfZendnn("TfZendnn", "cpu");
+  return new amdinfer::workers::TfZendnn("TfZendnn", "CPU", true);
 }
 }  // extern C
diff --git a/src/amdinfer/workers/worker.hpp b/src/amdinfer/workers/worker.hpp
index 8c01f5c1e..4280faad7 100644
--- a/src/amdinfer/workers/worker.hpp
+++ b/src/amdinfer/workers/worker.hpp
@@ -21,6 +21,7 @@
 #ifndef GUARD_AMDINFER_WORKERS_WORKER
 #define GUARD_AMDINFER_WORKERS_WORKER
 
+#include <cassert>
 #include <limits>
 #include <memory>
 #include <string>
@@ -34,6 +35,9 @@
 #include "amdinfer/core/memory_pool/pool.hpp"
 #include "amdinfer/core/model_metadata.hpp"
 #include "amdinfer/observation/logging.hpp"
+#include "amdinfer/observation/metrics.hpp"
+#include "amdinfer/util/ctpl.hpp"    // for ThreadPool
+#include "amdinfer/util/thread.hpp"  // for setThreadName
 
 namespace amdinfer {
 
@@ -59,21 +63,13 @@ enum class WorkerStatus {
 class Worker {
  public:
   /// The constructor only initializes the private class members
-  Worker(const std::string& name, const std::string& platform)
-    : metadata_(name, platform) {
+  Worker(const std::string& name, const std::string& platform, bool allow_next)
+    : metadata_(name, platform), allow_next_(allow_next) {
     this->status_ = WorkerStatus::New;
   }
   virtual ~Worker() = default;  ///< Destroy the Worker object
 
-  /**
-   * @brief Starts the worker's run() method as a separate thread and returns it
-   *
-   * @param input_queue queue used to send requests to the started worker thread
-   * @return std::thread
-   */
-  virtual std::thread spawn(BatchPtrQueue* input_queue) = 0;
-
-  /// Allocate some buffers that are used to hold input and output data
+  /// Get the memory allocators supported by this worker
   [[nodiscard]] virtual std::vector<MemoryAllocators> getAllocators() const = 0;
 
   /// Perform low-cost initialization of the worker
@@ -92,15 +88,11 @@ class Worker {
    *
    * @param input_queue queue that receives incoming requests
    */
-  void run(BatchPtrQueue* input_queue) {
-    this->status_ = WorkerStatus::Run;
-    this->doRun(input_queue);
-    this->status_ = WorkerStatus::Inactive;
-  }
+  virtual void run(BatchPtrQueue* input_queue, const MemoryPool* pool) = 0;
   /// Release any hardware resources
   void release() {
-    this->status_ = WorkerStatus::Release;
-    this->metadata_.setReady(false);
+    status_ = WorkerStatus::Release;
+    metadata_.setReady(false);
     this->doRelease();
   }
   /// Perform any final operations before the worker thread is joined
@@ -110,8 +102,6 @@ class Worker {
     this->status_ = WorkerStatus::Dead;
   }
 
-  void setPool(MemoryPool* pool) { pool_ = pool; }
-
   [[nodiscard]] size_t getBatchSize() const { return this->batch_size_; }
   [[nodiscard]] WorkerStatus getStatus() const { return this->status_; }
 
@@ -135,33 +125,41 @@ class Worker {
 
   ModelMetadata getMetadata() const { return this->metadata_; }
 
+  const std::string& getName() const { return metadata_.getName(); }
+
+  void setNext(BatchPtrQueue* batcher) {
+    if (allow_next_) {
+      next_ = batcher;
+    }
+  }
+  void setNextAllocators(const std::vector<MemoryAllocators>& allocators) {
+    next_allocators_ = allocators;
+  }
+
  protected:
 #ifdef AMDINFER_ENABLE_LOGGING
   [[nodiscard]] const Logger& getLogger() const { return logger_; };
 #endif
 
-  void returnInputBuffers(std::unique_ptr<Batch> batch) {
-    auto buffers = batch->getInputBuffers();
-    for (auto& buffer : buffers) {
-      pool_->put(std::move(buffer));
-    }
-  }
-
   size_t batch_size_ = 1;
   ModelMetadata metadata_;
-  MemoryPool* pool_;
+  std::vector<MemoryAllocators> next_allocators_;
+  BatchPtrQueue* next_ = nullptr;
+  WorkerStatus status_;
 
- private:
-  /// Perform low-cost initialization of the worker
-  virtual void doInit(ParameterMap* parameters) = 0;
-  /// Acquire any hardware resources or perform high-cost initialization
-  virtual void doAcquire(ParameterMap* parameters) = 0;
   /**
    * @brief The main body of the worker executes the work
    *
    * @param input_queue queue that receives incoming requests
    */
-  virtual void doRun(BatchPtrQueue* input_queue) = 0;
+  virtual std::unique_ptr<Batch> doRun(Batch* batch,
+                                       const MemoryPool* pool) = 0;
+
+ private:
+  /// Perform low-cost initialization of the worker
+  virtual void doInit(ParameterMap* parameters) = 0;
+  /// Acquire any hardware resources or perform high-cost initialization
+  virtual void doAcquire(ParameterMap* parameters) = 0;
   /// Release any hardware resources
   virtual void doRelease() = 0;
   /// Perform any final operations before the worker's run thread is joined
@@ -170,8 +168,170 @@ class Worker {
 #ifdef AMDINFER_ENABLE_LOGGING
   Logger logger_{Loggers::Server};
 #endif
+  bool allow_next_ = true;
+};
 
-  WorkerStatus status_;
+class SingleThreadedWorker : public Worker {
+ public:
+  using Worker::Worker;
+  /**
+   * @brief The main body of the worker executes the work
+   *
+   * @param input_queue queue that receives incoming requests
+   */
+  void run(BatchPtrQueue* input_queue, const MemoryPool* pool) override {
+    this->status_ = WorkerStatus::Run;
+    const auto& name = this->getName();
+    const auto logger = this->getLogger();
+    util::setThreadName(name);
+
+    while (true) {
+      BatchPtr batch;
+      input_queue->wait_dequeue(batch);
+      if (batch == nullptr) {
+        break;
+      }
+
+      [[maybe_unused]] auto batch_size = batch->size();
+
+#ifdef AMDINFER_ENABLE_TRACING
+      for (auto i = 0U; i < batch_size; ++i) {
+        const auto& trace = batch->getTrace(i);
+        trace->startSpan(name.c_str());
+      }
+#endif
+
+      AMDINFER_LOG_INFO(logger, "Got request in " + name);
+#ifdef AMDINFER_ENABLE_METRICS
+      Metrics::getInstance().incrementCounter(
+        MetricCounterIDs::PipelineIngressWorker);
+#endif
+
+      auto new_batch = this->doRun(batch.get(), pool);
+
+      if (next_ != nullptr) {
+        assert(new_batch != nullptr);
+        assert(new_batch->size() == batch_size);
+#ifdef AMDINFER_ENABLE_TRACING
+        for (auto i = 0U; i < batch_size; ++i) {
+          auto& trace = batch->getTrace(i);
+          trace->endSpan();
+          new_batch->addTrace(std::move(trace));
+        }
+#endif
+        next_->enqueue(std::move(new_batch));
+      }
+
+      const auto& buffers = batch->getInputBuffers();
+      for (const auto& buffer : buffers) {
+        buffer->free();
+      }
+    }
+
+    AMDINFER_LOG_INFO(logger, name + " ending");
+
+    status_ = WorkerStatus::Inactive;
+  }
+
+ private:
+  using Worker::next_;
+  using Worker::status_;
+};
+
+class MultiThreadedWorker : public Worker {
+ public:
+  using Worker::Worker;
+  /**
+   * @brief The main body of the worker executes the work
+   *
+   * @param input_queue queue that receives incoming requests
+   */
+  void run(BatchPtrQueue* input_queue, const MemoryPool* pool) override {
+    this->status_ = WorkerStatus::Run;
+    const auto& name = this->getName();
+    const auto logger = this->getLogger();
+    util::setThreadName(name);
+    std::atomic_int32_t outstanding_batches = 0;
+    // 4 is arbitrary
+    const int max_outstanding_batches = thread_pool_.getSize() * 4;
+    // 10 is arbitrary
+    const auto thread_pool_delay = std::chrono::milliseconds(10);
+
+    while (true) {
+      BatchPtr batch;
+      input_queue->wait_dequeue(batch);
+      if (batch == nullptr) {
+        break;
+      }
+
+      [[maybe_unused]] auto batch_size = batch->size();
+
+#ifdef AMDINFER_ENABLE_TRACING
+      for (auto i = 0U; i < batch_size; ++i) {
+        const auto& trace = batch->getTrace(i);
+        trace->startSpan(name.c_str());
+      }
+#endif
+
+      AMDINFER_LOG_INFO(logger, "Got request in " + name);
+#ifdef AMDINFER_ENABLE_METRICS
+      Metrics::getInstance().incrementCounter(
+        MetricCounterIDs::PipelineIngressWorker);
+#endif
+
+      outstanding_batches++;
+      if (outstanding_batches > max_outstanding_batches) {
+        std::this_thread::sleep_for(thread_pool_delay);
+      }
+
+      thread_pool_.push([this, batch = std::move(batch), &outstanding_batches,
+                         pool]([[maybe_unused]] int id) {
+        [[maybe_unused]] auto batch_size = batch->size();
+        auto new_batch = this->doRun(batch.get(), pool);
+
+        if (next_ != nullptr) {
+          assert(new_batch != nullptr);
+          assert(new_batch->size() == batch_size);
+#ifdef AMDINFER_ENABLE_TRACING
+          for (auto i = 0U; i < batch_size; ++i) {
+            auto& trace = batch->getTrace(i);
+            trace->endSpan();
+            new_batch->addTrace(std::move(trace));
+          }
+#endif
+
+#ifdef AMDINFER_ENABLE_METRICS
+          for (auto i = 0U; i < batch_size; ++i) {
+            new_batch->addTime(batch->getTime(i));
+          }
+#endif
+
+          next_->enqueue(std::move(new_batch));
+        }
+
+        const auto& buffers = batch->getInputBuffers();
+        for (const auto& buffer : buffers) {
+          buffer->free();
+        }
+
+        outstanding_batches--;
+      });
+    }
+
+    AMDINFER_LOG_INFO(logger, name + " ending");
+
+    status_ = WorkerStatus::Inactive;
+  }
+
+ protected:
+  void createThreadPool(int threads) { thread_pool_.resize(threads); }
+
+  void destroyThreadPool() { thread_pool_.stop(true); }
+
+ private:
+  using Worker::next_;
+  using Worker::status_;
+  util::ThreadPool thread_pool_;
 };
 
 }  // namespace workers
diff --git a/src/amdinfer/workers/xmodel.cpp b/src/amdinfer/workers/xmodel.cpp
index 0e61b547a..79650d47e 100644
--- a/src/amdinfer/workers/xmodel.cpp
+++ b/src/amdinfer/workers/xmodel.cpp
@@ -55,7 +55,7 @@
 #include "amdinfer/declarations.hpp"              // for BufferPtrs, Infere...
 #include "amdinfer/observation/observer.hpp"      // for Loggers, Metrics...
 #include "amdinfer/util/containers.hpp"           // for containerProduct
-#include "amdinfer/util/ctpl.hpp"                 // for ThreadPool
+#include "amdinfer/util/memory.hpp"               // for copy
 #include "amdinfer/util/parse_env.hpp"            // for autoExpandEnvironm...
 #include "amdinfer/util/queue.hpp"                // for BufferPtrsQueue
 #include "amdinfer/util/thread.hpp"               // for setThreadName
@@ -70,16 +70,15 @@ namespace amdinfer::workers {
  * to the FPGA using the Vitis AI runtime libraries.
  *
  */
-class XModel : public Worker {
+class XModel : public MultiThreadedWorker {
  public:
-  XModel() : Worker("XModel", "XModel") {}
-  std::thread spawn(BatchPtrQueue* input_queue) override;
+  using MultiThreadedWorker::MultiThreadedWorker;
   [[nodiscard]] std::vector<MemoryAllocators> getAllocators() const override;
 
  private:
   void doInit(ParameterMap* parameters) override;
   void doAcquire(ParameterMap* parameters) override;
-  void doRun(BatchPtrQueue* input_queue) override;
+  BatchPtr doRun(Batch* batch, const MemoryPool* pool) override;
   void doRelease() override;
   void doDestroy() override;
 
@@ -91,13 +90,8 @@ class XModel : public Worker {
   std::unique_ptr<vart::Runner> runner_;
   std::vector<DataType> output_type_;
   std::vector<uint32_t> output_size_;
-  util::ThreadPool thread_pool_;
 };
 
-std::thread XModel::spawn(BatchPtrQueue* input_queue) {
-  return std::thread(&XModel::run, this, input_queue);
-}
-
 std::vector<MemoryAllocators> XModel::getAllocators() const {
   return {MemoryAllocators::VartTensor};
 }
@@ -154,7 +148,7 @@ void XModel::doAcquire(ParameterMap* parameters) {
   if (parameters->has("threads")) {
     threads = parameters->get<int32_t>("threads");
   }
-  this->thread_pool_.resize(threads);
+  this->createThreadPool(threads);
 
   runner_ = vart::Runner::create_runner(this->subgraph_, "run");
   auto input_tensors = runner_->get_input_tensors();
@@ -177,183 +171,106 @@ void XModel::doAcquire(ParameterMap* parameters) {
   }
 }
 
-void XModel::doRun(BatchPtrQueue* input_queue) {
-  std::atomic_int32_t thread_pool_size = 0;
-  const int max_thread_pool_size =
-    this->thread_pool_.getSize() * 4;  // 4 is arbitrary
-  util::setThreadName("XModel");
+BatchPtr XModel::doRun(Batch* batch, const MemoryPool* pool) {
+  const auto& input_buffers = batch->getInputBuffers();
+  // const auto& output_buffers = batch->getOutputBuffers();
+
 #ifdef AMDINFER_ENABLE_LOGGING
-  const auto& logger = this->getLogger();
-#endif
+  for (const auto& buffer : input_buffers) {
+    logTraceBuffer(getLogger(), buffer->data(0));
+  }
+#endif  // AMDINFER_ENABLE_LOGGING
+
+  std::vector<vart::TensorBuffer*> inputs_ptr;
+  for (const auto& buffer : input_buffers) {
+    auto* vart = dynamic_cast<VartTensorBuffer*>(buffer.get());
+    inputs_ptr.emplace_back(vart->getTensorBuffer());
+  }
+
+  BufferPtrs output_buffers;
+  BufferPtrs new_input_buffers;
+  auto output_tensors = this->getRunner()->get_output_tensors();
+  for (const auto* tensor : output_tensors) {
+    auto xir_shape = tensor->get_shape();
+    std::vector<size_t> shape{xir_shape.begin(), xir_shape.end()};
+    auto xir_type = tensor->get_data_type();
+    auto type = mapXirToType(xir_type);
+    InferenceRequestInput input(nullptr, shape, type, tensor->get_name());
+    // the shape includes the batch size so use external batch size 1
+    output_buffers.push_back(
+      pool->get({MemoryAllocators::VartTensor}, input, 1));
+    new_input_buffers.push_back(pool->get(next_allocators_, input, 1));
+  }
 
-  const auto thread_pool_delay = std::chrono::milliseconds(10);
+  std::vector<vart::TensorBuffer*> outputs_ptr;
+  for (const auto& buffer : output_buffers) {
+    auto* vart = dynamic_cast<VartTensorBuffer*>(buffer.get());
+    outputs_ptr.emplace_back(vart->getTensorBuffer());
+  }
 
-  while (true) {
-    BatchPtr batch;
-    input_queue->wait_dequeue(batch);
-    if (batch == nullptr) {
-      break;
-    }
-    AMDINFER_LOG_INFO(
-      logger, "Got request in xmodel: " + std::to_string(batch->size()));
-#ifdef AMDINFER_ENABLE_METRICS
-    Metrics::getInstance().incrementCounter(
-      MetricCounterIDs::PipelineIngressWorker);
-#endif
-    thread_pool_size++;
-    if (thread_pool_size > max_thread_pool_size) {
-      std::this_thread::sleep_for(thread_pool_delay);
+  // auto outputs_ptr = this->getRunner()->get_outputs();
+
+  for (auto* input : inputs_ptr) {
+    const auto* tensor = input->get_tensor();
+    auto num = tensor->get_element_num();
+    auto batches = (tensor->get_shape())[0];
+    input->sync_for_write(0, num / batches);
+  }
+
+  std::queue<std::pair<uint32_t, int>> futures;
+  futures.push(getRunner()->execute_async(inputs_ptr, outputs_ptr));
+
+  while (!futures.empty()) {
+    auto job_id = futures.front();
+    futures.pop();
+    getRunner()->wait(job_id.first, -1);
+  }
+
+  for (auto* output : outputs_ptr) {
+    const auto* tensor = output->get_tensor();
+    output->sync_for_read(0,
+                          tensor->get_element_num() / (tensor->get_shape())[0]);
+  }
+
+  const auto num_batches = batch->size();
+  auto new_batch = batch->propagate();
+  for (unsigned int k = 0; k < num_batches; k++) {
+    const auto& req = batch->getRequest(k);
+
+    auto new_request = req->propagate();
+
+    const auto num_outputs = outputs_ptr.size();
+    for (unsigned int i = 0; i < num_outputs; i++) {
+      auto output_shape = output_tensors[i]->get_shape();
+      std::vector<uint64_t> new_shape;
+      new_shape.reserve(output_shape.size() - 1);
+      for (auto j = 1U; j < output_shape.size(); j++) {
+        new_shape.push_back(output_shape[j]);
+      }
+      auto* output_index =
+        reinterpret_cast<void*>(outputs_ptr.at(i)->data().first);
+
+      auto* data_ptr = new_input_buffers.at(i)->data(k * output_size_[i] *
+                                                     (output_type_[i]).size());
+      new_request->addInputTensor(
+        InferenceRequestInput{data_ptr, new_shape, output_type_[i], ""});
+      util::copy(
+        reinterpret_cast<int8_t*>(output_index) + (i * output_size_[i]),
+        static_cast<std::byte*>(data_ptr),
+        output_size_[i] * output_type_[i].size());
     }
-    this->thread_pool_.push(
-      [this, batch = std::move(batch), &thread_pool_size](int id) {
-        (void)id;  // suppress unused variable warning
-#ifdef AMDINFER_ENABLE_TRACING
-        for (unsigned int j = 0; j < batch->size(); j++) {
-          auto& trace = batch->getTrace(j);
-          trace->startSpan("xmodel");
-        }
-#endif
-
-        auto input_buffers = batch->getInputBuffers();
-      // const auto& output_buffers = batch->getOutputBuffers();
 
-#ifdef AMDINFER_ENABLE_LOGGING
-        for (const auto& buffer : input_buffers) {
-          logTraceBuffer(getLogger(), buffer->data(0));
-        }
-#endif  // AMDINFER_ENABLE_LOGGING
+    new_batch->addRequest(new_request);
 
-        std::vector<vart::TensorBuffer*> inputs_ptr;
-        for (const auto& buffer : input_buffers) {
-          auto* vart = dynamic_cast<VartTensorBuffer*>(buffer.get());
-          inputs_ptr.emplace_back(vart->getTensorBuffer());
-        }
-
-        BufferPtrs output_buffers;
-        auto output_tensors = this->getRunner()->get_output_tensors();
-        for (const auto* tensor : output_tensors) {
-          auto xir_shape = tensor->get_shape();
-          std::vector<size_t> shape{xir_shape.begin(), xir_shape.end()};
-          auto xir_type = tensor->get_data_type();
-          auto type = mapXirToType(xir_type);
-          InferenceRequestInput input(nullptr, shape, type, tensor->get_name());
-          // the shape includes the batch size so use external batch size 1
-          output_buffers.push_back(
-            pool_->get({MemoryAllocators::VartTensor}, input, 1));
-        }
-
-        std::vector<vart::TensorBuffer*> outputs_ptr;
-        for (const auto& buffer : output_buffers) {
-          auto* vart = dynamic_cast<VartTensorBuffer*>(buffer.get());
-          outputs_ptr.emplace_back(vart->getTensorBuffer());
-        }
-
-        // auto outputs_ptr = this->getRunner()->get_outputs();
-
-        for (auto* input : inputs_ptr) {
-          const auto* tensor = input->get_tensor();
-          auto num = tensor->get_element_num();
-          auto batches = (tensor->get_shape())[0];
-          input->sync_for_write(0, num / batches);
-        }
-
-        std::queue<std::pair<uint32_t, int>> futures;
-        futures.push(getRunner()->execute_async(inputs_ptr, outputs_ptr));
-
-        std::vector<InferenceResponse> responses;
-        responses.reserve(batch->size());
-
-        for (const auto& req : *batch) {
-          auto& resp = responses.emplace_back();
-          resp.setID(req->getID());
-          resp.setModel("xmodel");
-        }
-
-        while (!futures.empty()) {
-          auto job_id = futures.front();
-          futures.pop();
-          getRunner()->wait(job_id.first, -1);
-        }
-
-        for (auto* output : outputs_ptr) {
-          const auto* tensor = output->get_tensor();
-          output->sync_for_read(
-            0, tensor->get_element_num() / (tensor->get_shape())[0]);
-        }
-
-        const auto num_batches = batch->size();
-        for (unsigned int k = 0; k < num_batches; k++) {
-          const auto& req = batch->getRequest(k);
-          auto inputs = req->getInputs();
-          auto outputs = req->getOutputs();
-          auto& resp = responses[k];
-
-          const auto num_outputs = outputs_ptr.size();
-          for (unsigned int i = 0; i < num_outputs; i++) {
-            // auto* output_index = output_buffers[i]->data(0);
-            auto* output_index =
-              reinterpret_cast<void*>(outputs_ptr.at(i)->data().first);
-            InferenceResponseOutput output;
-            auto output_tensors = getRunner()->get_output_tensors();
-            auto output_shape = output_tensors[i]->get_shape();
-            std::vector<uint64_t> new_shape;
-            new_shape.reserve(output_shape.size() - 1);
-            for (size_t j = 0; j < output_shape.size() - 1; j++) {
-              new_shape.push_back(output_shape[j + 1]);
-            }
-            output.setShape(new_shape);
-
-            output.setDatatype(this->output_type_[i]);
-
-            std::vector<std::byte> buffer;
-            buffer.resize(this->output_size_[i] * sizeof(uint8_t));
-            memcpy(buffer.data(),
-                   reinterpret_cast<int8_t*>(output_index) +
-                     (i * this->output_size_[i]),
-                   this->output_size_[i] * sizeof(uint8_t));
-            output.setData(std::move(buffer));
-
-            std::string output_name;
-            if (i < outputs.size()) {
-              output_name = outputs[i].getName();
-            }
-
-            if (output_name.empty()) {
-              output.setName(inputs[0].getName());
-            } else {
-              output.setName(output_name);
-            }
-
-            resp.addOutput(output);
-          }
-
-#ifdef AMDINFER_ENABLE_TRACING
-          auto context = batch->getTrace(k)->propagate();
-          resp.setContext(std::move(context));
-#endif
-
-          req->runCallbackOnce(resp);
-#ifdef AMDINFER_ENABLE_METRICS
-          Metrics::getInstance().incrementCounter(
-            MetricCounterIDs::PipelineEgressWorker);
-          util::Timer timer{batch->getTime(k)};
-          timer.stop();
-          auto duration = timer.count<std::micro>();
-          Metrics::getInstance().observeSummary(
-            MetricSummaryIDs::RequestLatency, duration);
-#endif
-        }
-        for (auto& buffer : input_buffers) {
-          pool_->put(std::move(buffer));
-        }
-        thread_pool_size--;
-      });
+    new_batch->setModel(k, "xmodel");
   }
-  AMDINFER_LOG_INFO(logger, "XModel ending");
+  new_batch->setBuffers(std::move(new_input_buffers), {});
+
+  return new_batch;
 }
 
-void XModel::doRelease() {}
-void XModel::doDestroy() { this->thread_pool_.stop(true); }
+void XModel::doRelease() { this->destroyThreadPool(); }
+void XModel::doDestroy() {}
 
 }  // namespace amdinfer::workers
 
@@ -361,6 +278,6 @@ extern "C" {
 // using smart pointer here may cause problems inside shared object so managing
 // manually
 amdinfer::workers::Worker* getWorker() {
-  return new amdinfer::workers::XModel();
+  return new amdinfer::workers::XModel("XModel", "FPGA", true);
 }
 }  // extern C
diff --git a/tools/benchmark.py b/tools/benchmark.py
index e9c30fde3..9f23eda84 100644
--- a/tools/benchmark.py
+++ b/tools/benchmark.py
@@ -1041,7 +1041,7 @@ def pytest_benchmarks(config: Config, quiet=False):
         # if pytest is disabled, run anyway with a small test suite to create a
         # new benchmark file that can be populated by later benchmarks
         tmp = config.benchmarks
-        config.benchmarks = "echo"
+        config.benchmarks = "cplusplus"
         pytest_benchmarks(config, True)
         config.benchmarks = tmp
         benchmarks = get_benchmark(normalize=True)


Test patch:
diff --git a/docker/docker-compose.autotest.yml b/docker/docker-compose.autotest.yml
index 0efe41c2f..439b40c87 100644
--- a/docker/docker-compose.autotest.yml
+++ b/docker/docker-compose.autotest.yml
@@ -16,6 +16,8 @@
 version: '3.7'
 services:
   amdinfer-autotest:
+    # don't load repository on startup
+    command: amdinfer-server
     devices: []
     extra_hosts:
       - "amdinfer-autotest:127.0.1.1"
@@ -27,6 +29,7 @@ services:
       # for test artifacts
       - $PWD/external:/workspace/amdinfer/external:ro
       - $PWD/external/artifacts:/opt/xilinx/amdinfer/artifacts:ro
+      - $PWD/external/artifacts/repository:/mnt/models:ro
       - $PWD/tests/assets:/workspace/amdinfer/tests/assets:ro
       - /opt/xilinx/overlaybins:/opt/xilinx/overlaybins:ro
     working_dir: /home/amdinfer-user
diff --git a/tests/api/test_infer_async.cpp b/tests/api/test_infer_async.cpp
index e4db1fdee..5b10957cf 100644
--- a/tests/api/test_infer_async.cpp
+++ b/tests/api/test_infer_async.cpp
@@ -28,8 +28,9 @@ namespace amdinfer {
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(HttpFixture, Ordered) {
   NativeClient client(&server_);
-  auto endpoint = client.workerLoad("echo", {});
-  EXPECT_EQ(endpoint, "echo");
+  auto endpoint =
+    client.workerLoad("cplusplus", {{"model"}, {std::string{"echo"}}});
+  EXPECT_EQ(endpoint, "cplusplus");
 
   std::vector<uint32_t> img_data;
   const auto shape = {1UL};
@@ -47,11 +48,13 @@ TEST_F(HttpFixture, Ordered) {
     reqs.push_back(request);
   }
 
-  auto resps = inferAsyncOrdered(&client, "echo", reqs);
+  auto resps = inferAsyncOrdered(&client, endpoint, reqs);
   EXPECT_EQ(resps.size(), data_size);
   for (const auto& resp : resps) {
     EXPECT_FALSE(resp.isError());
   }
+
+  client.workerUnload(endpoint);
 }
 #endif
 
diff --git a/tests/api/test_model_infer.cpp b/tests/api/test_model_infer.cpp
index 58d9a21d9..1917022a9 100644
--- a/tests/api/test_model_infer.cpp
+++ b/tests/api/test_model_infer.cpp
@@ -20,9 +20,12 @@
 #include "amdinfer/amdinfer.hpp"                // for InferenceResponse, Grp...
 #include "amdinfer/testing/gtest_fixtures.hpp"  // for GrpcFixture
 
-void test(amdinfer::Client* client) {
-  auto endpoint = client->workerLoad("echo", {});
-  EXPECT_EQ(endpoint, "echo");
+namespace amdinfer {
+
+void test(const Client* client) {
+  auto endpoint =
+    client->workerLoad("cplusplus", {{"model"}, {std::string{"echo"}}});
+  EXPECT_EQ(endpoint, "cplusplus");
 
   std::vector<uint32_t> img_data;
   auto shape = {1UL};
@@ -30,9 +33,9 @@ void test(amdinfer::Client* client) {
   img_data.reserve(size);
   img_data.push_back(1);
 
-  amdinfer::InferenceRequest request;
+  InferenceRequest request;
   request.addInputTensor(static_cast<void*>(img_data.data()), shape,
-                         amdinfer::DataType::Uint32);
+                         DataType::Uint32);
 
   auto response = client->modelInfer(endpoint, request);
 
@@ -42,12 +45,12 @@ void test(amdinfer::Client* client) {
 
   auto outputs = response.getOutputs();
   EXPECT_EQ(outputs.size(), 1);
-  for (auto& output : outputs) {
+  for (const auto& output : outputs) {
     const auto* data = static_cast<uint32_t*>(output.getData());
     EXPECT_EQ(data[0], 2);
   }
 
-  client->modelUnload(endpoint);
+  client->workerUnload(endpoint);
 }
 
 #ifdef AMDINFER_ENABLE_GRPC
@@ -57,7 +60,7 @@ TEST_F(GrpcFixture, ModelInfer) { test(client_.get()); }
 
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(BaseFixture, ModelInfer) {
-  amdinfer::NativeClient client(&server_);
+  NativeClient client(&server_);
   test(&client);
 }
 
@@ -65,3 +68,5 @@ TEST_F(BaseFixture, ModelInfer) {
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(HttpFixture, ModelInfer) { test(client_.get()); }
 #endif
+
+}  // namespace amdinfer
diff --git a/tests/api/test_model_infer.py b/tests/api/test_model_infer.py
index de1aac233..d868f44ae 100644
--- a/tests/api/test_model_infer.py
+++ b/tests/api/test_model_infer.py
@@ -32,8 +32,9 @@ def test_model_infer(self):
         Test making an inference
         """
 
-        endpoint = self.rest_client.workerLoad("echo")
-        assert endpoint == "echo"
+        parameters = amdinfer.ParameterMap(["model"], ["echo"])
+        endpoint = self.rest_client.workerLoad("cplusplus", parameters)
+        assert endpoint == "cplusplus"
 
         input_data = amdinfer.InferenceRequestInput()
         input_data.shape = [1]
@@ -51,10 +52,6 @@ def test_model_infer(self):
         assert len(output_data) == 1
         assert output_data[0] == 2
 
-        self.rest_client.modelUnload(endpoint)
+        self.rest_client.workerUnload(endpoint)
 
-        try:
-            while self.rest_client.modelReady(endpoint):
-                time.sleep(1)
-        except amdinfer.RuntimeError:
-            pass
+        amdinfer.waitUntilModelNotReady(self.rest_client, endpoint)
diff --git a/tests/api/test_model_infer_async.cpp b/tests/api/test_model_infer_async.cpp
index 10a71099d..df4204290 100644
--- a/tests/api/test_model_infer_async.cpp
+++ b/tests/api/test_model_infer_async.cpp
@@ -22,9 +22,12 @@
 #include "amdinfer/amdinfer.hpp"                // for InferenceResponse, Grp...
 #include "amdinfer/testing/gtest_fixtures.hpp"  // for GrpcFixture
 
-void test(amdinfer::Client* client) {
-  auto endpoint = client->workerLoad("echo", {});
-  EXPECT_EQ(endpoint, "echo");
+namespace amdinfer {
+
+void test(const Client* client) {
+  auto endpoint =
+    client->workerLoad("cplusplus", {{"model"}, {std::string{"echo"}}});
+  EXPECT_EQ(endpoint, "cplusplus");
 
   std::vector<uint32_t> img_data;
   auto shape = {1UL};
@@ -32,12 +35,12 @@ void test(amdinfer::Client* client) {
   img_data.reserve(size);
   img_data.push_back(1);
 
-  amdinfer::InferenceRequest request;
+  InferenceRequest request;
   request.addInputTensor(static_cast<void*>(img_data.data()), shape,
-                         amdinfer::DataType::Uint32);
+                         DataType::Uint32);
 
   const auto num_requests = 16;
-  std::queue<amdinfer::InferenceResponseFuture> q;
+  std::queue<InferenceResponseFuture> q;
   for (auto i = 0; i < num_requests; ++i) {
     q.push(client->modelInferAsync(endpoint, request));
   }
@@ -59,7 +62,7 @@ void test(amdinfer::Client* client) {
     }
   }
 
-  client->modelUnload(endpoint);
+  client->workerUnload(endpoint);
 }
 
 #ifdef AMDINFER_ENABLE_GRPC
@@ -69,7 +72,7 @@ TEST_F(GrpcFixture, ModelInfer) { test(client_.get()); }
 
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(BaseFixture, ModelInfer) {
-  amdinfer::NativeClient client(&server_);
+  NativeClient client(&server_);
   test(&client);
 }
 
@@ -77,3 +80,5 @@ TEST_F(BaseFixture, ModelInfer) {
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(HttpFixture, ModelInfer) { test(client_.get()); }
 #endif
+
+}  // namespace amdinfer
diff --git a/tests/api/test_model_list.cpp b/tests/api/test_model_list.cpp
index 980c7a74f..d0fcc5e34 100644
--- a/tests/api/test_model_list.cpp
+++ b/tests/api/test_model_list.cpp
@@ -23,13 +23,18 @@
 #include "amdinfer/amdinfer.hpp"                // for NativeClient, GrpcClient
 #include "amdinfer/testing/gtest_fixtures.hpp"  // for AssertionResult, Message
 
-void test(amdinfer::Client* client) {
-  const std::string worker = "echo";
+namespace amdinfer {
+
+void test(const Client* client) {
+  const std::string worker = "cplusplus";
 
   auto models_0 = client->modelList();
   EXPECT_TRUE(models_0.empty());
 
-  const auto endpoint = client->workerLoad(worker, {});
+  ParameterMap parameters_0;
+  parameters_0.put("model", "echo");
+  const auto endpoint =
+    client->workerLoad(worker, {{"model"}, {std::string{"echo"}}});
   EXPECT_EQ(endpoint, worker);
   EXPECT_TRUE(client->modelReady(endpoint));
 
@@ -37,9 +42,9 @@ void test(amdinfer::Client* client) {
   EXPECT_EQ(models.size(), 1);
   EXPECT_EQ(models.at(0), endpoint);
 
-  const std::string worker_2 = "invertimage";
-  const auto endpoint_2 = client->workerLoad(worker_2, {});
-  EXPECT_EQ(endpoint_2, worker_2);
+  ParameterMap parameters_2;
+  parameters_2.put("model", "echo_multi");
+  const auto endpoint_2 = client->workerLoad(worker, parameters_2);
   EXPECT_TRUE(client->modelReady(endpoint_2));
 
   auto models_2 = client->modelList();
@@ -66,7 +71,7 @@ TEST_F(GrpcFixture, ModelList) { test(client_.get()); }
 
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(BaseFixture, ModelList) {
-  amdinfer::NativeClient client(&server_);
+  NativeClient client(&server_);
   test(&client);
 }
 
@@ -74,3 +79,5 @@ TEST_F(BaseFixture, ModelList) {
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(HttpFixture, ModelList) { test(client_.get()); }
 #endif
+
+}  // namespace amdinfer
diff --git a/tests/api/test_model_list.py b/tests/api/test_model_list.py
index 57ae8aaed..1378dee57 100644
--- a/tests/api/test_model_list.py
+++ b/tests/api/test_model_list.py
@@ -17,6 +17,8 @@
 
 import pytest
 
+import amdinfer
+
 
 @pytest.mark.usefixtures("server", "assign_client")
 class TestModelList:
@@ -31,16 +33,18 @@ def test_model_list(self):
         models_0 = self.rest_client.modelList()
         assert len(models_0) == 0
 
-        endpoint = self.rest_client.workerLoad("echo")
-        assert endpoint == "echo"
+        parameters = amdinfer.ParameterMap(["model"], ["echo"])
+        endpoint = self.rest_client.workerLoad("cplusplus", parameters)
+        assert endpoint == "cplusplus"
         assert self.rest_client.modelReady(endpoint)
 
         models = self.rest_client.modelList()
         assert len(models) == 1
         assert models[0] == endpoint
 
-        endpoint_2 = self.rest_client.workerLoad("invertimage")
-        assert endpoint_2 == "invertimage"
+        parameters = amdinfer.ParameterMap(["model"], ["echo_multi"])
+        endpoint_2 = self.rest_client.workerLoad("cplusplus", parameters)
+        assert endpoint_2 == "cplusplus-0"
         assert self.rest_client.modelReady(endpoint_2)
 
         models = self.rest_client.modelList()
@@ -48,8 +52,8 @@ def test_model_list(self):
         assert endpoint in models
         assert endpoint_2 in models
 
-        self.rest_client.modelUnload(endpoint)
-        self.rest_client.modelUnload(endpoint_2)
+        self.rest_client.workerUnload(endpoint)
+        self.rest_client.workerUnload(endpoint_2)
 
         models_3 = self.rest_client.modelList()
         while len(models_3) > 0:
diff --git a/tests/api/test_model_load.cpp b/tests/api/test_model_load.cpp
index 46d53e461..617480869 100644
--- a/tests/api/test_model_load.cpp
+++ b/tests/api/test_model_load.cpp
@@ -22,9 +22,10 @@
 #include "amdinfer/amdinfer.hpp"                // for NativeClient, GrpcClient
 #include "amdinfer/testing/gtest_fixtures.hpp"  // for AssertionResult, Suite...
 
-void test(amdinfer::Client* client) {
-  auto metadata = client->serverMetadata();
-  if (metadata.extensions.find("tfzendnn") == metadata.extensions.end()) {
+namespace amdinfer {
+
+void test(const amdinfer::Client* client) {
+  if (!serverHasExtension(client, "tfzendnn")) {
     GTEST_SKIP() << "This test requires TF+ZenDNN support.";
   }
 
@@ -47,12 +48,12 @@ void test(amdinfer::Client* client) {
 #ifdef AMDINFER_ENABLE_GRPC
 // @pytest.mark.extensions(["tfzendnn"])
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
-TEST_F(GrpcFixture, workerLoad) { test(client_.get()); }
+TEST_F(GrpcFixture, modelLoad) { test(client_.get()); }
 #endif
 
 // @pytest.mark.extensions(["tfzendnn"])
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
-TEST_F(BaseFixture, workerLoad) {
+TEST_F(BaseFixture, modelLoad) {
   amdinfer::NativeClient client(&server_);
   test(&client);
 }
@@ -62,3 +63,5 @@ TEST_F(BaseFixture, workerLoad) {
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(HttpFixture, modelLoad) { test(client_.get()); }
 #endif
+
+}  // namespace amdinfer
diff --git a/tests/api/test_model_load.py b/tests/api/test_model_load.py
index 238c8e2c9..da4592c98 100644
--- a/tests/api/test_model_load.py
+++ b/tests/api/test_model_load.py
@@ -20,51 +20,22 @@
 import amdinfer
 
 
-# TODO(varunsh): update test to use modelLoad like C++ variant
+@pytest.mark.extensions(["tfzendnn"])
 @pytest.mark.usefixtures("server", "assign_client")
-class TestLoad:
-    """
-    Base class for Echo worker tests
-    """
-
-    def test_loads(self):
+class TestModelLoad:
+    def test_model_load(self):
         """
         Test loading multiple times
         """
 
+        model = "mnist"
         models = self.rest_client.modelList()
         assert len(models) == 0
 
-        endpoint_0 = self.rest_client.workerLoad("echo")  # this loads the model
-        assert endpoint_0 == "echo"
-        response = self.rest_client.workerLoad(
-            "echo"
-        )  # this will do nothing and return 200
-        assert response == "echo"
-
-        parameters = amdinfer.ParameterMap()
-        parameters.put("max_buffer_num", 100)
-        endpoint_1 = self.rest_client.workerLoad(
-            "echo", parameters
-        )  # load echo with a different config
-        assert endpoint_1 == "echo-0"
-
-        # load echo with the same config as earlier but force allocation of a new worker
-        parameters.put("share", False)
-        response = self.rest_client.workerLoad("echo", parameters)
-        assert response == "echo-0"
+        self.rest_client.modelLoad(model)  # this loads the model
+        assert self.rest_client.modelReady(model)
 
-        assert self.rest_client.modelReady(endpoint_0)
-        assert self.rest_client.modelReady(endpoint_1)
+        self.rest_client.modelUnload(model)
 
-        self.rest_client.modelUnload(endpoint_0)  # this unloads the model echo
-        self.rest_client.modelUnload(endpoint_0)  # this will do nothing and return 200
-        self.rest_client.modelUnload(endpoint_1)  # this unloads the model echo-0
-        self.rest_client.modelUnload(
-            endpoint_1
-        )  # this unloads the second echo-0 worker
-
-        while self.rest_client.modelReady(endpoint_0):
-            time.sleep(1)
-        while self.rest_client.modelReady(endpoint_1):
+        while self.rest_client.modelList():
             time.sleep(1)
diff --git a/tests/api/test_model_metadata.cpp b/tests/api/test_model_metadata.cpp
index 7c5c4873a..7325fa56c 100644
--- a/tests/api/test_model_metadata.cpp
+++ b/tests/api/test_model_metadata.cpp
@@ -22,19 +22,21 @@
 #include "amdinfer/amdinfer.hpp"                // for ModelMetadata, NativeC...
 #include "amdinfer/testing/gtest_fixtures.hpp"  // for AssertionResult, Suite...
 
-void test(amdinfer::Client* client) {
-  const std::string model = "echo";
+namespace amdinfer {
+
+void test(const Client* client) {
+  const std::string model = "cplusplus";
 
   EXPECT_TRUE(client->modelList().empty());
 
   // load one worker
-  client->workerLoad(model, {});
+  client->workerLoad(model, {{"model"}, {std::string{"echo"}}});
 
   EXPECT_TRUE(client->modelReady(model));
 
   auto metadata = client->modelMetadata(model);
-  EXPECT_EQ(metadata.getName(), "echo");
-  EXPECT_EQ(metadata.getPlatform(), "cpu");
+  EXPECT_EQ(metadata.getName(), "CPlusPlus");
+  EXPECT_EQ(metadata.getPlatform(), "CPU");
   // TODO(varunsh): add other assertions
 
   client->workerUnload(model);  // unload the model
@@ -51,7 +53,7 @@ TEST_F(GrpcFixture, modelMetadata) { test(client_.get()); }
 
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(BaseFixture, modelMetadata) {
-  amdinfer::NativeClient client(&server_);
+  NativeClient client(&server_);
   test(&client);
 }
 
@@ -59,3 +61,5 @@ TEST_F(BaseFixture, modelMetadata) {
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(HttpFixture, modelMetadata) { test(client_.get()); }
 #endif
+
+}  // namespace amdinfer
diff --git a/tests/api/test_model_ready.cpp b/tests/api/test_model_ready.cpp
index 565c44ce3..35dd28a4b 100644
--- a/tests/api/test_model_ready.cpp
+++ b/tests/api/test_model_ready.cpp
@@ -22,23 +22,26 @@
 #include "amdinfer/amdinfer.hpp"                // for NativeClient, GrpcClient
 #include "amdinfer/testing/gtest_fixtures.hpp"  // for AssertionResult, Suite...
 
-bool isReady(amdinfer::Client* client, const std::string& endpoint) {
+namespace amdinfer {
+
+bool isReady(const Client* client, const std::string& endpoint) {
   try {
     return client->modelReady(endpoint);
-  } catch (const amdinfer::bad_status&) {
+  } catch (const bad_status&) {
     return false;
   }
 }
 
-void test(amdinfer::Client* client) {
-  const std::string worker = "echo";
+void test(const Client* client) {
+  const std::string worker = "cplusplus";
 
   auto models_0 = client->modelList();
   EXPECT_TRUE(models_0.empty());
 
   EXPECT_FALSE(client->modelReady(worker));
 
-  const auto endpoint = client->workerLoad(worker, {});
+  const auto endpoint =
+    client->workerLoad(worker, {{"model"}, {std::string{"echo"}}});
   EXPECT_EQ(endpoint, worker);
 
   // arbitrarily set to 10ms
@@ -69,7 +72,7 @@ TEST_F(GrpcFixture, ModelReady) { test(client_.get()); }
 
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(BaseFixture, ModelReady) {
-  amdinfer::NativeClient client(&server_);
+  NativeClient client(&server_);
   test(&client);
 }
 
@@ -77,3 +80,5 @@ TEST_F(BaseFixture, ModelReady) {
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(HttpFixture, ModelReady) { test(client_.get()); }
 #endif
+
+}  // namespace amdinfer
diff --git a/tests/api/test_model_ready.py b/tests/api/test_model_ready.py
index 6e797e13f..62547276f 100644
--- a/tests/api/test_model_ready.py
+++ b/tests/api/test_model_ready.py
@@ -31,24 +31,25 @@ def test_model_ready(self):
         Test that modelReady correctly throws errors and
         """
 
-        worker = "echo"
+        worker = "cplusplus"
 
         models = self.rest_client.modelList()
         assert len(models) == 0
 
         assert not self.rest_client.modelReady(worker)
 
-        endpoint_0 = self.rest_client.workerLoad(worker)
-        assert endpoint_0 == "echo"
-        while not self.rest_client.modelReady(worker):
+        parameters = amdinfer.ParameterMap(["model"], ["echo"])
+        endpoint_0 = self.rest_client.workerLoad(worker, parameters)
+        assert endpoint_0 == "cplusplus"
+        while not self.rest_client.modelReady(endpoint_0):
             time.sleep(1)
 
         models = self.rest_client.modelList()
         assert len(models) == 1
 
-        self.rest_client.modelUnload(worker)
+        self.rest_client.modelUnload(endpoint_0)
 
-        while self.rest_client.modelReady(worker):
+        while self.rest_client.modelReady(endpoint_0):
             time.sleep(1)
 
         models = self.rest_client.modelList()
diff --git a/tests/api/test_worker_load.cpp b/tests/api/test_worker_load.cpp
index 808e1e9d9..943ae6efb 100644
--- a/tests/api/test_worker_load.cpp
+++ b/tests/api/test_worker_load.cpp
@@ -21,37 +21,40 @@
 #include "amdinfer/amdinfer.hpp"                // for Client, RequestParame...
 #include "amdinfer/testing/gtest_fixtures.hpp"  // for AssertionResult, Message
 
-void test(amdinfer::Client* client) {
-  std::string worker = "echo";
+namespace amdinfer {
+
+void test(const Client* client) {
+  std::string worker = "cplusplus";
+  ParameterMap parameters{{"model"}, {std::string{"echo"}}};
 
   EXPECT_TRUE(client->modelList().empty());
 
   // load one worker
-  auto endpoint = client->workerLoad(worker, {});
+  auto endpoint = client->workerLoad(worker, parameters);
   EXPECT_EQ(endpoint, worker);
   // do a redundant load
-  endpoint = client->workerLoad(worker, {});
+  endpoint = client->workerLoad(worker, parameters);
   EXPECT_EQ(endpoint, worker);
 
   // load the same worker with a different config
-  amdinfer::ParameterMap parameters;
   // arbitrarily set to 100 just to create a different config
   const auto max_buffer_num = 100;
   parameters.put("max_buffer_num", max_buffer_num);
   auto endpoint_1 = client->workerLoad(worker, parameters);
-  EXPECT_EQ(endpoint_1, "echo-0");
+  const auto expected_endpoint = worker + "-0";
+  EXPECT_EQ(endpoint_1, expected_endpoint);
 
   parameters.put("share", false);
   endpoint_1 = client->workerLoad(worker, parameters);
-  EXPECT_EQ(endpoint_1, "echo-0");
+  EXPECT_EQ(endpoint_1, expected_endpoint);
 
   EXPECT_TRUE(client->modelReady(endpoint));
   EXPECT_TRUE(client->modelReady(endpoint_1));
 
   client->modelUnload(endpoint);    // unload the first
   client->modelUnload(endpoint);    // this will do nothing
-  client->modelUnload(endpoint_1);  // unload first echo-0 worker
-  client->modelUnload(endpoint_1);  // unload second echo-0 worker
+  client->modelUnload(endpoint_1);  // unload first endpoint_1 worker
+  client->modelUnload(endpoint_1);  // unload second endpoint_1 worker
 
   while (!client->modelList().empty()) {
     std::this_thread::yield();
@@ -65,7 +68,7 @@ TEST_F(GrpcFixture, workerLoad) { test(client_.get()); }
 
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(BaseFixture, workerLoad) {
-  amdinfer::NativeClient client(&server_);
+  NativeClient client(&server_);
   test(&client);
 }
 
@@ -73,3 +76,5 @@ TEST_F(BaseFixture, workerLoad) {
 // NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
 TEST_F(HttpFixture, workerLoad) { test(client_.get()); }
 #endif
+
+}  // namespace amdinfer
diff --git a/tests/api/test_worker_load.py b/tests/api/test_worker_load.py
index 173f6d6a1..1fb789a6f 100644
--- a/tests/api/test_worker_load.py
+++ b/tests/api/test_worker_load.py
@@ -34,34 +34,39 @@ def test_loads(self):
         models = self.rest_client.modelList()
         assert len(models) == 0
 
-        endpoint_0 = self.rest_client.workerLoad("echo")  # this loads the model
-        assert endpoint_0 == "echo"
-        response = self.rest_client.workerLoad(
-            "echo"
-        )  # this will do nothing and return 200
-        assert response == "echo"
+        worker = "cplusplus"
+        parameters = amdinfer.ParameterMap(["model"], ["echo"])
 
-        parameters = amdinfer.ParameterMap()
+        # this loads the model
+        endpoint_0 = self.rest_client.workerLoad(worker, parameters)
+        assert endpoint_0 == worker
+        # this will do nothing and return the same endpoint
+        endpoint_0 = self.rest_client.workerLoad(worker, parameters)
+        assert endpoint_0 == worker
+
+        # load the same worker with a different config
+        # arbitrarily set to 100 just to create a different config
         parameters.put("max_buffer_num", 100)
-        endpoint_1 = self.rest_client.workerLoad(
-            "echo", parameters
-        )  # load echo with a different config
-        assert endpoint_1 == "echo-0"
+        endpoint_1 = self.rest_client.workerLoad(worker, parameters)
+        expected_endpoint = worker + "-0"
+        assert endpoint_1 == expected_endpoint
 
-        # load echo with the same config as earlier but force allocation of a new worker
+        # load worker with the same config as earlier but force allocation of a new worker
         parameters.put("share", False)
-        response = self.rest_client.workerLoad("echo", parameters)
-        assert response == "echo-0"
+        endpoint_1 = self.rest_client.workerLoad(worker, parameters)
+        assert endpoint_1 == expected_endpoint
 
         assert self.rest_client.modelReady(endpoint_0)
         assert self.rest_client.modelReady(endpoint_1)
 
-        self.rest_client.modelUnload(endpoint_0)  # this unloads the model echo
-        self.rest_client.modelUnload(endpoint_0)  # this will do nothing and return 200
-        self.rest_client.modelUnload(endpoint_1)  # this unloads the model echo-0
-        self.rest_client.modelUnload(
-            endpoint_1
-        )  # this unloads the second echo-0 worker
+        # this unloads the first model
+        self.rest_client.modelUnload(endpoint_0)
+        # this will do nothing and return
+        self.rest_client.modelUnload(endpoint_0)
+        # this unloads the one copy of the second worker
+        self.rest_client.modelUnload(endpoint_1)
+        # this unloads the second copy of the second worker
+        self.rest_client.modelUnload(endpoint_1)
 
         while self.rest_client.modelReady(endpoint_0):
             time.sleep(1)
diff --git a/tests/conftest.py b/tests/conftest.py
index 219d931e4..65e281f48 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -210,28 +210,26 @@ def startup_check(self):
 def load(request, server):
     test_model, test_parameters = request.cls.get_config()
 
-    assert test_model
+    if isinstance(test_model, str):
+        test_model = [test_model]
+        test_parameters = [test_parameters]
 
-    parameters = amdinfer.ParameterMap()
-    if test_parameters is not None:
-        for key, value in test_parameters.items():
-            parameters.put(key, value)
+    assert test_model
+    assert len(test_model) == len(test_parameters)
 
     request.cls.rest_client = rest_client(request)
     request.cls.ws_client = ws_client(request)
 
-    response = request.cls.rest_client.workerLoad(test_model, parameters)
-    request.cls.endpoint = response
+    endpoints = amdinfer.loadEnsemble(
+        request.cls.rest_client, test_model, test_parameters
+    )
 
-    while not request.cls.rest_client.modelReady(response):
-        time.sleep(1)
+    # update the endpoint with the first worker in the ensemble
+    request.cls.endpoint = endpoints[0]
 
     yield  # perform testing
 
-    request.cls.rest_client.modelUnload(response)
-
-    while request.cls.rest_client.modelReady(response):
-        time.sleep(1)
+    amdinfer.unloadModels(request.cls.rest_client, endpoints)
 
     request.cls.ws_client = None
     request.cls.rest_client = None
diff --git a/tests/performance/batching/benchmark_soft.cpp b/tests/performance/batching/benchmark_soft.cpp
index 3e6415c95..09ab201cf 100644
--- a/tests/performance/batching/benchmark_soft.cpp
+++ b/tests/performance/batching/benchmark_soft.cpp
@@ -87,7 +87,8 @@ class PerfSoftBatcherFixture : public ::benchmark::Fixture {
     this->batcher_->setName("test");
     this->batcher_->setBatchSize(batch_size);
 
-    this->worker_.emplace("", &parameters, &pool_);
+    std::vector<MemoryAllocators> next;
+    this->worker_.emplace("", &parameters, &pool_, nullptr, next);
     // for (size_t i = 0; i < buffer_num; i++) {
     //   BufferPtrs vec;
     //   vec.emplace_back(std::make_unique<VectorBuffer>(batch_size * data_size,
diff --git a/tests/src/amdinfer/core/worker_info_buffers_finite.cpp b/tests/src/amdinfer/core/worker_info_buffers_finite.cpp
index 712688185..99a0b8d76 100644
--- a/tests/src/amdinfer/core/worker_info_buffers_finite.cpp
+++ b/tests/src/amdinfer/core/worker_info_buffers_finite.cpp
@@ -36,7 +36,9 @@
 namespace amdinfer {
 
 WorkerInfo::WorkerInfo(const std::string& name, ParameterMap* parameters,
-                       MemoryPool* pool) {
+                       MemoryPool* pool, BatchPtrQueue* next,
+                       const std::vector<MemoryAllocators>& next_allocators)
+  : next_(next), next_allocators_(next_allocators) {
   this->batch_size_ = 1;
 
   this->addAndStartWorker(name, parameters, pool);
diff --git a/tests/src/amdinfer/core/worker_info_buffers_infinite.cpp b/tests/src/amdinfer/core/worker_info_buffers_infinite.cpp
index c64d03ef4..d6cf69869 100644
--- a/tests/src/amdinfer/core/worker_info_buffers_infinite.cpp
+++ b/tests/src/amdinfer/core/worker_info_buffers_infinite.cpp
@@ -35,7 +35,9 @@
 namespace amdinfer {
 
 WorkerInfo::WorkerInfo(const std::string& name, ParameterMap* parameters,
-                       MemoryPool* pool) {
+                       MemoryPool* pool, BatchPtrQueue* next,
+                       const std::vector<MemoryAllocators>& next_allocators)
+  : next_(next), next_allocators_(next_allocators) {
   this->batch_size_ = 1;
 
   this->addAndStartWorker(name, parameters, pool);
diff --git a/tests/unit/batching/test_soft.cpp b/tests/unit/batching/test_soft.cpp
index 21eea6c35..b4ea5f823 100644
--- a/tests/unit/batching/test_soft.cpp
+++ b/tests/unit/batching/test_soft.cpp
@@ -44,7 +44,7 @@ TEST(UnitSoftBatcher, ConstructAndStart) {
   SoftBatcher batcher(&pool);
   batcher.setName("test");
 
-  WorkerInfo fake("", nullptr, &pool);
+  WorkerInfo fake("", nullptr, &pool, nullptr, {});
   batcher.start({MemoryAllocators::Cpu});
 
   batcher.enqueue(nullptr);
diff --git a/tests/unit/batching/test_soft_batching.cpp b/tests/unit/batching/test_soft_batching.cpp
index 054ae9f8c..c8665377d 100644
--- a/tests/unit/batching/test_soft_batching.cpp
+++ b/tests/unit/batching/test_soft_batching.cpp
@@ -96,7 +96,8 @@ class UnitSoftBatcherFixture : public testing::TestWithParam<BatchConfig> {
     this->batcher_->setName("test");
     this->batcher_->setBatchSize(batch_size);
 
-    this->worker_.emplace("", &parameters, &pool_);
+    std::vector<MemoryAllocators> next;
+    this->worker_.emplace("", &parameters, &pool_, nullptr, next);
     // for (size_t i = 0; i < buffer_num; i++) {
     //   BufferPtrs vec;
     //   vec.emplace_back(std::make_unique<VectorBuffer>(batch_size * data_size,
diff --git a/tests/unit/buffers/CMakeLists.txt b/tests/unit/buffers/CMakeLists.txt
index 473c01145..9e6f7e74d 100644
--- a/tests/unit/buffers/CMakeLists.txt
+++ b/tests/unit/buffers/CMakeLists.txt
@@ -15,6 +15,10 @@
 
 list(APPEND tests cpu)
 
-list(APPEND tests_libs "buffer~data_types~fake_observation~cpu_buffer")
+list(
+  APPEND
+    tests_libs
+    "buffers~data_types~fake_observation~memory_pool~data_types_internal~tensor"
+)
 
 amdinfer_add_unit_tests("${tests}" "${tests_libs}")
diff --git a/tests/unit/clients/test_grpc_internal.cpp b/tests/unit/clients/test_grpc_internal.cpp
index 496211a00..4b6aeaad4 100644
--- a/tests/unit/clients/test_grpc_internal.cpp
+++ b/tests/unit/clients/test_grpc_internal.cpp
@@ -134,6 +134,7 @@ TEST_P(Fixture, TestRequestToProto) {  // NOLINT
   input.setData(data.data());
   switchOverTypes(AssignData(), datatype, data.data());
   input.setDatatype(datatype);
+  input.setShape({1});
   request.addInputTensor(input);
 
   inference::ModelInferRequest proto_request;
diff --git a/tests/unit/core/memory_pool/CMakeLists.txt b/tests/unit/core/memory_pool/CMakeLists.txt
index a2c515184..8559b1ee9 100644
--- a/tests/unit/core/memory_pool/CMakeLists.txt
+++ b/tests/unit/core/memory_pool/CMakeLists.txt
@@ -16,7 +16,7 @@ list(APPEND tests cpu_allocator pool)
 
 list(
   APPEND tests_libs
-         "cpu_allocator~buffers~inference_request~data_types~parameters~\
+         "memory_pool~buffers~inference_request~data_types~parameters~\
            data_types_internal~inference_response"
          "memory_pool~buffers~inference_request~data_types~parameters~\
            data_types_internal~inference_response"
diff --git a/tests/unit/core/memory_pool/test_pool.cpp b/tests/unit/core/memory_pool/test_pool.cpp
index 22dd55e5d..48e9067b2 100644
--- a/tests/unit/core/memory_pool/test_pool.cpp
+++ b/tests/unit/core/memory_pool/test_pool.cpp
@@ -34,7 +34,7 @@ TEST(UnitPool, Basic) {
 
   auto buffer = pool.get({MemoryAllocators::Cpu}, input, 1);
 
-  pool.put(std::move(buffer));
+  buffer->free();
 }
 
 }  // namespace amdinfer
diff --git a/tests/unit/core/test_parameter_map.cpp b/tests/unit/core/test_parameter_map.cpp
index b97859e26..7cb2bb89f 100644
--- a/tests/unit/core/test_parameter_map.cpp
+++ b/tests/unit/core/test_parameter_map.cpp
@@ -19,7 +19,7 @@
 #include <vector>   // for vector
 
 #include "amdinfer/core/parameters.hpp"  // for ParameterMap
-#include "gtest/gtest.h"                 // for AssertionResult, Message
+#include "amdinfer/testing/gtest.hpp"    // for EXPECT_THROW_CHECK
 
 namespace amdinfer {
 
@@ -74,4 +74,27 @@ TEST(UnitParameterMap, SerDes) {
             new_params.get<double>(keys[index_double]));
 }
 
+// NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
+TEST(UnitParameterMap, InitializingConstructorConstChar) {
+  ParameterMap parameters{{"model"}, {"echo"}};
+
+  // Passing const char* to a variant type stores it as a bool instead of a
+  // string if both types are present. This has been fixed in C++20
+  // (https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0608r3.html)
+  std::string model;
+  EXPECT_THROW_CHECK(model = parameters.get<std::string>("model");
+                     , EXPECT_STREQ(e.what(), "Unexpected index"),
+                     std::bad_variant_access);
+
+  parameters.erase("model");
+  parameters.put("model", std::string{"echo"});
+  model = parameters.get<std::string>("model");
+  EXPECT_EQ(model, "echo");
+
+  parameters.erase("model");
+  parameters.put("model", "echo");
+  model = parameters.get<std::string>("model");
+  EXPECT_EQ(model, "echo");
+}
+
 }  // namespace amdinfer
diff --git a/tests/workers/CMakeLists.txt b/tests/workers/CMakeLists.txt
index 346769922..3a938e21f 100644
--- a/tests/workers/CMakeLists.txt
+++ b/tests/workers/CMakeLists.txt
@@ -13,10 +13,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-list(APPEND tests echo)
+set(tests cplusplus)
 
 if(${AMDINFER_ENABLE_AKS})
-  set(tests ${tests} facedetect)
+  list(APPEND tests facedetect)
 endif()
 
 amdinfer_add_system_tests("${tests}")
diff --git a/tests/workers/facedetect.hpp b/tests/workers/facedetect.hpp
index 1b4d38716..00d3d7241 100644
--- a/tests/workers/facedetect.hpp
+++ b/tests/workers/facedetect.hpp
@@ -87,8 +87,8 @@ inline std::string load(const amdinfer::NativeClient& client, int workers) {
                  "graph_facedetect_u200_u250_amdinfer.json");
   parameters.put("share", false);
 
-  std::string endpoint;
-  endpoint = client.workerLoad("AksDetect", parameters);
+  auto endpoint = client.workerLoad("AksDetect", parameters);
+
   for (int i = 0; i < workers - 1; i++) {
     const auto new_endpoint = client.workerLoad("AksDetect", parameters);
     assert(endpoint == new_endpoint);
diff --git a/tests/workers/test_aks.py b/tests/workers/test_aks.py
deleted file mode 100644
index 3736de9cf..000000000
--- a/tests/workers/test_aks.py
+++ /dev/null
@@ -1,66 +0,0 @@
-# Copyright 2021 Xilinx, Inc.
-# Copyright 2022 Advanced Micro Devices, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import numpy as np
-import pytest
-
-import amdinfer
-
-
-@pytest.mark.usefixtures("load")
-@pytest.mark.extensions(["aks", "vitis"])
-class TestAks:
-    @staticmethod
-    def get_config():
-        model = "aks"
-        parameters = {"batch_size": 1}
-        return (model, parameters)
-
-    def test_aks_0(self):
-        numbers = [3.0]
-        request = amdinfer.InferenceRequest()
-        request.id = "hello_world"
-
-        for num in numbers:
-            input_0 = amdinfer.InferenceRequestInput()
-            input_0.name = "aks"
-            input_0.setFp32Data(np.asarray([num], np.float32))
-            input_0.shape = [1]
-            input_0.datatype = amdinfer.DataType.FP32
-            request.addInputTensor(input_0)
-        try:
-            response: amdinfer.InferenceResponse = self.rest_client.modelInfer(
-                self.endpoint, request
-            )
-        except ConnectionError:
-            pytest.fail(
-                "Connection to the amdinfer server ended without response!", False
-            )
-
-        assert not response.isError(), response.getError()
-
-        assert response.id == "hello_world"
-        assert response.model == "aks"
-
-        outputs = response.getOutputs()
-        assert len(outputs) == len(numbers)
-        for index, num in enumerate(numbers):
-            reply = outputs[index]
-            data = reply.getFp32Data()
-            assert len(data) == 1
-            assert reply.name == "aks"
-            assert data[0] == num + 60
-            assert reply.datatype == amdinfer.DataType.FP32
-            assert reply.parameters.empty()
diff --git a/tests/workers/test_echo.cpp b/tests/workers/test_cplusplus.cpp
similarity index 92%
rename from tests/workers/test_echo.cpp
rename to tests/workers/test_cplusplus.cpp
index d6fd978cb..bdab35bf9 100644
--- a/tests/workers/test_echo.cpp
+++ b/tests/workers/test_cplusplus.cpp
@@ -61,7 +61,7 @@ class EchoParamFixture : public testing::TestWithParam<Params> {
 
     if (params.add_outputs) {
       InferenceRequestOutput output;
-      output.setName("echo");
+      output.setName("");
       if (params.add_input_parameters) {
         ParameterMap parameters;
         parameters.put("key", "another_value");
@@ -102,7 +102,7 @@ class EchoParamFixture : public testing::TestWithParam<Params> {
       const auto* data = static_cast<uint32_t*>(output.getData());
       EXPECT_EQ(data[0], golden_outputs[0]);
       EXPECT_EQ(output.getDatatype(), DataType::Uint32);
-      EXPECT_EQ(output.getName(), "echo");
+      EXPECT_EQ(output.getName(), "");
       EXPECT_TRUE(output.getParameters().empty());
       auto shape = output.getShape();
       EXPECT_EQ(shape.size(), 1);
@@ -119,7 +119,8 @@ class EchoParamFixture : public testing::TestWithParam<Params> {
 
 TEST_P(EchoParamFixture, EchoNative) {  // NOLINT
   amdinfer::NativeClient client(&server);
-  const auto endpoint = client.workerLoad("echo", {});
+  auto endpoint =
+    client.workerLoad("cplusplus", {{"model"}, {std::string{"echo"}}});
 
   auto request = this->constructRequest();
 
@@ -127,7 +128,7 @@ TEST_P(EchoParamFixture, EchoNative) {  // NOLINT
 
   validate(response);
 
-  client.modelUnload(endpoint);
+  client.workerUnload(endpoint);
 }
 
 #ifdef AMDINFER_ENABLE_GRPC
@@ -135,18 +136,17 @@ TEST_P(EchoParamFixture, EchoGrpc) {  // NOLINT
   server.startGrpc(kDefaultGrpcPort);
   auto client =
     amdinfer::GrpcClient("localhost:" + std::to_string(kDefaultGrpcPort));
-  while (!client.serverLive()) {
-    std::this_thread::sleep_for(std::chrono::seconds(1));
-  }
+  waitUntilServerReady(&client);
 
-  const auto endpoint = client.workerLoad("echo", {});
+  auto endpoint =
+    client.workerLoad("cplusplus", {{"model"}, {std::string{"echo"}}});
 
   auto request = this->constructRequest();
 
   auto response = client.modelInfer(endpoint, request);
   validate(response);
 
-  client.modelUnload(endpoint);
+  client.workerUnload(endpoint);
 }
 #endif
 
diff --git a/tests/workers/test_cplusplus.py b/tests/workers/test_cplusplus.py
index 09a00769a..31ab0ed80 100644
--- a/tests/workers/test_cplusplus.py
+++ b/tests/workers/test_cplusplus.py
@@ -26,8 +26,8 @@ class TestCPlusPlus:
 
     @staticmethod
     def get_config():
-        model = "CPlusPlus"
-        parameters = {"model": "echo"}
+        model = ["CPlusPlus"]
+        parameters = [amdinfer.ParameterMap(["model"], ["echo"])]
         return (model, parameters)
 
     inputs = [3]
@@ -73,7 +73,7 @@ def send_request(self, request, input_tensors=1):
             )
 
         assert not response.isError(), response.getError()
-        assert response.model == "CPlusPlus"
+        assert response.model == "echo"
 
         outputs = response.getOutputs()
         assert len(outputs) == 1 * input_tensors
@@ -106,8 +106,12 @@ class TestCPlusPlus2:
 
     @staticmethod
     def get_config():
-        model = "CPlusPlus"
-        parameters = {"model": "echo_multi", "batch_size": 2, "timeout": 1000}
+        model = ["CPlusPlus"]
+        parameters = [
+            amdinfer.ParameterMap(
+                ["model", "batch_size", "timeout"], ["echo_multi", 2, 1000]
+            )
+        ]
         return (model, parameters)
 
     inputs = [[3], [2, 7]]
@@ -157,7 +161,7 @@ def send_request(self, request, input_tensors=1):
 
         for response in responses:
             assert not response.isError(), response.getError()
-            assert response.model == "CPlusPlus"
+            assert response.model == "echo_multi"
 
             outputs = response.getOutputs()
             assert len(outputs) == len(self.golden_outputs)
diff --git a/tests/workers/test_echo.py b/tests/workers/test_echo.py
deleted file mode 100644
index 363034a3b..000000000
--- a/tests/workers/test_echo.py
+++ /dev/null
@@ -1,316 +0,0 @@
-# Copyright 2021 Xilinx, Inc.
-# Copyright 2022 Advanced Micro Devices, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import numpy as np
-import pytest
-
-import amdinfer
-
-from helper import run_benchmark
-
-
-@pytest.mark.usefixtures("load")
-class TestEcho:
-    """
-    Test the Echo worker
-    """
-
-    @staticmethod
-    def get_config():
-        model = "echo"
-        parameters = None
-        return (model, parameters)
-
-    inputs = [3]
-    golden_outputs = [4]
-
-    @classmethod
-    def construct_request(
-        cls, add_id, add_input_parameters, add_request_parameters, multiplier=1
-    ):
-        """
-        Construct a standard request subject to some options
-
-        Args:
-            add_id (bool): Add an ID to the request
-            add_input_parameters (bool): Add parameters to the input tensors
-            add_request_parameters (bool): Add parameters to the request
-
-        Returns:
-            dict: The constructed request
-        """
-
-        input_0 = amdinfer.InferenceRequestInput()
-        input_0.name = "echo"
-        input_0.datatype = amdinfer.DataType.UINT32
-        input_0.shape = [1]
-        input_0.setUint32Data(np.array([cls.inputs[0]], np.uint32))
-        if add_input_parameters:
-            parameters = amdinfer.ParameterMap()
-            parameters.put("key", "value")
-            input_0.parameters = parameters
-
-        request = amdinfer.InferenceRequest()
-        for _ in range(multiplier):
-            request.addInputTensor(input_0)
-
-        if add_id:
-            request.id = "hello_world"
-
-        if add_request_parameters:
-            parameters = amdinfer.ParameterMap()
-            parameters.put("key3", True)
-            parameters.put("key4", 1.2)
-            request.parameters = parameters
-
-        return request
-
-    @staticmethod
-    def add_outputs(request: amdinfer.InferenceRequest, add_parameters):
-        """
-        Add output tensors to the request. This is optional.
-
-        Args:
-            request (dict): The request to add outputs to
-            add_parameters (bool): Add parameters to the output tensors
-
-        Returns:
-            dict: Updated request
-        """
-        output_0 = amdinfer.InferenceRequestOutput()
-        output_0.name = "echo"
-        if add_parameters:
-            parameters = amdinfer.ParameterMap()
-            parameters.put("key", "value2")
-            output_0.parameters = parameters
-
-        request.addOutputTensor(output_0)
-        request.addOutputTensor(output_0)
-        return request
-
-    def send_request(self, request, id_present, input_tensors=1):
-        """
-        Sends the given request to the server and asserts common checks
-
-        Args:
-            request (InferenceRequest): Request to send to the server
-            input_tensors (int): Number of times the two input tensors are repeated
-
-        Returns:
-            dict: Response as a dictionary
-        """
-
-        try:
-            response = self.rest_client.modelInfer(self.endpoint, request)
-        except ConnectionError:
-            pytest.fail(
-                "Connection to the amdinfer server ended without response!", False
-            )
-
-        assert not response.isError(), response.getError()
-        assert response.model == "echo"
-
-        outputs = response.getOutputs()
-        assert len(outputs) == 1 * input_tensors
-
-        for i in range(input_tensors):
-            output = outputs[i]
-            data = output.getUint32Data()
-            assert len(data) == 1
-            assert data[0] == self.golden_outputs[0]
-            assert output.datatype == amdinfer.DataType.UINT32
-            assert output.name == "echo"
-            assert output.parameters.empty()
-            assert output.shape == [1]
-
-        if id_present:
-            assert response.id == "hello_world"
-
-        return response
-
-    def test_echo_0(self):
-        """
-        Send a request to echo with all optional parameters
-        """
-        add_id = True
-        request = self.construct_request(add_id, True, True)
-        request = self.add_outputs(request, True)
-
-        self.send_request(request, add_id)
-
-    def test_echo_1(self):
-        """
-        Send a request to echo with most optional parameters (remove request
-        parameters)
-        """
-        add_id = True
-        request = self.construct_request(add_id, True, False)
-        request = self.add_outputs(request, True)
-
-        self.send_request(request, add_id)
-
-    def test_echo_2(self):
-        """
-        Send a request to echo with some optional parameters (remove all
-        parameters)
-        """
-        add_id = True
-        request = self.construct_request(add_id, False, False)
-        request = self.add_outputs(request, False)
-
-        self.send_request(request, add_id)
-
-    def test_echo_3(self):
-        """
-        Send a request to echo with some optional parameters (remove all
-        parameters and remove the ID)
-        """
-        add_id = False
-        request = self.construct_request(add_id, False, False)
-        request = self.add_outputs(request, False)
-
-        self.send_request(request, add_id)
-
-    def test_echo_4(self):
-        """
-        Send a request to echo with no optional parameters (remove all
-        parameters, remove the ID and output tensors)
-        """
-        add_id = False
-        request = self.construct_request(add_id, False, False)
-
-        self.send_request(request, add_id)
-
-    @pytest.mark.benchmark(group="echo")
-    def test_benchmark_echo_0(self, benchmark):
-        request = self.construct_request(False, False, False)
-        model, parameters = self.get_config()
-        options = {
-            "model": model,
-            "parameters": parameters,
-            "type": "rest (pytest)",
-            "config": "N/A",
-        }
-        run_benchmark(
-            benchmark, "echo_4", self.rest_client.modelInfer, request, **options
-        )
-
-
-# TODO(varunsh): these tests cannot run now with the new Pybind11 API as it
-# doesn't allow creating these kinds of bad requests like this. We'd need to
-# reimplement them another way to test these cases
-# @pytest.mark.usefixtures("load")
-# class TestEchoErrors:
-#     """
-#     Test the Echo worker with error-causing requests
-#     """
-
-#     def send_request(self, request, error):
-#         """
-#         Send the request to echo and check the error message
-
-#         Args:
-#             request (dict): The request to send
-#             error (str): Error message to expect
-#         """
-#         try:
-#             response = self.rest_client.modelInfer("echo", request)
-#         except ConnectionError:
-#             pytest.fail(
-#                 "Connection to the amdinfer server ended without response!", False
-#             )
-
-#         assert response.error
-#         assert response.status_code == 400
-#         assert response.error_msg == error
-
-#     def test_no_inputs(self):
-#         """
-#         Providing no input tensors is an error
-#         """
-#         request = {"foo": "bar"}
-
-#         self.send_request(request, "No 'inputs' key present in request")
-
-#     def test_bad_inputs_0(self):
-#         """
-#         Providing non-array input tensors is an error
-#         """
-#         request = {"inputs": "bar"}
-
-#         self.send_request(request, "'inputs' is not an array")
-
-#     def test_bad_inputs_1(self):
-#         """
-#         If inputs is an array, each element must be an object (e.g. a dict)
-#         """
-#         request = {"inputs": [0]}
-
-#         self.send_request(request, "At least one element in 'inputs' is not an obj")
-
-#     def test_bad_inputs_2(self):
-#         """
-#         Each input tensor must be named
-#         """
-#         request = {"inputs": [{"foo": "bar"}], "outputs": []}
-
-#         self.send_request(request, "No 'name' key present in request input")
-
-#     def test_bad_inputs_3(self):
-#         """
-#         Each input tensor must define its shape
-#         """
-#         request = {"inputs": [{"name": "bar"}], "outputs": []}
-
-#         self.send_request(request, "No 'shape' key present in request input")
-
-#     def test_bad_inputs_4(self):
-#         """
-#         An input tensor's shape must be specified with a value that can be
-#         coerced into a unint64 value
-#         """
-#         request = {"inputs": [{"name": "bar", "shape": ["foo"]}], "outputs": []}
-
-#         self.send_request(request, "'shape' must be specified by uint64 elements")
-
-#     def test_bad_inputs_5(self):
-#         """
-#         All input tensors must define their datatype
-#         """
-#         request = {"inputs": [{"name": "bar", "shape": [1]}], "outputs": []}
-
-#         self.send_request(request, "No 'datatype' key present in request input")
-
-#     def test_bad_inputs_6(self):
-#         """
-#         Input tensors' datatypes must be one of the supported values
-#         """
-#         request = {
-#             "inputs": [{"name": "bar", "shape": [1], "datatype": "BAD_TYPE"}],
-#             "outputs": [],
-#         }
-
-#         self.send_request(request, "Unknown datatype: BAD_TYPE")
-
-#     def test_bad_inputs_7(self):
-#         """
-#         Input tensors must define their input data
-#         """
-#         request = {
-#             "inputs": [{"name": "bar", "shape": [1], "datatype": "UINT32"}],
-#             "outputs": [],
-#         }
-
-#         self.send_request(request, "No 'data' key present in request input")
diff --git a/tests/workers/test_echo_multi.py b/tests/workers/test_echo_multi.py
deleted file mode 100644
index 90dda9358..000000000
--- a/tests/workers/test_echo_multi.py
+++ /dev/null
@@ -1,103 +0,0 @@
-# Copyright 2022 Advanced Micro Devices, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import numpy as np
-import pytest
-
-import amdinfer
-
-
-@pytest.mark.usefixtures("load")
-class TestEchoMulti:
-    """
-    Test the EchoMulti worker
-    """
-
-    @staticmethod
-    def get_config():
-        model = "echoMulti"
-        parameters = {"batch_size": 2, "timeout": 1000}
-        return (model, parameters)
-
-    inputs = [[3], [2, 7]]
-    golden_outputs = [[3], [2, 7, 3, 2], [7, 3, 2]]
-
-    @classmethod
-    def construct_request(cls):
-        """
-        Construct a standard request
-
-        Returns:
-            dict: The constructed request
-        """
-
-        request = amdinfer.InferenceRequest()
-        for i in cls.inputs:
-            input_0 = amdinfer.InferenceRequestInput()
-            input_0.name = "echoMulti0"
-            input_0.datatype = amdinfer.DataType.UINT32
-            input_0.shape = [len(i)]
-            input_0.setUint32Data(np.array(i, np.uint32))
-            request.addInputTensor(input_0)
-
-        return request
-
-    def send_request(self, request):
-        """
-        Sends the given request to the server and asserts common checks
-
-        Args:
-            request (InferenceRequest): Request to send to the server
-            input_tensors (int): Number of times the two input tensors are repeated
-
-        Returns:
-            dict: Response as a dictionary
-        """
-
-        requests = [request] * 2
-        try:
-            responses = amdinfer.inferAsyncOrdered(
-                self.rest_client, self.endpoint, requests
-            )
-        except ConnectionError:
-            pytest.fail(
-                "Connection to the amdinfer server ended without response!", False
-            )
-
-        for response in responses:
-            assert not response.isError(), response.getError()
-            assert response.model == "echoMulti"
-
-            outputs = response.getOutputs()
-            assert len(outputs) == len(self.golden_outputs)
-
-            for i in range(len(outputs)):
-                output = outputs[i]
-                golden_output = self.golden_outputs[i]
-                if not isinstance(golden_output, list):
-                    golden_output = [golden_output]
-                data: list = output.getUint32Data()
-                assert len(data) == len(golden_output)
-                assert (data == golden_output).all(), data
-                assert output.datatype == amdinfer.DataType.UINT32
-                assert output.name == "echoMulti0"
-                assert output.parameters.empty()
-                assert output.shape == [len(golden_output)]
-
-    def test_multi_echo_0(self):
-        """
-        Send a request to echoMulti
-        """
-        request = self.construct_request()
-        self.send_request(request)
diff --git a/tests/workers/test_facedetect.cpp b/tests/workers/test_facedetect.cpp
index 598ace4da..95b8ff12b 100644
--- a/tests/workers/test_facedetect.cpp
+++ b/tests/workers/test_facedetect.cpp
@@ -56,8 +56,8 @@ void dequeueValidate(FutureQueue& my_queue, int num_images) {
       auto num_boxes = 1;
       auto shape = output.getShape();
       EXPECT_EQ(shape.size(), 2);
-      EXPECT_EQ(shape[0], kGoldResponseSize);
-      EXPECT_EQ(shape[1], num_boxes);
+      EXPECT_EQ(shape[0], num_boxes);
+      EXPECT_EQ(shape[1], kGoldResponseSize);
       EXPECT_EQ(size, kGoldResponseSize);
       const auto tolerance = 0.05;
       for (size_t j = 0; j < kGoldResponseSize; j++) {
diff --git a/tests/workers/test_facedetect.py b/tests/workers/test_facedetect.py
index 5577d0353..dfaf34032 100644
--- a/tests/workers/test_facedetect.py
+++ b/tests/workers/test_facedetect.py
@@ -34,12 +34,16 @@ class TestInferImageFacedetectDPUCADF8H:
 
     @staticmethod
     def get_config():
-        model = "AksDetect"
-        parameters = {
-            "aks_graph_name": "facedetect",
-            "aks_graph": "${AKS_ROOT}/graph_zoo/graph_facedetect_u200_u250_amdinfer.json",
-        }
-        return (model, parameters)
+        model = ["CPlusPlus", "AksDetect"]
+        parameters = amdinfer.ParameterMap(["model"], ["base64_decode"])
+
+        aks_parameters = amdinfer.ParameterMap()
+        aks_parameters.put("aks_graph_name", "facedetect")
+        aks_parameters.put(
+            "aks_graph",
+            "${AKS_ROOT}/graph_zoo/graph_facedetect_u200_u250_amdinfer.json",
+        )
+        return (model, [parameters, aks_parameters])
 
     def send_request(self, request, check_asserts=True):
         """
@@ -79,12 +83,12 @@ def send_request(self, request, check_asserts=True):
             outputs = response.getOutputs()
             assert len(outputs) == num_inputs
             for index, output in enumerate(outputs):
-                assert output.name == "input" + str(index)
+                assert output.name == ""
                 assert output.datatype == amdinfer.DataType.FP32
                 assert output.parameters.empty()
                 data = output.getFp32Data()
                 num_boxes = int(len(data) / 6)
-                assert output.shape == [6, num_boxes]
+                assert output.shape == [num_boxes, 6]
                 assert len(data) == len(
                     gold_response_output
                 ), f"{len(data)} != {len(gold_response_output)}"
diff --git a/tests/workers/test_facedetect_stream.py b/tests/workers/test_facedetect_stream.py
index d4b46ad2e..f7916b2a8 100644
--- a/tests/workers/test_facedetect_stream.py
+++ b/tests/workers/test_facedetect_stream.py
@@ -32,10 +32,13 @@ class TestFacedetectStream:
     @staticmethod
     def get_config():
         model = "AksDetectStream"
-        parameters = {
-            "aks_graph_name": "facedetect",
-            "aks_graph": "${AKS_ROOT}/graph_zoo/graph_facedetect_u200_u250_amdinfer.json",
-        }
+        parameters = amdinfer.ParameterMap(
+            ["aks_graph_name", "aks_graph"],
+            [
+                "facedetect",
+                "${AKS_ROOT}/graph_zoo/graph_facedetect_u200_u250_amdinfer.json",
+            ],
+        )
         return (model, parameters)
 
     def construct_request(self, requested_frames_count):
diff --git a/tests/workers/test_invert_image.py b/tests/workers/test_invert_image.py
index fff7ed83b..d38ed84f7 100644
--- a/tests/workers/test_invert_image.py
+++ b/tests/workers/test_invert_image.py
@@ -50,6 +50,41 @@ def compare_jpgs(test_image, reference_image):
     assert m_norm_per_pixel < 2  # arbitrarily use a threshold of 2 for comparison
 
 
+def construct_request(asTensor):
+    image_path = amdinfer.testing.getPathToAsset("asset_dog-3619020_640.jpg")
+
+    request = amdinfer.ImageInferenceRequest(image_path, asTensor)
+
+    image = cv2.imread(image_path)
+    image = cv2.bitwise_not(image)
+
+    return request, image
+
+
+def validate(response, image):
+    assert not response.isError(), response.getError()
+
+    assert response.model == "invert_image"
+
+    outputs = response.getOutputs()
+    assert len(outputs) == 1
+
+    output = outputs[0]
+    if output.datatype == amdinfer.DataType.STRING:
+        data = output.getStringData()
+        compare_jpgs(data, image)
+        assert output.parameters.empty()
+    else:
+        assert output.shape == [*image.shape]
+        assert output.datatype == amdinfer.DataType.UINT8
+        data = output.getUint8Data()
+        assert len(data) == image.size
+        assert (data == image.flatten().tolist()).all()
+        assert output.parameters.empty()
+
+    return response
+
+
 @pytest.mark.usefixtures("load")
 class TestInvertImage:
     """
@@ -58,8 +93,9 @@ class TestInvertImage:
 
     @staticmethod
     def get_config():
-        model = "InvertImage"
-        parameters = None
+        model = "CPlusPlus"
+        parameters = amdinfer.ParameterMap()
+        parameters.put("model", "invert_image")
         return (model, parameters)
 
     def send_request(self, request, image):
@@ -72,7 +108,7 @@ def send_request(self, request, image):
             check_asserts (bool): Verify image against golden
 
         Returns:
-            Response: Response as a dictionary
+            Response: Response
         """
 
         try:
@@ -82,65 +118,27 @@ def send_request(self, request, image):
                 "Connection to the amdinfer server ended without response!", False
             )
 
-        assert not response.isError(), response.getError()
-
-        assert response.model == "invert_image"
-
-        outputs = response.getOutputs()
-        assert len(outputs) == 1
-
-        output = outputs[0]
-        if output.datatype == amdinfer.DataType.STRING:
-            data = output.getStringData()
-            compare_jpgs(data, image)
-            assert output.parameters.empty()
-        else:
-            assert output.shape == [*image.shape]
-            assert output.datatype == amdinfer.DataType.UINT8
-            data = output.getUint8Data()
-            assert len(data) == image.size
-            assert (data == image.flatten().tolist()).all()
-            assert output.parameters.empty()
-
-        return response
+        validate(response, image)
 
-    def construct_request(self, asTensor):
-        image_path = amdinfer.testing.getPathToAsset("asset_dog-3619020_640.jpg")
-
-        request = amdinfer.ImageInferenceRequest(image_path, asTensor)
-
-        image = cv2.imread(image_path)
-        image = cv2.bitwise_not(image)
-
-        return request, image
-
-    def test_invert_image_0(self):
+    def test_invert_image(self):
         """
         Send a request to InvertImage as tensor data
         """
-        request, image = self.construct_request(True)
-        self.send_request(request, image)
-
-    def test_invert_image_1(self):
-        """
-        Send a request to InvertImage as a base64-encoded image
-        """
-
-        request, image = self.construct_request(False)
+        request, image = construct_request(True)
         self.send_request(request, image)
 
     @pytest.mark.benchmark(group="invert_image")
-    def test_benchmark_invert_image_0(self, benchmark):
+    def test_benchmark_invert_image(self, benchmark):
         options = {
             "model": self.model,
             "parameters": self.parameters,
             "type": "rest (pytest)",
             "config": "N/A",
         }
-        run_benchmark_func(benchmark, self.test_invert_image_0, **options)
+        run_benchmark_func(benchmark, self.test_invert_image, **options)
 
     @pytest.mark.benchmark(group="invert_image")
-    def test_benchmark_invert_image_0_request(self, benchmark):
+    def test_benchmark_invert_image_request(self, benchmark):
         request, _ = self.construct_request(True)
         options = {
             "model": self.model,
@@ -152,18 +150,66 @@ def test_benchmark_invert_image_0_request(self, benchmark):
             benchmark, "invert_image_0", self.rest_client.modelInfer, request, **options
         )
 
+
+@pytest.mark.usefixtures("load")
+class TestInvertImage2:
+    """
+    Test the InvertImage worker
+    """
+
+    @staticmethod
+    def get_config():
+        model = ["CPlusPlus", "CPlusPlus", "CPlusPlus"]
+        parameters_0 = amdinfer.ParameterMap()
+        parameters_0.put("model", "base64_decode")
+        parameters_1 = amdinfer.ParameterMap()
+        parameters_1.put("model", "invert_image")
+        parameters_2 = amdinfer.ParameterMap()
+        parameters_2.put("model", "base64_encode")
+        return (model, [parameters_0, parameters_1, parameters_2])
+
+    def send_request(self, request, image):
+        """
+        Sends the given request to the server and asserts common checks
+
+        Args:
+            request (dict): request to send to the server
+            image (np.ndarray): Golden image to check against
+            check_asserts (bool): Verify image against golden
+
+        Returns:
+            Response: Response
+        """
+
+        try:
+            response = self.rest_client.modelInfer(self.endpoint, request)
+        except ConnectionError:
+            pytest.fail(
+                "Connection to the amdinfer server ended without response!", False
+            )
+
+        validate(response, image)
+
+    def test_invert_image(self):
+        """
+        Send a request to InvertImage as a base64-encoded image
+        """
+
+        request, image = construct_request(False)
+        self.send_request(request, image)
+
     @pytest.mark.benchmark(group="invert_image")
-    def test_benchmark_invert_image_1(self, benchmark):
+    def test_benchmark_invert_image(self, benchmark):
         options = {
             "model": self.model,
             "parameters": self.parameters,
             "type": "rest (pytest)",
             "config": "N/A",
         }
-        run_benchmark_func(benchmark, self.test_invert_image_1, **options)
+        run_benchmark_func(benchmark, self.test_invert_image, **options)
 
     @pytest.mark.benchmark(group="invert_image")
-    def test_benchmark_invert_image_1_request(self, benchmark):
+    def test_benchmark_invert_image_request(self, benchmark):
         request, _ = self.construct_request(False)
         options = {
             "model": self.model,
diff --git a/tests/workers/test_invert_video.py b/tests/workers/test_invert_video.py
index 2cec1fde1..1d46b155d 100644
--- a/tests/workers/test_invert_video.py
+++ b/tests/workers/test_invert_video.py
@@ -34,7 +34,7 @@ class TestInvertVideo:
     @staticmethod
     def get_config():
         model = "InvertVideo"
-        parameters = None
+        parameters = amdinfer.ParameterMap()
         return (model, parameters)
 
     def construct_request(self, video_path, requested_frames_count):
diff --git a/tests/workers/test_migraphx.py b/tests/workers/test_migraphx.py
index 485d7260d..e40540d97 100644
--- a/tests/workers/test_migraphx.py
+++ b/tests/workers/test_migraphx.py
@@ -77,7 +77,9 @@ class TestMigraphx:
     @staticmethod
     def get_config():
         model = "Migraphx"
-        parameters = {"model": amdinfer.testing.getPathToAsset("onnx_resnet50")}
+        parameters = amdinfer.ParameterMap(
+            ["model"], [amdinfer.testing.getPathToAsset("onnx_resnet50")]
+        )
         return (model, parameters)
 
     def send_request(self, request, check_asserts=True):
@@ -109,7 +111,7 @@ def send_request(self, request, check_asserts=True):
             outputs = response.getOutputs()
             assert len(outputs) == num_inputs
             for index, output in enumerate(outputs):
-                assert output.name == "input" + str(index)
+                assert output.name == ""
                 assert output.datatype == amdinfer.DataType.FP32
                 assert output.parameters.empty()
         return response
diff --git a/tests/workers/test_ptzendnn.py b/tests/workers/test_ptzendnn.py
index 8b2f761c0..8c01b2e4a 100644
--- a/tests/workers/test_ptzendnn.py
+++ b/tests/workers/test_ptzendnn.py
@@ -74,12 +74,11 @@ class TestPtZendnn:
     @staticmethod
     def get_config():
         model = "PtZendnn"
-        parameters = {
-            "model": amdinfer.testing.getPathToAsset("pt_resnet50"),
-            "input_size": 224,
-            "output_classes": 1000,
-            "batch_size": 8,
-        }
+        parameters = amdinfer.ParameterMap()
+        parameters.put("model", amdinfer.testing.getPathToAsset("pt_resnet50"))
+        parameters.put("input_size", 224)
+        parameters.put("output_classes", 1000)
+        parameters.put("batch_size", 8)
         return (model, parameters)
 
     def send_request(self, request, check_asserts=True):
@@ -111,7 +110,7 @@ def send_request(self, request, check_asserts=True):
             outputs = response.getOutputs()
             assert len(outputs) == num_inputs
             for index, output in enumerate(outputs):
-                assert output.name == "input" + str(index)
+                assert output.name == ""
                 assert output.datatype == amdinfer.DataType.FP32
                 assert output.parameters.empty()
         return response
diff --git a/tests/workers/test_resnet50_dpucadf8h.py b/tests/workers/test_resnet50_dpucadf8h.py
index 358bbe992..efcd3693d 100644
--- a/tests/workers/test_resnet50_dpucadf8h.py
+++ b/tests/workers/test_resnet50_dpucadf8h.py
@@ -22,6 +22,15 @@
 from helper import root_path, run_benchmark
 
 
+def construct_request(asTensor, batches=1):
+    image_path = amdinfer.testing.getPathToAsset("asset_dog-3619020_640.jpg")
+
+    # TODO(vishalk): AKS gives a segfault if batch != 4
+    images = [image_path] * batches
+
+    return amdinfer.ImageInferenceRequest(images, asTensor)
+
+
 @pytest.mark.extensions(["aks", "vitis"])
 @pytest.mark.fpgas("DPUCADF8H", 1)
 @pytest.mark.usefixtures("load")
@@ -33,10 +42,12 @@ class TestInferImageResNet50DPUCADF8H:
     @staticmethod
     def get_config():
         model = "resnet50"
-        parameters = {
-            "aks_graph_name": "resnet50",
-            "aks_graph": "${AKS_ROOT}/graph_zoo/graph_tf_resnet_v1_50_u200_u250_amdinfer.json",
-        }
+        parameters = amdinfer.ParameterMap()
+        parameters.put("aks_graph_name", "resnet50")
+        parameters.put(
+            "aks_graph",
+            "${AKS_ROOT}/graph_zoo/graph_tf_resnet_v1_50_u200_u250_amdinfer.json",
+        )
         return (model, parameters)
 
     def send_request(self, request, check_asserts=True):
@@ -69,7 +80,7 @@ def send_request(self, request, check_asserts=True):
             outputs = response.getOutputs()
             assert len(outputs) == num_inputs
             for index, output in enumerate(outputs):
-                assert output.name == "input" + str(index)
+                assert output.name == ""
                 assert output.datatype == amdinfer.DataType.UINT32
                 assert output.parameters.empty()
                 assert output.shape == [5, 1, 1]
@@ -78,31 +89,16 @@ def send_request(self, request, check_asserts=True):
                 np.testing.assert_almost_equal(gold_response_output, data, 2)
         return response
 
-    def construct_request(self, asTensor, batches=1):
-        image_path = amdinfer.testing.getPathToAsset("asset_dog-3619020_640.jpg")
-
-        # TODO(vishalk): AKS gives a segfault if batch != 4
-        images = [image_path] * batches
-
-        return amdinfer.ImageInferenceRequest(images, asTensor)
-
     def test_resnet50_dpucadf8h_0(self):
         """
         Send a request to resnet50 as tensor data
         """
-        request = self.construct_request(True)
-        self.send_request(request)
-
-    def test_resnet50_dpucadf8h_1(self):
-        """
-        Send a request to resnet50 as base64-encoded data
-        """
-        request = self.construct_request(False)
+        request = construct_request(True)
         self.send_request(request)
 
     @pytest.mark.benchmark(group="resnet50_dpucadf8h")
     def test_benchmark_resnet50_dpucadf8h_0(self, benchmark):
-        request = self.construct_request(True)
+        request = construct_request(True)
         options = {
             "model": self.model,
             "parameters": self.parameters,
@@ -113,9 +109,77 @@ def test_benchmark_resnet50_dpucadf8h_0(self, benchmark):
             benchmark, "resnet50_0", self.rest_client.modelInfer, request, **options
         )
 
+
+@pytest.mark.extensions(["aks", "vitis"])
+@pytest.mark.fpgas("DPUCADF8H", 1)
+@pytest.mark.usefixtures("load")
+class TestInferImageResNet50DPUCADF8H2:
+    """
+    Test the Resnet50 worker
+    """
+
+    @staticmethod
+    def get_config():
+        model = ["CPlusPlus", "resnet50"]
+        parameters = amdinfer.ParameterMap(["model"], ["base64_decode"])
+
+        aks_parameters = amdinfer.ParameterMap()
+        aks_parameters.put("aks_graph_name", "resnet50")
+        aks_parameters.put(
+            "aks_graph",
+            "${AKS_ROOT}/graph_zoo/graph_tf_resnet_v1_50_u200_u250_amdinfer.json",
+        )
+        return (model, [parameters, aks_parameters])
+
+    def send_request(self, request, check_asserts=True):
+        """
+        Sends the given request to the server and asserts common checks
+
+        Args:
+            request (dict): request to send to the server
+            image (np.ndarray): Golden image to check against
+            check_asserts (bool): Verify image against golden
+
+        Returns:
+            Response: Response as a dictionary
+        """
+
+        try:
+            response = self.rest_client.modelInfer(self.endpoint, request)
+        except amdinfer.ConnectionError:
+            pytest.fail(
+                "Connection to the amdinfer server ended without response!", False
+            )
+
+        num_inputs = len(request.getInputs())
+        gold_response_output = [259, 261, 154, 152, 204]
+
+        if check_asserts:
+            assert not response.isError(), response.getError()
+            assert response.id == ""
+            assert response.model == "resnet50"
+            outputs = response.getOutputs()
+            assert len(outputs) == num_inputs
+            for index, output in enumerate(outputs):
+                assert output.name == ""
+                assert output.datatype == amdinfer.DataType.UINT32
+                assert output.parameters.empty()
+                assert output.shape == [5, 1, 1]
+                data = output.getUint32Data()
+                assert len(data) == len(gold_response_output)
+                np.testing.assert_almost_equal(gold_response_output, data, 2)
+        return response
+
+    def test_resnet50_dpucadf8h_1(self):
+        """
+        Send a request to resnet50 as base64-encoded data
+        """
+        request = construct_request(False)
+        self.send_request(request)
+
     @pytest.mark.benchmark(group="resnet50_dpucadf8h")
     def test_benchmark_resnet50_dpucadf8h_1(self, benchmark):
-        request = self.construct_request(False)
+        request = construct_request(False)
         options = {
             "model": self.model,
             "parameters": self.parameters,
diff --git a/tests/workers/test_resnet50_stream.py b/tests/workers/test_resnet50_stream.py
index c1ad7c34a..8d1f50abb 100644
--- a/tests/workers/test_resnet50_stream.py
+++ b/tests/workers/test_resnet50_stream.py
@@ -34,7 +34,7 @@ class TestResnet50Stream:
     @staticmethod
     def get_config():
         model = "Resnet50Stream"
-        parameters = None
+        parameters = amdinfer.ParameterMap()
         return (model, parameters)
 
     def construct_request(self, requested_frames_count):
diff --git a/tests/workers/test_tfzendnn.py b/tests/workers/test_tfzendnn.py
index e6a2f67bc..f38a60fd4 100644
--- a/tests/workers/test_tfzendnn.py
+++ b/tests/workers/test_tfzendnn.py
@@ -68,16 +68,15 @@ class TestTfZendnn:
     @staticmethod
     def get_config():
         model = "TfZendnn"
-        parameters = {
-            "model": amdinfer.testing.getPathToAsset("tf_resnet50"),
-            "input_node": "input",
-            "output_node": "resnet_v1_50/predictions/Reshape_1",
-            "input_size": 224,
-            "output_classes": 1000,
-            "inter_op": 64,
-            "intra_op": 1,
-            "batch_size": 8,
-        }
+        parameters = amdinfer.ParameterMap()
+        parameters.put("model", amdinfer.testing.getPathToAsset("tf_resnet50"))
+        parameters.put("input_node", "input")
+        parameters.put("output_node", "resnet_v1_50/predictions/Reshape_1")
+        parameters.put("input_size", 224)
+        parameters.put("output_classes", 1000)
+        parameters.put("inter_op", 64)
+        parameters.put("intra_op", 1)
+        parameters.put("batch_size", 8)
         return (model, parameters)
 
     def send_request(self, request, check_asserts=True):
@@ -109,7 +108,7 @@ def send_request(self, request, check_asserts=True):
             outputs = response.getOutputs()
             assert len(outputs) == num_inputs
             for index, output in enumerate(outputs):
-                assert output.name == "input" + str(index)
+                assert output.name == ""
                 assert output.datatype == amdinfer.DataType.FP32
                 assert output.parameters.empty()
         return response
diff --git a/tests/workers/test_yolov3.py b/tests/workers/test_yolov3.py
index e6d816bee..e0257e1c4 100644
--- a/tests/workers/test_yolov3.py
+++ b/tests/workers/test_yolov3.py
@@ -35,12 +35,15 @@ class TestInferImageYoloV3DPUCADF8H:
 
     @staticmethod
     def get_config():
-        model = "AksDetect"
-        parameters = {
-            "aks_graph_name": "yolov3",
-            "aks_graph": "${AKS_ROOT}/graph_zoo/graph_yolov3_u200_u250_amdinfer.json",
-        }
-        return (model, parameters)
+        model = ["CPlusPlus", "AksDetect"]
+        parameters = amdinfer.ParameterMap(["model"], ["base64_decode"])
+
+        aks_parameters = amdinfer.ParameterMap()
+        aks_parameters.put("aks_graph_name", "yolov3")
+        aks_parameters.put(
+            "aks_graph", "${AKS_ROOT}/graph_zoo/graph_yolov3_u200_u250_amdinfer.json"
+        )
+        return (model, [parameters, aks_parameters])
 
     def send_request(self, request, check_asserts=True):
         """
@@ -87,12 +90,12 @@ def send_request(self, request, check_asserts=True):
             outputs = response.getOutputs()
             assert len(outputs) == num_inputs
             for index, output in enumerate(outputs):
-                assert output.name == "input" + str(index)
+                assert output.name == ""
                 assert output.datatype == amdinfer.DataType.FP32
                 assert output.parameters.empty()
                 data = output.getFp32Data()
                 num_boxes = int(len(data) / 6)
-                assert output.shape == [6, num_boxes]
+                assert output.shape == [num_boxes, 6]
                 assert len(data) == len(gold_response_output)
                 assert np.allclose(data, gold_response_output, 0.01, 0)
         return response
diff --git a/tests/workers/xmodel/test_resnet50.py b/tests/workers/xmodel/test_resnet50.py
index fe2d9a85f..51bb8a105 100644
--- a/tests/workers/xmodel/test_resnet50.py
+++ b/tests/workers/xmodel/test_resnet50.py
@@ -45,7 +45,7 @@ def validate(responses):
         outputs = response.getOutputs()
         assert len(outputs) == 1
         for index, output in enumerate(outputs):
-            assert output.name == "input" + str(index)
+            assert output.name == ""
             assert output.datatype == amdinfer.DataType.INT8
             assert output.parameters.empty()
             top_categories = postprocess(output, k)
@@ -63,7 +63,9 @@ class TestXmodel:
     @staticmethod
     def get_config():
         model = "Xmodel"
-        parameters = {"model": amdinfer.testing.getPathToAsset("u250_resnet50")}
+        parameters = amdinfer.ParameterMap(
+            ["model"], [amdinfer.testing.getPathToAsset("u250_resnet50")]
+        )
         return (model, parameters)
 
     def test_xmodel_0(self):