Repository: Xilinx/inference-server (language: python)

Issue:
Enqueuing requests with the C++ API requires the data to remain in scope after the call
Creating an InferenceRequestInput object to enqueue a request doesn't actually copy the data passed in by the user. This was an intentional choice to reduce copies. Instead, the copy occurs in the batcher when the request is batched. However, there is no way for the user to know when this actually occurs so there's no way to know how long the data that the user passed in with the request needs to remain in scope and valid.

Some possible solutions:

- Copy data when the request object is created. This is simple but adds a copy
- Nested futures? Not sure if possible but the idea is there are two things to wait on: when the data is batched (i.e. copied) and when the result is ready.
- Something else?
Enqueuing requests with the C++ API requires the data to remain in scope after the call
Creating an InferenceRequestInput object to enqueue a request doesn't actually copy the data passed in by the user. This was an intentional choice to reduce copies. Instead, the copy occurs in the batcher when the request is batched. However, there is no way for the user to know when this actually occurs so there's no way to know how long the data that the user passed in with the request needs to remain in scope and valid.

Some possible solutions:

- Copy data when the request object is created. This is simple but adds a copy
- Nested futures? Not sure if possible but the idea is there are two things to wait on: when the data is batched (i.e. copied) and when the result is ready.
- Something else?
Enqueuing requests with the C++ API requires the data to remain in scope after the call
Creating an InferenceRequestInput object to enqueue a request doesn't actually copy the data passed in by the user. This was an intentional choice to reduce copies. Instead, the copy occurs in the batcher when the request is batched. However, there is no way for the user to know when this actually occurs so there's no way to know how long the data that the user passed in with the request needs to remain in scope and valid.

Some possible solutions:

- Copy data when the request object is created. This is simple but adds a copy
- Nested futures? Not sure if possible but the idea is there are two things to wait on: when the data is batched (i.e. copied) and when the result is ready.
- Something else?


Discussion / hints:

Fix patch:
diff --git a/apps/mlcommons/src/main.cpp b/apps/mlcommons/src/main.cpp
index 43e6f86ab..d9fa8668f 100644
--- a/apps/mlcommons/src/main.cpp
+++ b/apps/mlcommons/src/main.cpp
@@ -232,8 +232,8 @@ int main(int argc, char* argv[]) {
   }
 
   // TODO(varunsh): expose to the outside
-  const uint16_t http_port = 8998;
-  const uint16_t grpc_port = 50'051;
+  [[maybe_unused]] const uint16_t http_port = 8998;
+  [[maybe_unused]] const uint16_t grpc_port = 50'051;
 
   if (client_id == "native") {
     if (remote_server) {
@@ -241,16 +241,20 @@ int main(int argc, char* argv[]) {
       return 1;
     }
     client = std::make_unique<amdinfer::NativeClient>(&(server.value()));
+#ifdef AMDINFER_ENABLE_HTTP
   } else if (client_id == "HTTP") {
     client = std::make_unique<amdinfer::HttpClient>(address);
     if (!remote_server) {
       server.value().startHttp(http_port);
     }
+#endif
+#ifdef AMDINFER_ENABLE_GRPC
   } else if (client_id == "gRPC") {
     client = std::make_unique<amdinfer::GrpcClient>(address);
     if (!remote_server) {
       server.value().startGrpc(grpc_port);
     }
+#endif
   } else {
     std::cerr << "Client must be one of 'native', 'HTTP', or 'gRPC'\n";
     return 1;
diff --git a/docs/architecture.rst b/docs/architecture.rst
index 2da495635..779f4ade3 100644
--- a/docs/architecture.rst
+++ b/docs/architecture.rst
@@ -45,11 +45,11 @@ Ingestion
 
 There are a number of ways to get data into the system for inference.
 In general, each protocol has a custom interface to the client and requires explicit handling in the Server to accept this initial request.
-After receiving the client's request in the Server, each supported protocols' handler packs the request into an implementation of an ``Interface`` object.
+After receiving the client's request in the Server, each supported protocols' handler packs the request into an implementation of a ``RequestContainer`` object.
 This virtual class is defined in the Server and provides the batcher with a standard way of interacting with data from different ingestion protocols.
 Since all protocols push an implementation of this class to the batcher, they are treated equally by the rest of the Server.
 The only protocol-specific logic applies at the end of the worker when it replies back to the client in a protocol-specific manner.
-This structure also enables easy extension of the Server to add new protocols by extending the ``Interface`` class.
+This structure also enables easy extension of the Server to add new protocols by extending the ``RequestContainer`` class.
 
 API
 ^^^
@@ -87,7 +87,7 @@ The various endpoints from the API are registered in the Drogon's HTTP controlle
 Drogon uses a configurable number of threads to run these request handlers.
 When a REST request is made to an endpoint, the request data and callback function are provided for the handler to process the request and then respond to the client.
 To avoid blocking the finite number of handler threads with potentially long-running inference requests, we use an asynchronous architecture in the handler.
-The received request is packed into an ``Interface`` object and pushed into a :github:`thread-safe lock-free multi producer/consumer queue <cameron314/concurrentqueue>` to go to the target worker's batcher.
+The received request is packed into a ``RequestContainer`` object and pushed into a :github:`thread-safe lock-free multi producer/consumer queue <cameron314/concurrentqueue>` to go to the target worker's batcher.
 The HTTP server code is in ``src/amdinfer/servers/http_server.*``.
 
 Drogon also provides a WebSocket server, which is currently used experimentally to run predictions on videos from certain workers.
@@ -103,7 +103,7 @@ As a result, using the C++ API will yield the highest performance of any ingesti
 
 The C++ API provides functions similar to the prediction API used in HTTP.
 The API lets users load workers and make inference requests.
-The inference request is packed into an ``Interface`` object and pushed to the target worker's batcher.
+The inference request is packed into a ``RequestContainer`` object and pushed to the target worker's batcher.
 An ``std::promise`` is returned to the user to retrieve the result.
 
 The public API is defined in ``include/amdinfer/clients/native.hpp`` and the implementation is in ``src/amdinfer/clients/native.cpp``.
@@ -129,7 +129,7 @@ The implementations of the batchers are in ``src/amdinfer/batching``.
 
 The base batcher class defines a common interface for all batcher implementations and has some basic common properties.
 Each batcher has two thread-safe queues (one for input and one for output), a configured batch size and a string identifying the worker group it's attached to.
-The batcher runs as a separate thread that monitors its input queue to process incoming ``Interface`` objects from all ingestion methods and pushes completed ``Batch`` objects on the output queue.
+The batcher runs as a separate thread that monitors its input queue to process incoming ``RequestContainer`` objects from all ingestion methods and pushes completed ``Batch`` objects on the output queue.
 Each batcher implementation defines a ``run()`` method that provides the logic with which the batcher produces a batch.
 A worker (and by extension, the worker group) specifies which batcher implementation should be used to prepare batches for it (as well as the batch size) and each worker group shares a set of batchers.
 This configuration is determined at compile-time and built into the definition of the worker.
@@ -144,9 +144,9 @@ In this case, the batcher's job is to take individual requests and move its data
 Batchers have some flexibility with how these batches are constructed, which is why multiple batcher implementations are possible and supported in the AMD Inference Server.
 For example, one batcher may allow partial batches to be pushed on after enough time whereas this may not be allowed by another batcher.
 
-Batchers use the ``Interface`` object's ``getRequest()`` method to help create batches.
+Batchers use the ``RequestContainer`` object's ``getRequest()`` method to help create batches.
 This method must be implemented by each interface and governs how, given some buffers and counters, the particular ingestion method's data should be converted to an ``InferenceRequest`` and its data is copied over to the buffers.
-THis method allows batchers to process all ingestion methods without knowing about the details of how the data may be stored internally in the ``Interface``.
+THis method allows batchers to process all ingestion methods without knowing about the details of how the data may be stored internally in the ``RequestContainer``.
 
 
 .. _architectureWorkers:
@@ -318,7 +318,7 @@ Manager
 -------
 
 The shared state of the AMD Inference Server is maintained by the Manager: the active workers, their buffer pools, the endpoints and load-time parameters associated with them and is visualized+ in :numref:`architecture_detail`.
-This information enables the ingestion protocols to query the Manager to retrieve a pointer to the correct batcher to use to push the ``Interface`` object to the right one corresponding to the targeted worker.
+This information enables the ingestion protocols to query the Manager to retrieve a pointer to the correct batcher to use to push the ``RequestContainer`` object to the right one corresponding to the targeted worker.
 To manage multiple versions of workers that may be running with different configurations, the Manager stores the load-time parameters, if any, and compares new parameters with ones its seen before to determine whether the newly loaded worker should be part of an existing worker group or a new one.
 In the case that it's assigned to an existing worker group, the previously allocated endpoint is returned to the client.
 If a new worker group is created, a new endpoint is reserved for this worker group and returned to the client.
diff --git a/include/amdinfer/core/predict_api.hpp b/include/amdinfer/core/predict_api.hpp
index efbf6596f..9ff68f99b 100644
--- a/include/amdinfer/core/predict_api.hpp
+++ b/include/amdinfer/core/predict_api.hpp
@@ -353,6 +353,14 @@ class InferenceRequest {
    * @param input an existing InferenceRequestInput object
    */
   void addInputTensor(InferenceRequestInput input);
+
+  /**
+   * @brief Set the data pointer for an input tensor, if it exists
+   *
+   * @param index index for the input tensor
+   * @param data pointer to assign to its data member
+   */
+  void setInputTensorData(size_t index, void *data);
   /**
    * @brief Adds a new output tensor to this request
    *
diff --git a/include/amdinfer/declarations.hpp b/include/amdinfer/declarations.hpp
index 65620f7dd..0df6634be 100644
--- a/include/amdinfer/declarations.hpp
+++ b/include/amdinfer/declarations.hpp
@@ -28,8 +28,8 @@ class Buffer;
 
 class Batch;
 
-class Interface;
-using InterfacePtr = std::unique_ptr<Interface>;
+class RequestContainer;
+using RequestContainerPtr = std::unique_ptr<RequestContainer>;
 class WorkerInfo;
 
 struct UpdateCommand;
diff --git a/src/amdinfer/batching/batcher.cpp b/src/amdinfer/batching/batcher.cpp
index 829927d3b..043803195 100644
--- a/src/amdinfer/batching/batcher.cpp
+++ b/src/amdinfer/batching/batcher.cpp
@@ -25,22 +25,22 @@
 #include <string>   // for string
 #include <utility>  // for move
 
-#include "amdinfer/buffers/buffer.hpp"         // IWYU pragma: keep
-#include "amdinfer/core/interface.hpp"         // IWYU pragma: keep
-#include "amdinfer/core/memory_pool/pool.hpp"  // for MemoryPool
-#include "amdinfer/core/worker_info.hpp"       // for WorkerInfo
-#include "amdinfer/observation/logging.hpp"    // for Logger, Loggers, Logger...
+#include "amdinfer/buffers/buffer.hpp"             // IWYU pragma: keep
+#include "amdinfer/core/memory_pool/pool.hpp"      // for MemoryPool
+#include "amdinfer/core/predict_api_internal.hpp"  // for InferenceRequestInput
+#include "amdinfer/core/worker_info.hpp"           // for WorkerInfo
+#include "amdinfer/observation/logging.hpp"  // for Logger, Loggers, Logger...
 
 namespace amdinfer {
 
 /**
- * @brief The C++ Interface class encapsulates incoming requests from the C++
- * API to the batcher.
+ * @brief The C++ RequestContainer class encapsulates incoming requests from the
+ * C++ API to the batcher.
  *
  */
 
 Batcher::Batcher(MemoryPool* pool) : pool_(pool) {
-  this->input_queue_ = std::make_shared<BlockingQueue<InterfacePtr>>();
+  this->input_queue_ = std::make_shared<BlockingQueue<RequestContainerPtr>>();
   this->output_queue_ = std::make_shared<BatchPtrQueue>();
   this->status_ = BatcherStatus::New;
 #ifdef AMDINFER_ENABLE_LOGGING
@@ -79,13 +79,13 @@ void Batcher::setName(const std::string& name) { this->model_ = name; }
 
 std::string Batcher::getName() const { return this->model_; }
 
-BlockingQueue<InterfacePtr>* Batcher::getInputQueue() {
+BlockingQueue<RequestContainerPtr>* Batcher::getInputQueue() {
   return this->input_queue_.get();
 }
 
 BatchPtrQueue* Batcher::getOutputQueue() { return this->output_queue_.get(); }
 
-void Batcher::enqueue(InterfacePtr request) const {
+void Batcher::enqueue(RequestContainerPtr request) const {
   this->input_queue_->enqueue(std::move(request));
 }
 
diff --git a/src/amdinfer/batching/batcher.hpp b/src/amdinfer/batching/batcher.hpp
index 21faca3ce..03d2874a8 100644
--- a/src/amdinfer/batching/batcher.hpp
+++ b/src/amdinfer/batching/batcher.hpp
@@ -95,7 +95,7 @@ class Batcher {
   [[nodiscard]] std::string getName() const;
 
   /// Get the batcher's input queue (used to enqueue new requests)
-  BlockingQueue<InterfacePtr>* getInputQueue();
+  BlockingQueue<RequestContainerPtr>* getInputQueue();
   /// Get the batcher's output queue (used to push batches to the worker group)
   BatchPtrQueue* getOutputQueue();
 
@@ -108,7 +108,7 @@ class Batcher {
    *
    * @param request
    */
-  void enqueue(InterfacePtr request) const;
+  void enqueue(RequestContainerPtr request) const;
 
   /// End the batcher
   void end();
@@ -119,7 +119,7 @@ class Batcher {
 #endif
 
   size_t batch_size_ = 1;
-  std::shared_ptr<BlockingQueue<InterfacePtr>> input_queue_;
+  std::shared_ptr<BlockingQueue<RequestContainerPtr>> input_queue_;
   std::shared_ptr<BatchPtrQueue> output_queue_;
   std::thread thread_;
   std::string model_;
@@ -129,7 +129,7 @@ class Batcher {
  private:
   /**
    * @brief The doRun method defines the exact process by which the batcher
-   * consumes incoming Interface objects and uses them to create batches.
+   * consumes incoming RequestContainer objects and uses them to create batches.
    *
    * @param allocators vector of allocators that may be used to get memory
    */
diff --git a/src/amdinfer/batching/hard.cpp b/src/amdinfer/batching/hard.cpp
index 05a986ccc..7f6fdc823 100644
--- a/src/amdinfer/batching/hard.cpp
+++ b/src/amdinfer/batching/hard.cpp
@@ -26,16 +26,17 @@
 #include <utility>  // for move
 #include <vector>   // for vector
 
+#include "amdinfer/buffers/cpu.hpp"            // for CpuBuffer
 #include "amdinfer/build_options.hpp"          // for AMDINFER_ENABLE_METRICS
 #include "amdinfer/core/exceptions.hpp"        // for invalid_argument
-#include "amdinfer/core/interface.hpp"         // for Interface
 #include "amdinfer/core/memory_pool/pool.hpp"  // for MemoryPool
-#include "amdinfer/core/worker_info.hpp"       // for WorkerInfo
-#include "amdinfer/declarations.hpp"           // for InterfacePtr
-#include "amdinfer/observation/metrics.hpp"    // for Metrics, MetricCounterIDs
-#include "amdinfer/observation/tracing.hpp"    // for Trace
-#include "amdinfer/util/queue.hpp"             // for BlockingConcurrentQueue
-#include "amdinfer/util/thread.hpp"            // for setThreadName
+#include "amdinfer/core/predict_api_internal.hpp"  // for InferenceRequestInput
+#include "amdinfer/core/worker_info.hpp"           // for WorkerInfo
+#include "amdinfer/declarations.hpp"               // for RequestContainerPtr
+#include "amdinfer/observation/metrics.hpp"  // for Metrics, MetricCounterIDs
+#include "amdinfer/observation/tracing.hpp"  // for Trace
+#include "amdinfer/util/queue.hpp"           // for BlockingConcurrentQueue
+#include "amdinfer/util/thread.hpp"          // for setThreadName
 
 // IWYU pragma: no_forward_declare amdinfer::Buffer
 
@@ -44,7 +45,7 @@ namespace amdinfer {
 void HardBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
   auto thread_name = "batch" + this->getName();
   util::setThreadName(thread_name);
-  InterfacePtr req;
+  RequestContainerPtr req;
   bool run = true;
 
   while (run) {
@@ -66,7 +67,7 @@ void HardBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
       }
 
 #ifdef AMDINFER_ENABLE_TRACING
-      auto trace = req->getTrace();
+      auto& trace = req->trace;
       trace->startSpan("hard_batcher");
 #endif
 
@@ -75,16 +76,16 @@ void HardBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
         MetricCounterIDs::PipelineIngressBatcher);
 #endif
 
-      auto input_size = req->getInputSize();
+      auto request = req->request;
+      const auto& inputs = request->getInputs();
+      auto input_size = inputs.size();
       if (input_size == 0) {
-        auto error = invalid_argument("Input size is zero");
-        req->errorHandler(error);
+        request->runCallbackError("Input size is zero");
         continue;
       }
 
       if (first_request) {
-        auto input_sizes = req->getInputSizes();
-        input_buffers.reserve(input_sizes.size());
+        input_buffers.reserve(input_size);
         // auto output_sizes = req->getOutputSizes();
         // TODO(varunsh): the spec does not require the request to have outputs
         // additionally, the output size could be variable so this should be
@@ -93,9 +94,10 @@ void HardBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
         // std::vector<BufferPtr> output_buffers;
         // output_buffers.reserve(output_sizes.size());
         // std::vector<size_t> output_offset(output_buffers.size(), 0);
-        for (const auto& tensor_size : input_sizes) {
-          input_buffers.push_back(
-            pool_->get(allocators, tensor_size * batch_size_));
+        for (const auto& input : inputs) {
+          input_buffers.push_back(pool_->get(
+            allocators,
+            input.getSize() * input.getDatatype().size() * batch_size_));
         }
         // for(const auto& tensor_size : output_sizes) {
         //   output_buffers.push_back(pool_->get(allocators, tensor_size));
@@ -108,23 +110,29 @@ void HardBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
       auto raw_outputs = batch->getRawOutputBuffers();
 
       auto old_input_offset = input_offset;
-      // auto old_output_offset = output_offset;
-      auto new_req =
-        req->getRequest(raw_inputs, input_offset, raw_outputs, output_offset);
-      if (new_req == nullptr) {
-        input_offset = old_input_offset;
-        // output_offset = old_output_offset;
-      } else {
-        batch->addRequest(new_req);
-        batch_size++;
+      for (auto i = 0U; i < input_size; ++i) {
+        const auto& input = inputs[i];
+        auto* raw_input = raw_inputs[i];
+        auto& offset = input_offset[i];
+
+        auto new_offset =
+          raw_input->write(input.getData(), offset,
+                           input.getSize() * input.getDatatype().size());
+        pool_->put(
+          std::make_unique<CpuBuffer>(input.getData(), MemoryAllocators::Cpu));
+        request->setInputTensorData(i, raw_input->data(offset));
+        offset = new_offset;
+      }
+
+      batch->addRequest(request);
+      batch_size++;
 #ifdef AMDINFER_ENABLE_TRACING
-        trace->endSpan();
-        batch->addTrace(std::move(trace));
+      trace->endSpan();
+      batch->addTrace(std::move(trace));
 #endif
 #ifdef AMDINFER_ENABLE_METRICS
-        batch->addTime(req->getTime());
+      batch->addTime(req->start_time);
 #endif
-      }
       first_request = false;
     } while (batch_size % this->batch_size_ != 0);
 
diff --git a/src/amdinfer/batching/soft.cpp b/src/amdinfer/batching/soft.cpp
index 64fa0a67b..19347034e 100644
--- a/src/amdinfer/batching/soft.cpp
+++ b/src/amdinfer/batching/soft.cpp
@@ -29,13 +29,14 @@
 #include <utility>    // for move
 #include <vector>     // for vector
 
+#include "amdinfer/buffers/cpu.hpp"      // for CpuBuffer
 #include "amdinfer/build_options.hpp"    // for AMDINFER_ENABLE_METRICS
 #include "amdinfer/core/exceptions.hpp"  // for invalid_argument
-#include "amdinfer/core/interface.hpp"   // for Interface
 #include "amdinfer/core/memory_pool/pool.hpp"
-#include "amdinfer/core/parameters.hpp"  // for ParameterMap
+#include "amdinfer/core/parameters.hpp"            // for ParameterMap
+#include "amdinfer/core/predict_api_internal.hpp"  // for InferenceRequestInput
 #include "amdinfer/core/worker_info.hpp"
-#include "amdinfer/declarations.hpp"         // for InterfacePtr
+#include "amdinfer/declarations.hpp"         // for RequestContainerPtr
 #include "amdinfer/observation/logging.hpp"  // for Logger, AMDINFER_LOG_DEBUG
 #include "amdinfer/observation/metrics.hpp"  // for Metrics, MetricCounterIDs
 #include "amdinfer/observation/tracing.hpp"  // for Trace
@@ -83,7 +84,7 @@ void SoftBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
     util::Timer timer{true};
 
     do {
-      InterfacePtr req;
+      RequestContainerPtr req;
       if (first_request) {
         // wait for the first request
         this->input_queue_->wait_dequeue(req);
@@ -107,17 +108,16 @@ void SoftBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
         break;
       }
 
-      auto input_size = req->getInputSize();
+      auto request = req->request;
+      auto& inputs = request->getInputs();
+      auto input_size = inputs.size();
       if (input_size == 0) {
-        auto error = invalid_argument("Input size is zero");
-        req->errorHandler(error);
+        request->runCallbackError("Input size is zero");
         continue;
       }
 
       if (first_request) {
-        auto input_sizes = req->getInputSizes();
-
-        input_buffers.reserve(input_sizes.size());
+        input_buffers.reserve(input_size);
         // auto output_sizes = req->getOutputSizes();
         // TODO(varunsh): the spec does not require the request to have outputs
         // additionally, the output size could be variable so this should be
@@ -126,9 +126,10 @@ void SoftBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
         // std::vector<BufferPtr> output_buffers;
         // output_buffers.reserve(output_sizes.size());
         // std::vector<size_t> output_offset(output_buffers.size(), 0);
-        for (const auto& tensor_size : input_sizes) {
-          input_buffers.push_back(
-            pool_->get(allocators, tensor_size * batch_size_));
+        for (const auto& input : inputs) {
+          input_buffers.push_back(pool_->get(
+            allocators,
+            input.getSize() * input.getDatatype().size() * batch_size_));
         }
         // for(const auto& tensor_size : output_sizes) {
         //   output_buffers.push_back(pool_->get(allocators, tensor_size));
@@ -141,7 +142,7 @@ void SoftBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
       auto raw_outputs = batch->getRawOutputBuffers();
 
 #ifdef AMDINFER_ENABLE_TRACING
-      auto trace = req->getTrace();
+      auto& trace = req->trace;
       trace->startSpan("soft_batcher");
 #endif
 
@@ -151,25 +152,30 @@ void SoftBatcher::doRun(const std::vector<MemoryAllocators>& allocators) {
 #endif
 
       auto old_input_offset = input_offset;
-      // auto old_output_offset = output_offset;
-      auto new_req =
-        req->getRequest(raw_inputs, input_offset, raw_outputs, output_offset);
-      if (new_req == nullptr) {
-        AMDINFER_LOG_DEBUG(logger, "Making request for " + this->model_ +
-                                     " failed. Reverting buffers.");
-        input_offset = old_input_offset;
-        // output_offset = old_output_offset;
-      } else {
-        batch->addRequest(new_req);
-        batch_size++;
+
+      for (auto i = 0U; i < input_size; ++i) {
+        const auto& input = inputs[i];
+        auto* raw_input = raw_inputs[i];
+        auto& offset = input_offset[i];
+
+        auto new_offset =
+          raw_input->write(input.getData(), offset,
+                           input.getSize() * input.getDatatype().size());
+        pool_->put(
+          std::make_unique<CpuBuffer>(input.getData(), MemoryAllocators::Cpu));
+        request->setInputTensorData(i, raw_input->data(offset));
+        offset = new_offset;
+      }
+
+      batch->addRequest(request);
+      batch_size++;
 #ifdef AMDINFER_ENABLE_TRACING
-        trace->endSpan();
-        batch->addTrace(std::move(trace));
+      trace->endSpan();
+      batch->addTrace(std::move(trace));
 #endif
 #ifdef AMDINFER_ENABLE_METRICS
-        batch->addTime(req->getTime());
+      batch->addTime(req->start_time);
 #endif
-      }
       first_request = false;
     } while (batch_size % this->batch_size_ != 0 && run);
 
diff --git a/src/amdinfer/clients/CMakeLists.txt b/src/amdinfer/clients/CMakeLists.txt
index 1cbce15fd..cbe6a92a2 100644
--- a/src/amdinfer/clients/CMakeLists.txt
+++ b/src/amdinfer/clients/CMakeLists.txt
@@ -13,12 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-list(
-  APPEND targets
-         client
-         native
-         native_internal
-)
+list(APPEND targets client native)
 
 if(${AMDINFER_ENABLE_HTTP})
   set(targets ${targets} http http_internal websocket)
diff --git a/src/amdinfer/clients/http_internal.cpp b/src/amdinfer/clients/http_internal.cpp
index b34e53108..c019f847e 100644
--- a/src/amdinfer/clients/http_internal.cpp
+++ b/src/amdinfer/clients/http_internal.cpp
@@ -39,10 +39,8 @@
 #include "amdinfer/buffers/buffer.hpp"             // for Buffer
 #include "amdinfer/core/data_types.hpp"            // for DataType, mapTypeToStr
 #include "amdinfer/core/exceptions.hpp"            // invalid_argument
-#include "amdinfer/core/interface.hpp"             // for InterfaceType, Inte...
 #include "amdinfer/core/predict_api_internal.hpp"  // for InferenceRequestOutput
 #include "amdinfer/observation/logging.hpp"        // for Logger
-#include "amdinfer/util/compression.hpp"           // for zDecompress
 #include "amdinfer/util/traits.hpp"                // IWYU pragma: keep
 #include "half/half.hpp"                           // for half
 
@@ -92,29 +90,6 @@ Json::Value mapParametersToJson(ParameterMap *parameters) {
   return json;
 }
 
-template <typename T>
-constexpr auto jsonValueToType(const Json::Value &datum) {
-  if constexpr (std::is_same_v<T, bool>) {
-    return datum.asBool();
-  } else if constexpr (util::is_any_v<T, uint8_t, uint16_t, uint32_t>) {
-    return datum.asUInt();
-  } else if constexpr (std::is_same_v<T, uint64_t>) {
-    return datum.asUInt64();
-  } else if constexpr (util::is_any_v<T, int8_t, int16_t, int32_t>) {
-    return datum.asInt();
-  } else if constexpr (std::is_same_v<T, int64_t>) {
-    return datum.asInt64();
-  } else if constexpr (util::is_any_v<T, fp16, float>) {
-    return datum.asFloat();
-  } else if constexpr (std::is_same_v<T, double>) {
-    return datum.asDouble();
-  } else if constexpr (std::is_same_v<T, char>) {
-    return datum.asString();
-  } else {
-    static_assert(!sizeof(T), "Invalid type to SetOutputData");
-  }
-}
-
 struct SetOutputData {
   template <typename T>
   void operator()(const Json::Value &json,
@@ -167,38 +142,6 @@ InferenceResponse mapJsonToResponse(Json::Value *json) {
   return response;
 }
 
-struct SetInputData {
-  template <typename T>
-  void operator()(Json::Value *json, void *src_data, size_t src_size) const {
-    auto *data = static_cast<T *>(src_data);
-    if constexpr (std::is_same_v<T, char>) {
-      std::string str{data, src_size};
-      json->append(str);
-    } else {
-      // NOLINTNEXTLINE(readability-identifier-naming)
-      constexpr auto getData = [](const T *data_ptr, size_t index) {
-        if constexpr (std::is_same_v<T, uint64_t>) {
-          return static_cast<Json::UInt64>(data_ptr[index]);
-        } else if constexpr (std::is_same_v<T, int64_t>) {
-          return static_cast<Json::Int64>(data_ptr[index]);
-        } else if constexpr (util::is_any_v<T, bool, uint8_t, uint16_t,
-                                            uint32_t, int8_t, int16_t, int32_t,
-                                            float, double>) {
-          return data_ptr[index];
-        } else if constexpr (util::is_any_v<T, fp16>) {
-          return half_float::half_cast<float>(data_ptr[index]);
-        } else {
-          static_assert(!sizeof(T), "Invalid type to SetInputData");
-        }
-      };
-
-      for (auto i = 0U; i < src_size; ++i) {
-        json->append(getData(data, i));
-      }
-    }
-  }
-};
-
 Json::Value mapRequestToJson(const InferenceRequest &request) {
   Json::Value json;
   json["id"] = request.getID();
@@ -230,291 +173,8 @@ Json::Value mapRequestToJson(const InferenceRequest &request) {
   return json;
 }
 
-struct WriteData {
-  template <typename T>
-  size_t operator()(Buffer *buffer, const Json::Value &value,
-                    size_t offset) const {
-    if constexpr (std::is_same_v<T, char>) {
-      return buffer->write(jsonValueToType<T>(value), offset);
-    } else {
-      return buffer->write(static_cast<T>(jsonValueToType<T>(value)), offset);
-    }
-  }
-};
-
-template <>
-class InferenceRequestInputBuilder<std::shared_ptr<Json::Value>> {
- public:
-  static InferenceRequestInput build(std::shared_ptr<Json::Value> const &req,
-                                     Buffer *input_buffer, size_t offset) {
-    InferenceRequestInput input;
-#ifdef AMDINFER_ENABLE_LOGGING
-    Logger logger{Loggers::Server};
-#endif
-    input.data_ = input_buffer->data(0);
-
-    if (!req->isMember("name")) {
-      throw invalid_argument("No 'name' key present in request input");
-    }
-    input.name_ = req->get("name", "").asString();
-    if (!req->isMember("shape")) {
-      throw invalid_argument("No 'shape' key present in request input");
-    }
-    auto shape = req->get("shape", Json::arrayValue);
-    for (auto const &i : shape) {
-      if (!i.isUInt64()) {
-        throw invalid_argument("'shape' must be specified by uint64 elements");
-      }
-      input.shape_.push_back(i.asUInt64());
-    }
-    if (!req->isMember("datatype")) {
-      throw invalid_argument("No 'datatype' key present in request input");
-    }
-    std::string data_type_str = req->get("datatype", "").asString();
-    input.data_type_ = DataType(data_type_str.c_str());
-    if (req->isMember("parameters")) {
-      auto parameters = req->get("parameters", Json::objectValue);
-      input.parameters_ = mapJsonToParameters(parameters);
-    } else {
-      input.parameters_ = std::make_unique<ParameterMap>();
-    }
-    if (!req->isMember("data")) {
-      throw invalid_argument("No 'data' key present in request input");
-    }
-    auto data = req->get("data", Json::arrayValue);
-    try {
-      for (auto const &i : data) {
-        offset = switchOverTypes(WriteData(), input.data_type_, input_buffer, i,
-                                 offset);
-      }
-    } catch (const Json::LogicError &) {
-      throw invalid_argument(
-        "Could not convert some data to the provided data type");
-    }
-    return input;
-  }
-};
-
-using InputBuilder = InferenceRequestInputBuilder<std::shared_ptr<Json::Value>>;
-
-template <>
-class InferenceRequestOutputBuilder<std::shared_ptr<Json::Value>> {
- public:
-  static InferenceRequestOutput build(const std::shared_ptr<Json::Value> &req) {
-    InferenceRequestOutput output;
-    output.data_ = nullptr;
-    output.name_ = req->get("name", "").asString();
-    if (req->isMember("parameters")) {
-      auto parameters = req->get("parameters", Json::objectValue);
-      output.parameters_ = mapJsonToParameters(parameters);
-    } else {
-      output.parameters_ = std::make_unique<ParameterMap>();
-    }
-    return output;
-  }
-};
-
-using OutputBuilder =
-  InferenceRequestOutputBuilder<std::shared_ptr<Json::Value>>;
-
-InferenceRequestPtr RequestBuilder::build(
-  const std::shared_ptr<Json::Value> &req, const BufferRawPtrs &input_buffers,
-  std::vector<size_t> &input_offsets, const BufferRawPtrs &output_buffers,
-  std::vector<size_t> &output_offsets) {
-  auto request = std::make_shared<InferenceRequest>();
-
-  if (req->isMember("id")) {
-    request->id_ = req->get("id", "").asString();
-  } else {
-    request->id_ = "";
-  }
-  if (req->isMember("parameters")) {
-    auto parameters = req->get("parameters", Json::objectValue);
-    request->parameters_ = mapJsonToParameters(parameters);
-  } else {
-    request->parameters_ = std::make_unique<ParameterMap>();
-  }
-
-  if (!req->isMember("inputs")) {
-    throw invalid_argument("No 'inputs' key present in request");
-  }
-  auto inputs = req->get("inputs", Json::arrayValue);
-  if (!inputs.isArray()) {
-    throw invalid_argument("'inputs' is not an array");
-  }
-
-  request->callback_ = nullptr;
-
-  assert(input_buffers.size() == inputs.size());
-  const auto input_num = input_buffers.size();
-  for (auto i = 0U; i < input_num; ++i) {
-    const auto &json_input = inputs[i];
-    if (!json_input.isObject()) {
-      throw invalid_argument("At least one element in 'inputs' is not an obj");
-    }
-    auto *buffer = input_buffers[i];
-    auto &offset = input_offsets[i];
-    auto input = InputBuilder::build(std::make_shared<Json::Value>(json_input),
-                                     buffer, offset);
-    offset += (input.getSize() * input.getDatatype().size());
-
-    request->inputs_.push_back(std::move(input));
-  }
-
-  // TODO(varunsh): output_offset is currently ignored! The size of the output
-  // needs to come from the worker but we have no such information.
-  const auto output_num = output_buffers.size();
-  if (req->isMember("outputs")) {
-    auto outputs = req->get("outputs", Json::arrayValue);
-    assert(output_buffers.size() == outputs.size());
-    for (auto i = 0U; i < output_num; ++i) {
-      const auto &json_output = outputs[i];
-      auto *buffer = output_buffers[i];
-      auto &offset = output_offsets[i];
-
-      auto output =
-        OutputBuilder::build(std::make_shared<Json::Value>(json_output));
-      output.setData(static_cast<std::byte *>(buffer->data(offset)));
-      request->outputs_.push_back(std::move(output));
-      // output += request->outputs_.back().getSize(); // see TODO
-    }
-  } else {
-    for (auto i = 0U; i < output_num; ++i) {
-      auto *buffer = output_buffers[i];
-      const auto &offset = output_offsets[i];
-
-      request->outputs_.emplace_back();
-      request->outputs_.back().setData(
-        static_cast<std::byte *>(buffer->data(offset)));
-    }
-  }
-
-  return request;
-}
-
-using RequestBuilder = InferenceRequestBuilder<std::shared_ptr<Json::Value>>;
-
 using drogon::HttpStatusCode;
 
-drogon::HttpResponsePtr errorHttpResponse(const std::string &error,
-                                          int status_code) {
-  Json::Value ret;
-  ret["error"] = error.data();
-  auto resp = drogon::HttpResponse::newHttpJsonResponse(ret);
-  resp->setStatusCode(static_cast<drogon::HttpStatusCode>(status_code));
-  return resp;
-}
-
-std::shared_ptr<Json::Value> parseJson(const drogon::HttpRequest *req) {
-#ifdef AMDINFER_ENABLE_LOGGING
-  Logger logger{Loggers::Server};
-#endif
-
-  // attempt to get the JSON object directly first
-  const auto &json_obj = req->getJsonObject();
-  if (json_obj != nullptr) {
-    return json_obj;
-  }
-
-  AMDINFER_LOG_DEBUG(logger, "Failed to get JSON data directly");
-
-  // if it's not valid, then we need to attempt to parse the body
-  auto root = std::make_shared<Json::Value>();
-
-  std::string errors;
-  Json::CharReaderBuilder builder;
-  Json::CharReader *reader = builder.newCharReader();
-  auto body = req->getBody();
-  bool success =
-    reader->parse(body.data(), body.data() + body.size(), root.get(), &errors);
-  if (success) {
-    return root;
-  }
-
-  AMDINFER_LOG_DEBUG(logger, "Failed to interpret body as JSON data");
-
-  // if it's still not valid, attempt to uncompress the body and convert to JSON
-  auto body_decompress = util::zDecompress(body.data(), body.length());
-  success = reader->parse(body_decompress.data(),
-                          body_decompress.data() + body_decompress.size(),
-                          root.get(), &errors);
-  if (success) {
-    return root;
-  }
-
-  throw invalid_argument("Failed to interpret request body as JSON");
-}
-
-DrogonHttp::DrogonHttp(const drogon::HttpRequestPtr &req,
-                       DrogonCallback callback)
-  : req_(req), callback_(std::move(callback)) {
-  this->type_ = InterfaceType::DrogonHttp;
-  this->json_ = parseJson(req.get());
-}
-
-size_t DrogonHttp::getInputSize() {
-  if (!this->json_->isMember("inputs")) {
-    throw invalid_argument("No 'inputs' key present in request");
-  }
-  auto inputs = this->json_->get("inputs", Json::arrayValue);
-  if (!inputs.isArray()) {
-    throw invalid_argument("'inputs' is not an array");
-  }
-
-  return inputs.size();
-}
-
-std::vector<size_t> DrogonHttp::getInputSizes() const {
-  std::vector<size_t> sizes;
-
-  if (!this->json_->isMember("inputs")) {
-    throw invalid_argument("No 'inputs' key present in request");
-  }
-  auto inputs = this->json_->get("inputs", Json::arrayValue);
-  if (!inputs.isArray()) {
-    throw invalid_argument("'inputs' is not an array");
-  }
-
-  for (const auto &tensor : inputs) {
-    // using asCString() doesn't work -> the string is empty
-    const auto raw_type = tensor.get("datatype", "UNKNOWN").asString();
-    auto datatype = DataType(raw_type.c_str());
-
-    const auto shape = tensor.get("shape", Json::arrayValue);
-    size_t size = 1;
-    for (const auto &index : shape) {
-      size *= index.asInt64();
-    }
-    sizes.push_back(size * datatype.size());
-  }
-  return sizes;
-}
-
-Json::Value parseResponse(InferenceResponse response) {
-  Json::Value ret;
-  ret["model_name"] = response.getModel();
-  ret["outputs"] = Json::arrayValue;
-  ret["id"] = response.getID();
-  auto outputs = response.getOutputs();
-  for (const InferenceResponseOutput &output : outputs) {
-    Json::Value json_output;
-    json_output["name"] = output.getName();
-    json_output["parameters"] = Json::objectValue;
-    json_output["data"] = Json::arrayValue;
-    json_output["shape"] = Json::arrayValue;
-    json_output["datatype"] = output.getDatatype().str();
-    const auto &shape = output.getShape();
-    for (const size_t &index : shape) {
-      json_output["shape"].append(static_cast<Json::UInt>(index));
-    }
-
-    switchOverTypes(SetInputData(), output.getDatatype(),
-                    &(json_output["data"]), output.getData(), output.getSize());
-    ret["outputs"].append(json_output);
-  }
-  return ret;
-}
-
 #ifdef AMDINFER_ENABLE_TRACING
 void propagate(drogon::HttpResponse *resp, const StringMap &context) {
   for (const auto &[key, value] : context) {
@@ -523,54 +183,6 @@ void propagate(drogon::HttpResponse *resp, const StringMap &context) {
 }
 #endif
 
-std::shared_ptr<InferenceRequest> DrogonHttp::getRequest(
-  const BufferRawPtrs &input_buffers, std::vector<size_t> &input_offsets,
-  const BufferRawPtrs &output_buffers, std::vector<size_t> &output_offsets) {
-#ifdef AMDINFER_ENABLE_LOGGING
-  const auto &logger = this->getLogger();
-#endif
-  try {
-    auto request =
-      RequestBuilder::build(this->json_, input_buffers, input_offsets,
-                            output_buffers, output_offsets);
-    Callback callback = [callback = std::move(this->callback_)](
-                          const InferenceResponse &response) {
-      drogon::HttpResponsePtr resp;
-      if (response.isError()) {
-        resp = errorHttpResponse(response.getError(),
-                                 HttpStatusCode::k400BadRequest);
-      } else {
-        try {
-          Json::Value ret = parseResponse(response);
-          resp = drogon::HttpResponse::newHttpJsonResponse(ret);
-        } catch (const invalid_argument &e) {
-          resp = errorHttpResponse(e.what(), HttpStatusCode::k400BadRequest);
-        }
-      }
-#ifdef AMDINFER_ENABLE_TRACING
-      const auto &context = response.getContext();
-      propagate(resp.get(), context);
-#endif
-      callback(resp);
-    };
-    request->setCallback(std::move(callback));
-    return request;
-  } catch (const invalid_argument &e) {
-    AMDINFER_LOG_INFO(logger, e.what());
-    this->callback_(
-      errorHttpResponse(e.what(), HttpStatusCode::k400BadRequest));
-    return nullptr;
-  }
-}
-
-void DrogonHttp::errorHandler(const std::exception &e) {
-#ifdef AMDINFER_ENABLE_LOGGING
-  [[maybe_unused]] const auto &logger = this->getLogger();
-  AMDINFER_LOG_DEBUG(logger, e.what());
-#endif
-  this->callback_(errorHttpResponse(e.what(), HttpStatusCode::k400BadRequest));
-}
-
 Json::Value modelMetadataTensorToJson(const ModelMetadataTensor &metadata) {
   Json::Value ret;
   ret["name"] = metadata.getName();
diff --git a/src/amdinfer/clients/http_internal.hpp b/src/amdinfer/clients/http_internal.hpp
index 8b66494ad..600c747a5 100644
--- a/src/amdinfer/clients/http_internal.hpp
+++ b/src/amdinfer/clients/http_internal.hpp
@@ -33,10 +33,10 @@
 #include <vector>      // for vector
 
 #include "amdinfer/build_options.hpp"              // for AMDINFER_ENABLE_TRA...
-#include "amdinfer/core/interface.hpp"             // for Interface
 #include "amdinfer/core/parameters.hpp"            // for ParameterMap (ptr ...
 #include "amdinfer/core/predict_api_internal.hpp"  // for InferenceRequestBui...
 #include "amdinfer/declarations.hpp"               // for BufferRawPtrs, Infe...
+#include "amdinfer/util/traits.hpp"                // for is_any_v
 
 namespace amdinfer {
 
@@ -52,62 +52,64 @@ Json::Value mapParametersToJson(ParameterMap *parameters);
 InferenceResponse mapJsonToResponse(Json::Value *json);
 Json::Value mapRequestToJson(const InferenceRequest &request);
 
-// class InferenceRequestOutputBuilder {
-//  public:
-//   static InferenceRequestOutput fromJson(
-//     std::shared_ptr<Json::Value> const &req);
-// };
-
-template <>
-class InferenceRequestBuilder<std::shared_ptr<Json::Value>> {
- public:
-  static InferenceRequestPtr build(const std::shared_ptr<Json::Value> &req,
-                                   const BufferRawPtrs &input_buffers,
-                                   std::vector<size_t> &input_offsets,
-                                   const BufferRawPtrs &output_buffers,
-                                   std::vector<size_t> &output_offsets);
-};
-
-using RequestBuilder = InferenceRequestBuilder<std::shared_ptr<Json::Value>>;
-
 #ifdef AMDINFER_ENABLE_TRACING
 void propagate(drogon::HttpResponse *resp, const StringMap &context);
 #endif
 
-using DrogonCallback = std::function<void(const drogon::HttpResponsePtr &)>;
-
-/**
- * @brief The DrogonHttp Interface class encapsulates incoming requests from
- * Drogon's HTTP interface to the batcher.
- *
- */
-class DrogonHttp : public Interface {
- public:
-  /**
-   * @brief Construct a new DrogonHttp object
-   *
-   * @param req
-   * @param callback
-   */
-  DrogonHttp(const drogon::HttpRequestPtr &req, DrogonCallback callback);
-
-  std::shared_ptr<InferenceRequest> getRequest(
-    const BufferRawPtrs &input_buffers, std::vector<size_t> &input_offsets,
-    const BufferRawPtrs &output_buffers,
-    std::vector<size_t> &output_offsets) override;
-
-  size_t getInputSize() override;
-  std::vector<size_t> getInputSizes() const override;
-  void errorHandler(const std::exception &e) override;
-
- private:
-  drogon::HttpRequestPtr req_;
-  DrogonCallback callback_;
-  std::shared_ptr<Json::Value> json_;
+struct SetInputData {
+  template <typename T>
+  void operator()(Json::Value *json, void *src_data, size_t src_size) const {
+    auto *data = static_cast<T *>(src_data);
+    if constexpr (std::is_same_v<T, char>) {
+      std::string str{data, src_size};
+      json->append(str);
+    } else {
+      // NOLINTNEXTLINE(readability-identifier-naming)
+      constexpr auto getData = [](const T *data_ptr, size_t index) {
+        if constexpr (std::is_same_v<T, uint64_t>) {
+          return static_cast<Json::UInt64>(data_ptr[index]);
+        } else if constexpr (std::is_same_v<T, int64_t>) {
+          return static_cast<Json::Int64>(data_ptr[index]);
+        } else if constexpr (util::is_any_v<T, bool, uint8_t, uint16_t,
+                                            uint32_t, int8_t, int16_t, int32_t,
+                                            float, double>) {
+          return data_ptr[index];
+        } else if constexpr (util::is_any_v<T, fp16>) {
+          return half_float::half_cast<float>(data_ptr[index]);
+        } else {
+          static_assert(!sizeof(T), "Invalid type to SetInputData");
+        }
+      };
+
+      for (auto i = 0U; i < src_size; ++i) {
+        json->append(getData(data, i));
+      }
+    }
+  }
 };
 
-drogon::HttpResponsePtr errorHttpResponse(const std::string &error,
-                                          int status_code);
+template <typename T>
+constexpr auto jsonValueToType(const Json::Value &datum) {
+  if constexpr (std::is_same_v<T, bool>) {
+    return datum.asBool();
+  } else if constexpr (util::is_any_v<T, uint8_t, uint16_t, uint32_t>) {
+    return datum.asUInt();
+  } else if constexpr (std::is_same_v<T, uint64_t>) {
+    return datum.asUInt64();
+  } else if constexpr (util::is_any_v<T, int8_t, int16_t, int32_t>) {
+    return datum.asInt();
+  } else if constexpr (std::is_same_v<T, int64_t>) {
+    return datum.asInt64();
+  } else if constexpr (util::is_any_v<T, fp16, float>) {
+    return datum.asFloat();
+  } else if constexpr (std::is_same_v<T, double>) {
+    return datum.asDouble();
+  } else if constexpr (std::is_same_v<T, char>) {
+    return datum.asString();
+  } else {
+    static_assert(!sizeof(T), "Invalid type to jsonValueToType");
+  }
+}
 
 /// convert the metadata to a JSON representation compatible with the server
 Json::Value modelMetadataToJson(const ModelMetadata &metadata);
diff --git a/src/amdinfer/clients/native.cpp b/src/amdinfer/clients/native.cpp
index b694e7e4e..ffbbbdc0b 100644
--- a/src/amdinfer/clients/native.cpp
+++ b/src/amdinfer/clients/native.cpp
@@ -26,12 +26,11 @@
 #include <string>   // for string
 #include <utility>  // for move
 
-#include "amdinfer/build_options.hpp"            // for AMDINFER_ENABLE_TRACING
-#include "amdinfer/clients/native_internal.hpp"  // for CppNativeApi
-#include "amdinfer/core/exceptions.hpp"          // for invalid_argument
-#include "amdinfer/core/interface.hpp"           // for Interface
-#include "amdinfer/core/parameters.hpp"          // for ParameterMap
-#include "amdinfer/core/shared_state.hpp"        // for SharedState
+#include "amdinfer/build_options.hpp"    // for AMDINFER_ENABLE_TRACING
+#include "amdinfer/core/exceptions.hpp"  // for invalid_argument
+#include "amdinfer/core/parameters.hpp"  // for ParameterMap
+#include "amdinfer/core/predict_api_internal.hpp"  // for RequestContainer
+#include "amdinfer/core/shared_state.hpp"          // for SharedState
 #include "amdinfer/observation/metrics.hpp"      // for Metrics, MetricCount...
 #include "amdinfer/observation/tracing.hpp"      // for startTrace, Trace
 #include "amdinfer/servers/server.hpp"           // for Server
@@ -82,6 +81,33 @@ std::string NativeClient::workerLoad(const std::string& worker,
   return impl_->state->workerLoad(worker_lower, parameters);
 }
 
+InferenceRequestPtr getRequest(const InferenceRequest& req,
+                               const MemoryPool* pool) {
+  auto request = std::make_shared<InferenceRequest>(req);
+
+  const auto& inputs = request->getInputs();
+  int i = 0;
+  for (const auto& input : inputs) {
+    auto size = input.getSize() * input.getDatatype().size();
+    auto buffer = pool->get({MemoryAllocators::Cpu}, size);
+    buffer->write(input.getData(), 0, size);
+    request->setInputTensorData(i, buffer->data(0));
+    i++;
+  }
+
+  return request;
+}
+
+InferenceResponseFuture setCallback(InferenceRequest* request) {
+  auto promise = std::make_shared<std::promise<amdinfer::InferenceResponse>>();
+  auto future = promise->get_future();
+  Callback callback = [promise](const InferenceResponse& response) {
+    promise->set_value(response);
+  };
+  request->setCallback(std::move(callback));
+  return future;
+}
+
 InferenceResponseFuture NativeClient::modelInferAsync(
   const std::string& model, const InferenceRequest& request) const {
 #ifdef AMDINFER_ENABLE_METRICS
@@ -92,13 +118,16 @@ InferenceResponseFuture NativeClient::modelInferAsync(
   auto trace = startTrace(&(__func__[0]));
   trace->startSpan("C++ enqueue");
 #endif
-  auto api = std::make_unique<CppNativeApi>(request);
-  auto future = api->getPromise()->get_future();
+  auto new_request = getRequest(request, impl_->state->getPool());
+  auto future = setCallback(new_request.get());
+  auto request_container = std::make_unique<RequestContainer>();
+  request_container->request = std::move(new_request);
+
 #ifdef AMDINFER_ENABLE_TRACING
   trace->endSpan();
-  api->setTrace(std::move(trace));
+  request_container->trace = std::move(trace);
 #endif
-  impl_->state->modelInfer(model, std::move(api));
+  impl_->state->modelInfer(model, std::move(request_container));
 
   return future;
 }
diff --git a/src/amdinfer/clients/native_internal.cpp b/src/amdinfer/clients/native_internal.cpp
deleted file mode 100644
index c22ca4dd7..000000000
--- a/src/amdinfer/clients/native_internal.cpp
+++ /dev/null
@@ -1,184 +0,0 @@
-// Copyright 2022 Xilinx, Inc.
-// Copyright 2022 Advanced Micro Devices, Inc.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-/**
- * @file
- * @brief Implements the internal objects used for the C++ API
- */
-
-#include "amdinfer/clients/native_internal.hpp"
-
-#include <cstddef>  // for byte, size_t
-#include <cstdint>  // for uint64_t
-#include <cstring>  // for memcpy
-#include <string>   // for string
-#include <utility>  // for move
-
-#include "amdinfer/buffers/buffer.hpp"       // for Buffer
-#include "amdinfer/core/data_types.hpp"      // for DataType
-#include "amdinfer/core/parameters.hpp"      // for ParameterMapPtr
-#include "amdinfer/observation/logging.hpp"  // for Logger, AMDINFER_LOG_ERROR
-#include "amdinfer/util/containers.hpp"      // for containerProduct
-
-namespace amdinfer {
-template <typename T>
-class InferenceRequestBuilder;
-
-template <typename T>
-class InferenceRequestInputBuilder;
-}  // namespace amdinfer
-
-namespace amdinfer {
-
-template <>
-class InferenceRequestInputBuilder<InferenceRequestInput> {
- public:
-  static InferenceRequestInput build(const InferenceRequestInput &req,
-                                     Buffer *input_buffer, size_t offset) {
-    InferenceRequestInput input;
-    input.data_ = req.getData();
-    input.name_ = req.name_;
-    input.shape_.reserve(req.shape_.size());
-    input.shape_ = req.shape_;
-    input.data_type_ = req.data_type_;
-    input.parameters_ = req.parameters_;
-    auto size = util::containerProduct(input.shape_) * input.data_type_.size();
-    auto *dest = static_cast<std::byte *>(input_buffer->data(offset));
-    memcpy(dest, input.data_, size);
-
-    input.data_ = dest;
-    return input;
-  }
-};
-
-using InputBuilder = InferenceRequestInputBuilder<InferenceRequestInput>;
-
-template <>
-class InferenceRequestBuilder<InferenceRequest> {
- public:
-  static InferenceRequestPtr build(const InferenceRequest &req,
-                                   const BufferRawPtrs &input_buffers,
-                                   std::vector<size_t> &input_offsets,
-                                   const BufferRawPtrs &output_buffers,
-                                   std::vector<size_t> &output_offsets) {
-    auto request = std::make_shared<InferenceRequest>();
-
-    request->id_ = req.id_;
-    request->parameters_ = req.parameters_;
-    request->callback_ = nullptr;
-
-    for (const auto &input : req.inputs_) {
-      const auto &buffers = input_buffers;
-      auto index = 0;
-      for (const auto &buffer : buffers) {
-        auto &offset = input_offsets[index];
-
-        request->inputs_.push_back(InputBuilder::build(input, buffer, offset));
-        const auto &last_input = request->inputs_.back();
-        offset += (last_input.getSize() * last_input.getDatatype().size());
-        index++;
-      }
-    }
-
-    // TODO(varunsh): output_offset is currently ignored! The size of the output
-    // needs to come from the worker but we have no such information.
-    if (!req.outputs_.empty()) {
-      for (const auto &output : req.outputs_) {
-        const auto &buffers = output_buffers;
-        auto index = 0;
-        for (const auto &buffer : buffers) {
-          const auto &offset = output_offsets[index];
-
-          request->outputs_.emplace_back(output);
-          request->outputs_.back().setData(
-            static_cast<std::byte *>(buffer->data(offset)));
-          index++;
-        }
-      }
-    } else {
-      for (const auto &input : req.inputs_) {
-        (void)input;  // suppress unused variable warning
-        const auto &buffers = output_buffers;
-        auto index = 0;
-        for (const auto &buffer : buffers) {
-          const auto &offset = output_offsets[index];
-
-          request->outputs_.emplace_back();
-          request->outputs_.back().setData(
-            static_cast<std::byte *>(buffer->data(offset)));
-          index++;
-        }
-      }
-    }
-
-    return request;
-  }
-};
-
-using RequestBuilder = InferenceRequestBuilder<InferenceRequest>;
-
-CppNativeApi::CppNativeApi(InferenceRequest request)
-  : request_(std::move(request)) {
-  this->promise_ =
-    std::make_unique<std::promise<amdinfer::InferenceResponse>>();
-}
-
-size_t CppNativeApi::getInputSize() { return this->request_.getInputSize(); }
-
-std::vector<size_t> CppNativeApi::getInputSizes() const {
-  std::vector<size_t> sizes;
-  for (const auto &tensor : request_.getInputs()) {
-    sizes.push_back(tensor.getSize() * tensor.getDatatype().size());
-  }
-  return sizes;
-}
-
-// std::vector<size_t> CppNativeApi::getOutputSizes() {
-//   std::vector<size_t> sizes;
-//   for(const auto& tensor : request_.getOutputs()){
-//     sizes.push_back(tensor.getSize() * tensor.getDatatype().size);
-//   }
-//   return sizes;
-// }
-
-std::promise<amdinfer::InferenceResponse> *CppNativeApi::getPromise() {
-  return this->promise_.get();
-}
-
-// void cppCallback(const InferenceResponsePromisePtr &promise,
-//                  const InferenceResponse &response) {
-//   promise->set_value(response);
-// }
-
-std::shared_ptr<InferenceRequest> CppNativeApi::getRequest(
-  const BufferRawPtrs &input_buffers, std::vector<size_t> &input_offsets,
-  const BufferRawPtrs &output_buffers, std::vector<size_t> &output_offsets) {
-  auto request =
-    RequestBuilder::build(this->request_, input_buffers, input_offsets,
-                          output_buffers, output_offsets);
-  Callback callback =
-    [promise = std::move(this->promise_)](const InferenceResponse &response) {
-      promise->set_value(response);
-    };
-  request->setCallback(std::move(callback));
-  return request;
-}
-
-void CppNativeApi::errorHandler(const std::exception &e) {
-  AMDINFER_LOG_ERROR(this->getLogger(), e.what());
-  this->getPromise()->set_value(InferenceResponse(e.what()));
-}
-
-}  // namespace amdinfer
diff --git a/src/amdinfer/clients/native_internal.hpp b/src/amdinfer/clients/native_internal.hpp
deleted file mode 100644
index 99806a32d..000000000
--- a/src/amdinfer/clients/native_internal.hpp
+++ /dev/null
@@ -1,58 +0,0 @@
-// Copyright 2022 Xilinx, Inc.
-// Copyright 2022 Advanced Micro Devices, Inc.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-/**
- * @file
- * @brief Defines the internal objects used for the C++ API
- */
-
-#ifndef GUARD_AMDINFER_CLIENTS_NATIVE_INTERNAL
-#define GUARD_AMDINFER_CLIENTS_NATIVE_INTERNAL
-
-#include <cstddef>    // for size_t
-#include <exception>  // for exception
-#include <future>     // for promise
-#include <memory>     // for shared_ptr
-#include <vector>     // for vector
-
-#include "amdinfer/core/interface.hpp"    // for Interface
-#include "amdinfer/core/predict_api.hpp"  // for InferenceRequest, Infere...
-#include "amdinfer/declarations.hpp"      // for BufferRawPtrs
-
-namespace amdinfer {
-
-class CppNativeApi : public Interface {
- public:
-  explicit CppNativeApi(InferenceRequest request);
-
-  std::shared_ptr<InferenceRequest> getRequest(
-    const BufferRawPtrs &input_buffers, std::vector<size_t> &input_offsets,
-    const BufferRawPtrs &output_buffers,
-    std::vector<size_t> &output_offsets) override;
-
-  size_t getInputSize() override;
-  std::vector<size_t> getInputSizes() const override;
-  // std::vector<size_t> getOutputSizes() override;
-  void errorHandler(const std::exception &e) override;
-  std::promise<amdinfer::InferenceResponse> *getPromise();
-
- private:
-  InferenceRequest request_;
-  InferenceResponsePromisePtr promise_;
-};
-
-}  // namespace amdinfer
-
-#endif  // GUARD_AMDINFER_CLIENTS_NATIVE_INTERNAL
diff --git a/src/amdinfer/core/CMakeLists.txt b/src/amdinfer/core/CMakeLists.txt
index 6d041647e..41bba600c 100644
--- a/src/amdinfer/core/CMakeLists.txt
+++ b/src/amdinfer/core/CMakeLists.txt
@@ -20,7 +20,6 @@ list(
          worker_info
          data_types
          data_types_internal
-         interface
          model_repository
          parameters
          shared_state
diff --git a/src/amdinfer/core/endpoints.cpp b/src/amdinfer/core/endpoints.cpp
index ce85ae55b..96871dbe9 100644
--- a/src/amdinfer/core/endpoints.cpp
+++ b/src/amdinfer/core/endpoints.cpp
@@ -22,13 +22,13 @@
 #include <cassert>      // for assert
 #include <type_traits>  // for __decay_and_strip<>::__type
 
-#include "amdinfer/batching/batcher.hpp"  // for Batcher
-#include "amdinfer/build_options.hpp"     // for kMaxModelNameSize
-#include "amdinfer/core/exceptions.hpp"   // for invalid_argument
-#include "amdinfer/core/interface.hpp"    // IWYU pragma: keep
-#include "amdinfer/core/parameters.hpp"   // for ParameterMap
-#include "amdinfer/core/worker_info.hpp"  // for WorkerInfo
-#include "amdinfer/util/thread.hpp"       // for setThreadName
+#include "amdinfer/batching/batcher.hpp"           // for Batcher
+#include "amdinfer/build_options.hpp"              // for kMaxModelNameSize
+#include "amdinfer/core/exceptions.hpp"            // for invalid_argument
+#include "amdinfer/core/parameters.hpp"            // for ParameterMap
+#include "amdinfer/core/predict_api_internal.hpp"  // for RequestContainer
+#include "amdinfer/core/worker_info.hpp"           // for WorkerInfo
+#include "amdinfer/util/thread.hpp"                // for setThreadName
 
 namespace amdinfer {
 
@@ -69,12 +69,12 @@ void Endpoints::unload(const std::string& endpoint) {
 
 // TODO(varunsh): race condition if workers are shutting down
 void Endpoints::infer(const std::string& endpoint,
-                      std::unique_ptr<Interface> request) const {
+                      std::unique_ptr<RequestContainer> request) const {
   WorkerInfo* worker = this->unsafeGet(endpoint);
   if (worker == nullptr) {
     throw invalid_argument("Worker " + endpoint + " not found");
   }
-  auto* batcher = worker->getBatcher();
+  const auto* batcher = worker->getBatcher();
   batcher->enqueue(std::move(request));
 }
 
@@ -140,6 +140,8 @@ ModelMetadata Endpoints::metadata(const std::string& endpoint) {
   return metadata;
 }
 
+const MemoryPool* Endpoints::getPool() const { return &pool_; }
+
 // TODO(varunsh): if multiple commands sent post-shutdown, they will linger
 // in the queue and may cause problems
 void Endpoints::shutdown() {
diff --git a/src/amdinfer/core/endpoints.hpp b/src/amdinfer/core/endpoints.hpp
index 7e9d39288..4eb16390c 100644
--- a/src/amdinfer/core/endpoints.hpp
+++ b/src/amdinfer/core/endpoints.hpp
@@ -37,7 +37,7 @@
 
 namespace amdinfer {
 
-class Interface;
+class RequestContainer;
 class ParameterMap;
 class WorkerInfo;
 
@@ -90,7 +90,7 @@ class Endpoints {
   void unload(const std::string& endpoint);
 
   void infer(const std::string& endpoint,
-             std::unique_ptr<Interface> request) const;
+             std::unique_ptr<RequestContainer> request) const;
 
   bool exists(const std::string& endpoint);
   // WorkerInfo* get(const std::string& endpoint);
@@ -99,6 +99,8 @@ class Endpoints {
   std::vector<std::string> list();
   ModelMetadata metadata(const std::string& endpoint);
 
+  const MemoryPool* getPool() const;
+
   void shutdown();
 
  private:
diff --git a/src/amdinfer/core/interface.cpp b/src/amdinfer/core/interface.cpp
deleted file mode 100644
index 9de9de9af..000000000
--- a/src/amdinfer/core/interface.cpp
+++ /dev/null
@@ -1,54 +0,0 @@
-// Copyright 2021 Xilinx, Inc.
-// Copyright 2022 Advanced Micro Devices, Inc.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-/**
- * @file
- * @brief Implements the base Interface class
- */
-
-#include "amdinfer/core/interface.hpp"
-
-#include <utility>  // for move
-
-#include "amdinfer/build_options.hpp"        // for AMDINFER_ENABLE_LOGGING
-#include "amdinfer/observation/tracing.hpp"  // for TracePtr
-
-namespace amdinfer {
-
-Interface::Interface() { this->type_ = InterfaceType::Unknown; }
-
-InterfaceType Interface::getType() const { return this->type_; }
-
-#ifdef AMDINFER_ENABLE_TRACING
-void Interface::setTrace(TracePtr&& trace) { this->trace_ = std::move(trace); }
-TracePtr&& Interface::getTrace() { return std::move(this->trace_); }
-#endif
-
-#ifdef AMDINFER_ENABLE_METRICS
-void Interface::setTime(
-  const std::chrono::high_resolution_clock::time_point& start_time) {
-  this->start_time_ = start_time;
-}
-
-std::chrono::high_resolution_clock::time_point Interface::getTime() const {
-  return this->start_time_;
-}
-#endif
-
-#ifdef AMDINFER_ENABLE_LOGGING
-const Logger& Interface::getLogger() const { return this->logger_; }
-#endif
-
-}  // namespace amdinfer
diff --git a/src/amdinfer/core/interface.hpp b/src/amdinfer/core/interface.hpp
deleted file mode 100644
index 54f5262dd..000000000
--- a/src/amdinfer/core/interface.hpp
+++ /dev/null
@@ -1,125 +0,0 @@
-// Copyright 2021 Xilinx, Inc.
-// Copyright 2022 Advanced Micro Devices, Inc.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-/**
- * @file
- * @brief Defines the base Interface class
- */
-
-#ifndef GUARD_AMDINFER_CORE_INTERFACE
-#define GUARD_AMDINFER_CORE_INTERFACE
-
-#include <chrono>     // for high_resolution_clock
-#include <cstddef>    // for size_t
-#include <exception>  // for exception
-#include <memory>     // for shared_ptr
-#include <vector>     // for vector
-
-#include "amdinfer/build_options.hpp"        // for AMDINFER_ENABLE_LOGGING
-#include "amdinfer/declarations.hpp"         // for BufferRawPtrs
-#include "amdinfer/observation/logging.hpp"  // for Logger, Loggers, Loggers...
-#include "amdinfer/observation/tracing.hpp"  // for TracePtr
-
-namespace amdinfer {
-class InferenceRequest;
-}  // namespace amdinfer
-
-namespace amdinfer {
-
-/**
- * @brief The InterfaceType identifies the Interface. New interfaces can be
- * added by extending this enumeration.
- *
- */
-enum class InterfaceType {
-  Unknown,
-  DrogonHttp,
-  DrogonWs,
-  Cpp,
-  Grpc,
-};
-
-/**
- * @brief The Interface class represents an input-agnostic wrapper class that
- * can encapsulate incoming requests from different protocols and present a
- * consistent object and interface to the batcher to process all the requests.
- *
- */
-class Interface {
- public:
-  Interface();                     ///< Constructor
-  virtual ~Interface() = default;  ///< Destructor
-  /// Get the type of the interface
-  [[nodiscard]] InterfaceType getType() const;
-#ifdef AMDINFER_ENABLE_TRACING
-  /// Store the active trace into the Interface for propagation
-  void setTrace(TracePtr &&trace);
-  /// Get the stored trace from the interface for propagation
-  TracePtr &&getTrace();
-#endif
-#ifdef AMDINFER_ENABLE_METRICS
-  /// Store a time. This is used to track how long requests take
-  void setTime(
-    const std::chrono::high_resolution_clock::time_point &start_time);
-  /// Get the stored time
-  [[nodiscard]] std::chrono::high_resolution_clock::time_point getTime() const;
-#endif
-  /// Get the number of input tensors in the request
-  virtual size_t getInputSize() = 0;
-
-  virtual std::vector<size_t> getInputSizes() const = 0;
-  // virtual std::vector<size_t> getOutputSizes() = 0;
-  /**
-   * @brief Construct an InferenceRequest using the data in the Interface
-   *
-   * @param input_buffers a vector of buffers to hold the input data
-   * @param input_offsets offsets of where to start storing input data
-   * @param output_buffers a vector of buffers to hold the output data
-   * @param output_offsets offsets of where to start storing output data
-   * @return std::shared_ptr<InferenceRequest>
-   */
-  virtual std::shared_ptr<InferenceRequest> getRequest(
-    const BufferRawPtrs &input_buffers, std::vector<size_t> &input_offsets,
-    const BufferRawPtrs &output_buffers,
-    std::vector<size_t> &output_offsets) = 0;
-
-  /**
-   * @brief Given an exception, handle the exception appropriately.
-   *
-   * @param e the raised exception
-   */
-  virtual void errorHandler(const std::exception &e) = 0;
-
- protected:
-  InterfaceType type_;
-#ifdef AMDINFER_ENABLE_TRACING
-  TracePtr trace_;
-#endif
-#ifdef AMDINFER_ENABLE_METRICS
-  std::chrono::_V2::system_clock::time_point start_time_;
-#endif
-#ifdef AMDINFER_ENABLE_LOGGING
-  [[nodiscard]] const Logger &getLogger() const;
-#endif
-
- private:
-#ifdef AMDINFER_ENABLE_LOGGING
-  Logger logger_{Loggers::Server};
-#endif
-};
-
-}  // namespace amdinfer
-
-#endif  // GUARD_AMDINFER_CORE_INTERFACE
diff --git a/src/amdinfer/core/memory_pool/pool.cpp b/src/amdinfer/core/memory_pool/pool.cpp
index e80b6625e..f07ef14b0 100644
--- a/src/amdinfer/core/memory_pool/pool.cpp
+++ b/src/amdinfer/core/memory_pool/pool.cpp
@@ -33,7 +33,7 @@ MemoryPool::MemoryPool() {
 }
 
 std::unique_ptr<Buffer> MemoryPool::get(
-  const std::vector<MemoryAllocators>& allocators, size_t size) {
+  const std::vector<MemoryAllocators>& allocators, size_t size) const {
   for (const auto& allocator : allocators) {
     void* address = nullptr;
     try {
@@ -48,7 +48,7 @@ std::unique_ptr<Buffer> MemoryPool::get(
   throw runtime_error("Memory could not be allocated");
 }
 
-void MemoryPool::put(std::unique_ptr<Buffer> memory) {
+void MemoryPool::put(std::unique_ptr<Buffer> memory) const {
   auto* buffer = dynamic_cast<CpuBuffer*>(memory.get());
   const auto allocator = buffer->getAllocator();
   const auto* address = buffer->data(0);
diff --git a/src/amdinfer/core/memory_pool/pool.hpp b/src/amdinfer/core/memory_pool/pool.hpp
index bd16d73f5..922ed588b 100644
--- a/src/amdinfer/core/memory_pool/pool.hpp
+++ b/src/amdinfer/core/memory_pool/pool.hpp
@@ -40,8 +40,8 @@ class MemoryPool {
   MemoryPool();
 
   std::unique_ptr<Buffer> get(const std::vector<MemoryAllocators>& allocators,
-                              size_t size);
-  void put(std::unique_ptr<Buffer> memory);
+                              size_t size) const;
+  void put(std::unique_ptr<Buffer> memory) const;
 
  private:
   std::unordered_map<MemoryAllocators, std::unique_ptr<MemoryAllocator>>
diff --git a/src/amdinfer/core/predict_api.cpp b/src/amdinfer/core/predict_api.cpp
index 1f237e9e8..33de47021 100644
--- a/src/amdinfer/core/predict_api.cpp
+++ b/src/amdinfer/core/predict_api.cpp
@@ -61,6 +61,13 @@ void InferenceRequest::addInputTensor(InferenceRequestInput input) {
   this->inputs_.push_back(std::move(input));
 }
 
+void InferenceRequest::setInputTensorData(size_t index, void *data) {
+  if (index < inputs_.size()) {
+    auto &input = inputs_.at(index);
+    input.setData(data);
+  }
+}
+
 const std::vector<InferenceRequestInput> &InferenceRequest::getInputs() const {
   return this->inputs_;
 }
diff --git a/src/amdinfer/core/predict_api_internal.hpp b/src/amdinfer/core/predict_api_internal.hpp
index 8230da342..3949fb1e3 100644
--- a/src/amdinfer/core/predict_api_internal.hpp
+++ b/src/amdinfer/core/predict_api_internal.hpp
@@ -26,56 +26,14 @@
 
 namespace amdinfer {
 
-template <typename T>
-class InferenceRequestBuilder {
- public:
-  /**
-   * @brief Construct a new InferenceRequest object
-   *
-   * @param req some valid type holding data
-   * @param buffer_index current buffer index to start with for the buffers
-   * @param input_buffers a vector of input buffers to store the inputs data
-   * @param input_offsets a vector of offsets for the input buffers to store
-   * data
-   * @param output_buffers a vector of output buffers
-   * @param output_offsets a vector of offsets for the output buffers
-   * @param batch_size batch size to use when creating the request
-   * @param batch_offset current batch offset to start with
-   */
-  static InferenceRequestPtr build(
-    T req, size_t &buffer_index,
-    const std::vector<BufferRawPtrs> &input_buffers,
-    std::vector<size_t> &input_offsets,
-    const std::vector<BufferRawPtrs> &output_buffers,
-    std::vector<size_t> &output_offsets, const size_t &batch_size,
-    size_t &batch_offset);
-};
-
-template <typename T>
-class InferenceRequestInputBuilder {
- public:
-  /**
-   * @brief Construct a new InferenceRequestInput object
-   *
-   * @param req some valid type holding data
-   * @param input_buffer buffer to hold the incoming data
-   * @param offset offset for the buffer to store data at
-   */
-  static InferenceRequestInput build(const T &req, Buffer *input_buffer,
-                                     size_t offset);
-};
-
-template <typename T>
-class InferenceRequestOutputBuilder {
- public:
-  /**
-   * @brief Construct a new InferenceRequestOutput object
-   *
-   * @param req some valid type holding data
-   * @param input_buffer buffer to hold the incoming data
-   * @param offset offset for the buffer to store data at
-   */
-  static InferenceRequestOutput build(const T &req);
+struct RequestContainer {
+  InferenceRequestPtr request;
+#ifdef AMDINFER_ENABLE_TRACING
+  TracePtr trace;
+#endif
+#ifdef AMDINFER_ENABLE_METRICS
+  std::chrono::_V2::system_clock::time_point start_time;
+#endif
 };
 
 }  // namespace amdinfer
diff --git a/src/amdinfer/core/shared_state.cpp b/src/amdinfer/core/shared_state.cpp
index 586e818ad..ccf4c30b2 100644
--- a/src/amdinfer/core/shared_state.cpp
+++ b/src/amdinfer/core/shared_state.cpp
@@ -33,12 +33,12 @@
 #include "amdinfer/build_options.hpp"          // for AMDINFER_ENABLE_VITIS
 #include "amdinfer/core/endpoints.hpp"         // for Endpoints
 #include "amdinfer/core/exceptions.hpp"        // for external_error, invalid...
-#include "amdinfer/core/interface.hpp"         // IWYU pragma: keep
 #include "amdinfer/core/model_repository.hpp"  // for ModelRepository
 #include "amdinfer/core/parameters.hpp"        // for ParameterMap
-#include "amdinfer/core/predict_api.hpp"       // for ServerMetadata, ModelMe...
-#include "amdinfer/util/string.hpp"            // for isLower
-#include "amdinfer/version.hpp"                // for kAmdinferVersion
+#include "amdinfer/core/predict_api_internal.hpp"  // for ServerMetadata, ModelMe...
+#include "amdinfer/observation/observer.hpp"
+#include "amdinfer/util/string.hpp"  // for isLower
+#include "amdinfer/version.hpp"      // for kAmdinferVersion
 
 #ifdef AMDINFER_ENABLE_VITIS
 #include <sockpp/socket.h>         // for socket, socket_initializer
@@ -99,7 +99,7 @@ void SharedState::workerUnload(const std::string& worker) {
 }
 
 void SharedState::modelInfer(const std::string& model,
-                             std::unique_ptr<Interface> request) {
+                             std::unique_ptr<RequestContainer> request) {
   endpoints_.infer(model, std::move(request));
 }
 
@@ -186,6 +186,8 @@ bool SharedState::hasHardware(const std::string& name, int num) {
   return kernel_iterator->second >= num;
 }
 
+const MemoryPool* SharedState::getPool() const { return endpoints_.getPool(); }
+
 void SharedState::setRepository(const fs::path& repository_path,
                                 bool load_existing) {
   repository_.setEndpoints(&endpoints_);
diff --git a/src/amdinfer/core/shared_state.hpp b/src/amdinfer/core/shared_state.hpp
index fa616d682..36b47e999 100644
--- a/src/amdinfer/core/shared_state.hpp
+++ b/src/amdinfer/core/shared_state.hpp
@@ -32,7 +32,7 @@
 
 namespace amdinfer {
 
-class Interface;
+class RequestContainer;
 class ParameterMap;
 
 class SharedState {
@@ -47,11 +47,14 @@ class SharedState {
   bool modelReady(const std::string& model);
   ModelMetadata modelMetadata(const std::string& model);
 
-  void modelInfer(const std::string& model, std::unique_ptr<Interface> request);
+  void modelInfer(const std::string& model,
+                  std::unique_ptr<RequestContainer> request);
 
   static Kernels getHardware();
   static bool hasHardware(const std::string& name, int num);
 
+  const MemoryPool* getPool() const;
+
   void setRepository(const std::filesystem::path& repository_path,
                      bool load_existing);
   void enableRepositoryMonitoring(bool use_polling);
diff --git a/src/amdinfer/core/worker_info.cpp b/src/amdinfer/core/worker_info.cpp
index 9f1687133..896848398 100644
--- a/src/amdinfer/core/worker_info.cpp
+++ b/src/amdinfer/core/worker_info.cpp
@@ -33,10 +33,9 @@
 
 #include "amdinfer/batching/batcher.hpp"  // for Batcher, BatcherStatus, Bat...
 #include "amdinfer/core/exceptions.hpp"   // for invalid_argument, external_...
-#include "amdinfer/core/interface.hpp"    // IWYU pragma: keep
-#include "amdinfer/core/memory_pool/pool.hpp"  // for MemoryPool
-#include "amdinfer/core/parameters.hpp"        // for ParameterMap
-#include "amdinfer/core/predict_api.hpp"       // for ModelMetadata
+#include "amdinfer/core/memory_pool/pool.hpp"      // for MemoryPool
+#include "amdinfer/core/parameters.hpp"            // for ParameterMap
+#include "amdinfer/core/predict_api_internal.hpp"  // for ModelMetadata
 #include "amdinfer/workers/worker.hpp"  // for Worker, WorkerStatus, Worke...
 
 namespace amdinfer {
diff --git a/src/amdinfer/servers/grpc_server.cpp b/src/amdinfer/servers/grpc_server.cpp
index 68f5dc9b4..cfdbeab41 100644
--- a/src/amdinfer/servers/grpc_server.cpp
+++ b/src/amdinfer/servers/grpc_server.cpp
@@ -41,7 +41,6 @@
 #include "amdinfer/clients/grpc_internal.hpp"      // for mapProtoToParameters
 #include "amdinfer/core/data_types.hpp"            // for DataType, DataType:...
 #include "amdinfer/core/exceptions.hpp"            // for invalid_argument
-#include "amdinfer/core/interface.hpp"             // for Interface, Interfac...
 #include "amdinfer/core/parameters.hpp"            // for ParameterMap
 #include "amdinfer/core/predict_api_internal.hpp"  // for InferenceRequestInput
 #include "amdinfer/core/shared_state.hpp"          // for SharedState
@@ -228,47 +227,6 @@ struct WriteData {
   }
 };
 
-template <>
-class InferenceRequestInputBuilder<
-  inference::ModelInferRequest_InferInputTensor> {
- public:
-  static InferenceRequestInput build(
-    const inference::ModelInferRequest_InferInputTensor& req,
-    Buffer* input_buffer, size_t offset) {
-    Observer observer;
-    AMDINFER_IF_LOGGING(observer.logger = Logger{Loggers::Server});
-
-    AMDINFER_LOG_TRACE(observer.logger,
-                       "Creating InferenceRequestInput from proto tensor");
-
-    InferenceRequestInput input;
-    input.name_ = req.name();
-    input.shape_.reserve(req.shape_size());
-    for (const auto& index : req.shape()) {
-      input.shape_.push_back(static_cast<size_t>(index));
-    }
-    input.data_type_ = DataType(req.datatype().c_str());
-
-    input.parameters_ = mapProtoToParameters(req.parameters());
-
-    auto size = input.getSize();
-    auto* dest = static_cast<std::byte*>(input_buffer->data(offset));
-    AMDINFER_LOG_TRACE(observer.logger, "Writing " + std::to_string(size) +
-                                          " elements of type " +
-                                          input.data_type_.str() + " to " +
-                                          util::addressToString(dest));
-
-    switchOverTypes(WriteData(), input.getDatatype(), input_buffer, &req,
-                    offset, size, observer);
-
-    input.data_ = dest;
-    return input;
-  }
-};
-
-using InputBuilder =
-  InferenceRequestInputBuilder<inference::ModelInferRequest_InferInputTensor>;
-
 #ifdef AMDINFER_ENABLE_LOGGING
 // NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
 #define CALLDATA_IMPL(endpoint, type)                                      \
@@ -336,79 +294,104 @@ const inference::ModelInferRequest& getRequest() const {
 inference::ModelInferResponse& getReply() { return this->reply_; }
 CALLDATA_IMPL_END
 
-template <>
-class InferenceRequestBuilder<CallDataModelInfer*> {
- public:
-  static InferenceRequestPtr build(const CallDataModelInfer* req,
-                                   const BufferRawPtrs& input_buffers,
-                                   std::vector<size_t>& input_offsets,
-                                   const BufferRawPtrs& output_buffers,
-                                   std::vector<size_t>& output_offsets) {
-    Observer observer;
-    AMDINFER_IF_LOGGING(observer.logger = Logger{Loggers::Server});
+InferenceRequestInput getInput(
+  const inference::ModelInferRequest_InferInputTensor& req,
+  const MemoryPool* pool) {
+  Observer observer;
+  AMDINFER_IF_LOGGING(observer.logger = Logger{Loggers::Server});
 
-    AMDINFER_LOG_TRACE(observer.logger,
-                       "Creating InferenceRequest from proto tensor");
+  AMDINFER_LOG_TRACE(observer.logger,
+                     "Creating InferenceRequestInput from proto tensor");
 
-    auto request = std::make_shared<InferenceRequest>();
-    const auto& grpc_request = req->getRequest();
+  InferenceRequestInput input;
+  input.setName(req.name());
 
-    request->id_ = grpc_request.id();
+  std::vector<uint64_t> shape_vector;
+  shape_vector.reserve(req.shape_size());
+  for (const auto& index : req.shape()) {
+    shape_vector.push_back(static_cast<size_t>(index));
+  }
+  input.setShape(shape_vector);
+  input.setDatatype(DataType(req.datatype().c_str()));
 
-    request->parameters_ = mapProtoToParameters(grpc_request.parameters());
+  input.setParameters(mapProtoToParameters(req.parameters()));
 
-    request->callback_ = nullptr;
+  auto size = input.getSize();
+  auto buffer =
+    pool->get({MemoryAllocators::Cpu}, size * input.getDatatype().size());
+  input.setData(buffer->data(0));
+  // auto* dest = static_cast<std::byte*>(input_buffer->data(offset));
+  AMDINFER_LOG_TRACE(observer.logger, "Writing " + std::to_string(size) +
+                                        " elements of type " +
+                                        input.getDatatype().str() + " to " +
+                                        util::addressToString(buffer->data(0)));
 
-    for (const auto& input : grpc_request.inputs()) {
-      const auto& buffers = input_buffers;
-      auto index = 0;
-      for (const auto& buffer : buffers) {
-        auto& offset = input_offsets[index];
+  switchOverTypes(WriteData(), input.getDatatype(), buffer.get(), &req, 0, size,
+                  observer);
 
-        request->inputs_.push_back(InputBuilder::build(input, buffer, offset));
-        const auto& last_input = request->inputs_.back();
-        offset += (last_input.getSize() * last_input.getDatatype().size());
-        index++;
-      }
-    }
+  return input;
+}
 
-    // TODO(varunsh): output_offset is currently ignored! The size of the
-    // output needs to come from the worker but we have no such information.
-    if (grpc_request.outputs_size() != 0) {
-      for (const auto& output : grpc_request.outputs()) {
-        // TODO(varunsh): we're ignoring incoming output data
-        (void)output;
-        const auto& buffers = output_buffers;
-        auto index = 0;
-        for (const auto& buffer : buffers) {
-          auto& offset = output_offsets[index];
-
-          request->outputs_.emplace_back();
-          request->outputs_.back().setData(
-            static_cast<std::byte*>(buffer->data(offset)));
-          index++;
-        }
-      }
-    } else {
-      for (const auto& input : grpc_request.inputs()) {
-        (void)input;  // suppress unused variable warning
-        const auto& buffers = output_buffers;
-        for (size_t j = 0; j < buffers.size(); j++) {
-          const auto& buffer = buffers[j];
-          const auto& offset = output_offsets[j];
-
-          request->outputs_.emplace_back();
-          request->outputs_.back().setData(
-            static_cast<std::byte*>(buffer->data(offset)));
-        }
-      }
+InferenceRequestOutput getOutput(
+  const inference::ModelInferRequest_InferRequestedOutputTensor& proto) {
+  InferenceRequestOutput output;
+  output.setData(nullptr);
+  output.setName(proto.name());
+  output.setParameters(mapProtoToParameters(proto.parameters()));
+  return output;
+}
+
+void setCallback(InferenceRequest* request, CallDataModelInfer* calldata) {
+  Callback callback = [calldata](const InferenceResponse& response) {
+    if (response.isError()) {
+      calldata->finish(
+        ::grpc::Status(StatusCode::UNKNOWN, response.getError()));
+      return;
     }
+    try {
+      mapResponseToProto(response, calldata->getReply());
+    } catch (const invalid_argument& e) {
+      calldata->finish(::grpc::Status(StatusCode::UNKNOWN, e.what()));
+      return;
+    }
+
+    // #ifdef AMDINFER_ENABLE_TRACING
+    //   const auto &context = response.getContext();
+    //   propagate(resp.get(), context);
+    // #endif
+    calldata->finish(::grpc::Status::OK);
+  };
+  request->setCallback(std::move(callback));
+}
+
+InferenceRequestPtr getRequest(const inference::ModelInferRequest& grpc_request,
+                               const MemoryPool* pool) {
+  Observer observer;
+  AMDINFER_IF_LOGGING(observer.logger = Logger{Loggers::Server});
+
+  AMDINFER_LOG_TRACE(observer.logger,
+                     "Creating InferenceRequest from proto tensor");
+
+  auto request = std::make_shared<InferenceRequest>();
+
+  request->setID(grpc_request.id());
 
-    return request;
+  request->setParameters(mapProtoToParameters(grpc_request.parameters()));
+
+  request->setCallback(nullptr);
+
+  for (const auto& input : grpc_request.inputs()) {
+    request->addInputTensor(getInput(input, pool));
+  }
+
+  if (grpc_request.outputs_size() != 0) {
+    for (const auto& output : grpc_request.outputs()) {
+      request->addOutputTensor(getOutput(output));
+    }
   }
-};
 
-using RequestBuilder = InferenceRequestBuilder<CallDataModelInfer*>;
+  return request;
+}
 
 // CALLDATA_IMPL(StreamModelInfer, ServerStream);
 
@@ -438,74 +421,6 @@ void grpcUnaryCallback(CallDataModelInfer* calldata,
   calldata->finish(::grpc::Status::OK);
 }
 
-class GrpcApiUnary : public Interface {
- public:
-  /**
-   * @brief Construct a new DrogonHttp object
-   *
-   * @param req
-   * @param callback
-   */
-  explicit GrpcApiUnary(CallDataModelInfer* calldata) : calldata_(calldata) {
-    this->type_ = InterfaceType::Grpc;
-  }
-
-  std::shared_ptr<InferenceRequest> getRequest(
-    const BufferRawPtrs& input_buffers, std::vector<size_t>& input_offsets,
-    const BufferRawPtrs& output_buffers,
-    std::vector<size_t>& output_offsets) override {
-#ifdef AMDINFER_ENABLE_LOGGING
-    const auto& logger = this->getLogger();
-#endif
-    try {
-      auto request =
-        RequestBuilder::build(this->calldata_, input_buffers, input_offsets,
-                              output_buffers, output_offsets);
-      // Callback callback =
-      //   std::bind(grpcUnaryCallback, this->calldata_, std::placeholders::_1);
-      Callback callback =
-        [calldata = this->calldata_](const InferenceResponse& response) {
-          grpcUnaryCallback(calldata, response);
-        };
-      request->setCallback(std::move(callback));
-      return request;
-    } catch (const invalid_argument& e) {
-      AMDINFER_LOG_INFO(logger, e.what());
-      errorHandler(e);
-      return nullptr;
-    }
-  }
-
-  size_t getInputSize() override {
-    return calldata_->getRequest().inputs_size();
-  }
-
-  [[nodiscard]] std::vector<size_t> getInputSizes() const override {
-    std::vector<size_t> sizes;
-
-    auto inputs = calldata_->getRequest().inputs();
-
-    for (const auto& tensor : inputs) {
-      const auto& raw_type = tensor.datatype();
-      auto datatype = DataType(raw_type.c_str());
-
-      const auto& shape = tensor.shape();
-      auto size = util::containerProduct(shape);
-
-      sizes.push_back(size * datatype.size());
-    }
-    return sizes;
-  }
-
-  void errorHandler(const std::exception& e) override {
-    AMDINFER_LOG_INFO(this->getLogger(), e.what());
-    calldata_->finish(::grpc::Status(StatusCode::UNKNOWN, e.what()));
-  }
-
- private:
-  CallDataModelInfer* calldata_;
-};
-
 CALLDATA_IMPL(ServerLive, Unary) {
   reply_.set_live(true);
   finish(::grpc::Status::OK);
@@ -639,12 +554,15 @@ void CallDataModelInfer::handleRequest() noexcept {
 #endif
 
   try {
-    auto request = std::make_unique<GrpcApiUnary>(this);
+    auto request = amdinfer::getRequest(request_, state_->getPool());
+    setCallback(request.get(), this);
+    auto request_container = std::make_unique<RequestContainer>();
+    request_container->request = request;
 #ifdef AMDINFER_ENABLE_TRACING
     trace->endSpan();
-    request->setTrace(std::move(trace));
+    request_container->trace = std::move(trace);
 #endif
-    state_->modelInfer(model, std::move(request));
+    state_->modelInfer(model, std::move(request_container));
   } catch (const invalid_argument& e) {
     AMDINFER_LOG_INFO(logger_, e.what());
     finish(::grpc::Status(StatusCode::NOT_FOUND, e.what()));
diff --git a/src/amdinfer/servers/http_server.cpp b/src/amdinfer/servers/http_server.cpp
index 372df6ff3..eef0ac05f 100644
--- a/src/amdinfer/servers/http_server.cpp
+++ b/src/amdinfer/servers/http_server.cpp
@@ -35,7 +35,6 @@
 #include "amdinfer/build_options.hpp"          // for AMDINFER_ENABLE_TRACING
 #include "amdinfer/clients/http_internal.hpp"  // for propagate, errorHtt...
 #include "amdinfer/core/exceptions.hpp"        // for runtime_error, inva...
-#include "amdinfer/core/interface.hpp"         // for Interface
 #include "amdinfer/core/parameters.hpp"        // for ParameterMapPtr
 #include "amdinfer/core/predict_api_internal.hpp"  // for ParameterMapPtr
 #include "amdinfer/core/shared_state.hpp"          // for SharedState
@@ -43,6 +42,8 @@
 #include "amdinfer/observation/metrics.hpp"       // for Metrics, MetricCoun...
 #include "amdinfer/observation/tracing.hpp"       // for startTrace, Trace
 #include "amdinfer/servers/websocket_server.hpp"  // for WebsocketServer
+#include "amdinfer/util/compression.hpp"          // for zDecompress
+#include "amdinfer/util/containers.hpp"           // for containerProduct
 #include "amdinfer/util/string.hpp"               // for toLower
 
 using drogon::HttpRequestPtr;
@@ -84,6 +85,94 @@ void stop() { drogon::app().quit(); }
 
 }  // namespace http
 
+std::shared_ptr<Json::Value> parseJson(const drogon::HttpRequest *req) {
+#ifdef AMDINFER_ENABLE_LOGGING
+  Logger logger{Loggers::Server};
+#endif
+
+  // attempt to get the JSON object directly first
+  const auto &json_obj = req->getJsonObject();
+  if (json_obj != nullptr) {
+    return json_obj;
+  }
+
+  AMDINFER_LOG_DEBUG(logger, "Failed to get JSON data directly");
+
+  // if it's not valid, then we need to attempt to parse the body
+  auto root = std::make_shared<Json::Value>();
+
+  std::string errors;
+  Json::CharReaderBuilder builder;
+  Json::CharReader *reader = builder.newCharReader();
+  auto body = req->getBody();
+  bool success =
+    reader->parse(body.data(), body.data() + body.size(), root.get(), &errors);
+  if (success) {
+    return root;
+  }
+
+  AMDINFER_LOG_DEBUG(logger, "Failed to interpret body as JSON data");
+
+  // if it's still not valid, attempt to uncompress the body and convert to JSON
+  auto body_decompress = util::zDecompress(body.data(), body.length());
+  success = reader->parse(body_decompress.data(),
+                          body_decompress.data() + body_decompress.size(),
+                          root.get(), &errors);
+  if (success) {
+    return root;
+  }
+
+  throw invalid_argument("Failed to interpret request body as JSON");
+}
+
+Json::Value parseResponse(InferenceResponse response) {
+  Json::Value ret;
+  ret["model_name"] = response.getModel();
+  ret["outputs"] = Json::arrayValue;
+  ret["id"] = response.getID();
+  auto outputs = response.getOutputs();
+  for (const InferenceResponseOutput &output : outputs) {
+    Json::Value json_output;
+    json_output["name"] = output.getName();
+    json_output["parameters"] = Json::objectValue;
+    json_output["data"] = Json::arrayValue;
+    json_output["shape"] = Json::arrayValue;
+    json_output["datatype"] = output.getDatatype().str();
+    const auto &shape = output.getShape();
+    for (const size_t &index : shape) {
+      json_output["shape"].append(static_cast<Json::UInt>(index));
+    }
+
+    switchOverTypes(SetInputData(), output.getDatatype(),
+                    &(json_output["data"]), output.getData(), output.getSize());
+    ret["outputs"].append(json_output);
+  }
+  return ret;
+}
+
+struct WriteData {
+  template <typename T>
+  size_t operator()(Buffer *buffer, const Json::Value &value,
+                    size_t offset) const {
+    if constexpr (std::is_same_v<T, char>) {
+      return buffer->write(jsonValueToType<T>(value), offset);
+    } else {
+      return buffer->write(static_cast<T>(jsonValueToType<T>(value)), offset);
+    }
+  }
+};
+
+drogon::HttpResponsePtr errorHttpResponse(const std::string &error,
+                                          int status_code) {
+  Json::Value ret;
+  ret["error"] = error.data();
+  auto resp = drogon::HttpResponse::newHttpJsonResponse(ret);
+  resp->setStatusCode(static_cast<drogon::HttpStatusCode>(status_code));
+  return resp;
+}
+
+using DrogonCallback = std::function<void(const drogon::HttpResponsePtr &)>;
+
 HttpServer::HttpServer(SharedState *state) : state_(state) {
   AMDINFER_LOG_DEBUG(logger_, "Constructed HttpServer");
 }
@@ -241,6 +330,148 @@ void HttpServer::hasHardware(
   callback(resp);
 }
 
+InferenceRequestInput getInput(const Json::Value &json,
+                               const MemoryPool *pool) {
+  InferenceRequestInput input;
+
+  input.setData(nullptr);
+
+  if (!json.isMember("name")) {
+    throw invalid_argument("No 'name' key present in request input");
+  }
+  input.setName(json.get("name", "").asString());
+  if (!json.isMember("shape")) {
+    throw invalid_argument("No 'shape' key present in request input");
+  }
+  auto shape = json.get("shape", Json::arrayValue);
+  std::vector<uint64_t> shape_vector;
+  shape_vector.reserve(shape.size());
+  for (auto const &i : shape) {
+    if (!i.isUInt64()) {
+      throw invalid_argument("'shape' must be specified by uint64 elements");
+    }
+    shape_vector.push_back(i.asUInt64());
+  }
+  input.setShape(shape_vector);
+
+  if (!json.isMember("datatype")) {
+    throw invalid_argument("No 'datatype' key present in request input");
+  }
+  // getting it as CString didn't work elsewhere
+  auto data_type_str = json.get("datatype", "").asString();
+  input.setDatatype(DataType(data_type_str.c_str()));
+  if (json.isMember("parameters")) {
+    auto parameters = json.get("parameters", Json::objectValue);
+    input.setParameters(mapJsonToParameters(parameters));
+  } else {
+    input.setParameters(std::make_unique<ParameterMap>());
+  }
+
+  auto size = input.getDatatype().size() * util::containerProduct(shape_vector);
+  auto buffer = pool->get({MemoryAllocators::Cpu}, size);
+  if (!json.isMember("data")) {
+    throw invalid_argument("No 'data' key present in request input");
+  }
+  auto data = json.get("data", Json::arrayValue);
+  size_t offset = 0;
+  input.setData(buffer->data(offset));
+  try {
+    for (auto const &i : data) {
+      offset = switchOverTypes(WriteData(), input.getDatatype(), buffer.get(),
+                               i, offset);
+    }
+  } catch (const Json::LogicError &) {
+    throw invalid_argument(
+      "Could not convert some data to the provided data type");
+  }
+  return input;
+}
+
+InferenceRequestOutput getOutput(const Json::Value &json) {
+  InferenceRequestOutput output;
+  output.setData(nullptr);
+  output.setName(json.get("name", "").asString());
+  if (json.isMember("parameters")) {
+    auto parameters = json.get("parameters", Json::objectValue);
+    output.setParameters(mapJsonToParameters(parameters));
+  } else {
+    output.setParameters(std::make_unique<ParameterMap>());
+  }
+  return output;
+}
+
+void setCallback(InferenceRequest *request, DrogonCallback &&drogon_callback) {
+  Callback callback = [callback = std::move(drogon_callback)](
+                        const InferenceResponse &response) {
+    drogon::HttpResponsePtr resp;
+    if (response.isError()) {
+      resp =
+        errorHttpResponse(response.getError(), HttpStatusCode::k400BadRequest);
+    } else {
+      try {
+        Json::Value ret = parseResponse(response);
+        resp = drogon::HttpResponse::newHttpJsonResponse(ret);
+      } catch (const invalid_argument &e) {
+        resp = errorHttpResponse(e.what(), HttpStatusCode::k400BadRequest);
+      }
+    }
+#ifdef AMDINFER_ENABLE_TRACING
+    const auto &context = response.getContext();
+    propagate(resp.get(), context);
+#endif
+    callback(resp);
+  };
+  request->setCallback(std::move(callback));
+}
+
+InferenceRequestPtr getRequest(const std::shared_ptr<Json::Value> &json,
+                               const MemoryPool *pool) {
+  auto request = std::make_shared<InferenceRequest>();
+
+  if (json->isMember("id")) {
+    request->setID(json->get("id", "").asString());
+  } else {
+    request->setID("");
+  }
+  if (json->isMember("parameters")) {
+    auto parameters = json->get("parameters", Json::objectValue);
+    request->setParameters(mapJsonToParameters(parameters));
+  } else {
+    request->setParameters(std::make_unique<ParameterMap>());
+  }
+
+  if (!json->isMember("inputs")) {
+    throw invalid_argument("No 'inputs' key present in request");
+  }
+  auto inputs = json->get("inputs", Json::arrayValue);
+  if (!inputs.isArray()) {
+    throw invalid_argument("'inputs' is not an array");
+  }
+
+  request->setCallback(nullptr);
+
+  const auto input_num = inputs.size();
+  for (auto i = 0U; i < input_num; ++i) {
+    const auto &input = inputs[i];
+    if (!input.isObject()) {
+      throw invalid_argument("At least one element in 'inputs' is not an obj");
+    }
+    request->addInputTensor(getInput(input, pool));
+  }
+
+  if (json->isMember("outputs")) {
+    auto outputs = json->get("outputs", Json::arrayValue);
+    const auto output_num = outputs.size();
+    for (auto i = 0U; i < output_num; ++i) {
+      const auto &json_output = outputs[i];
+
+      request->addOutputTensor(getOutput(json_output));
+    }
+  }
+
+  return request;
+}
+
 void HttpServer::modelInfer(
   const HttpRequestPtr &req,
   std::function<void(const HttpResponsePtr &)> &&callback,
@@ -260,16 +491,20 @@ void HttpServer::modelInfer(
   trace->startSpan("request_handler");
 #endif
 
+  auto json = req->getJsonObject();
   try {
-    auto request = std::make_unique<DrogonHttp>(req, std::move(callback));
+    auto request = getRequest(json, state_->getPool());
+    setCallback(request.get(), std::move(callback));
+    auto request_container = std::make_unique<RequestContainer>();
+    request_container->request = request;
 #ifdef AMDINFER_ENABLE_METRICS
-    request->setTime(now);
+    request_container->start_time = now;
 #endif
 #ifdef AMDINFER_ENABLE_TRACING
     trace->endSpan();
-    request->setTrace(std::move(trace));
+    request_container->trace = std::move(trace);
 #endif
-    state_->modelInfer(model, std::move(request));
+    state_->modelInfer(model, std::move(request_container));
   } catch (const invalid_argument &e) {
     AMDINFER_LOG_INFO(logger_, e.what());
     auto resp = errorHttpResponse(e.what(), HttpStatusCode::k400BadRequest);
diff --git a/src/amdinfer/servers/http_server.hpp b/src/amdinfer/servers/http_server.hpp
index 7085c527a..431d0479a 100644
--- a/src/amdinfer/servers/http_server.hpp
+++ b/src/amdinfer/servers/http_server.hpp
@@ -26,7 +26,8 @@
 #include <string>      // for allocator, string
 
 #include "amdinfer/build_options.hpp"  // for AMDINFER_ENABLE_HTTP, PROT...
-#include "amdinfer/observation/logging.hpp"  // for LoggerPtr
+#include "amdinfer/core/predict_api_internal.hpp"  // for InferenceRequestBuilder
+#include "amdinfer/observation/logging.hpp"        // for LoggerPtr
 
 #ifdef AMDINFER_ENABLE_HTTP
 #include <drogon/HttpController.h>  // for ADD_METHOD_TO, HttpContro...
@@ -38,9 +39,13 @@
 namespace amdinfer {
 
 class SharedState;
+class MemoryPool;
 
 #ifdef AMDINFER_ENABLE_HTTP
 
+InferenceRequestPtr getRequest(const std::shared_ptr<Json::Value> &json,
+                               const MemoryPool *pool);
+
 /**
  * @brief The HTTP server for handling REST requests extends the base
  * HttpController in Drogon and adds the endpoints of interest.
diff --git a/src/amdinfer/servers/websocket_server.cpp b/src/amdinfer/servers/websocket_server.cpp
index 78f9f6464..f7f808a87 100644
--- a/src/amdinfer/servers/websocket_server.cpp
+++ b/src/amdinfer/servers/websocket_server.cpp
@@ -24,11 +24,11 @@
 #include <string>     // for string, operator+, char_t...
 #include <utility>    // for move
 
-#include "amdinfer/clients/http_internal.hpp"      // for RequestBuilder
 #include "amdinfer/core/exceptions.hpp"            // for invalid_argument
 #include "amdinfer/core/predict_api_internal.hpp"  // for ParameterMapPtr
 #include "amdinfer/core/shared_state.hpp"          // for SharedState
 #include "amdinfer/observation/tracing.hpp"        // for startSpan, Span
+#include "amdinfer/servers/http_server.hpp"        // for RequestBuilder
 
 using drogon::HttpRequestPtr;
 using drogon::WebSocketConnectionPtr;
@@ -36,6 +36,23 @@ using drogon::WebSocketMessageType;
 
 namespace amdinfer::http {
 
+void setCallback(InferenceRequest *request,
+                 drogon::WebSocketConnectionPtr conn) {
+  Callback callback = [conn =
+                         std::move(conn)](const InferenceResponse &response) {
+    if (response.isError()) {
+      conn->send(response.getError());
+    } else {
+      auto outputs = response.getOutputs();
+      const auto *msg = static_cast<char *>(outputs[0].getData());
+      if (conn->connected()) {
+        conn->send(msg, outputs[0].getSize());
+      }
+    }
+  };
+  request->setCallback(std::move(callback));
+}
+
 WebsocketServer::WebsocketServer(SharedState *state) : state_(state) {
   AMDINFER_LOG_INFO(logger_, "Constructed WebsocketServer");
 }
@@ -81,14 +98,17 @@ void WebsocketServer::handleNewMessage(const WebSocketConnectionPtr &conn,
   std::transform(model.begin(), model.end(), model.begin(),
                  [](unsigned char c) { return std::tolower(c); });
 
-  auto request = std::make_unique<DrogonWs>(conn, std::move(json));
+  auto request = getRequest(json, state_->getPool());
+  setCallback(request.get(), conn);
+  auto request_container = std::make_unique<RequestContainer>();
+  request_container->request = request;
 #ifdef AMDINFER_ENABLE_TRACING
   trace->endSpan();
-  request->setTrace(std::move(trace));
+  request_container->trace = std::move(trace);
 #endif
 
   try {
-    state_->modelInfer(model, std::move(request));
+    state_->modelInfer(model, std::move(request_container));
   } catch (const runtime_error &e) {
     AMDINFER_LOG_INFO(logger_, e.what());
     conn->shutdown(drogon::CloseCode::kInvalidMessage, e.what());
@@ -110,82 +130,4 @@ void WebsocketServer::handleNewConnection(const HttpRequestPtr &req,
   (void)req;   // suppress unused variable warning
 }
 
-DrogonWs::DrogonWs(const drogon::WebSocketConnectionPtr &conn,
-                   std::shared_ptr<Json::Value> json) {
-  this->conn_ = conn;
-  this->type_ = InterfaceType::DrogonWs;
-  this->json_ = std::move(json);
-}
-
-size_t DrogonWs::getInputSize() {
-  auto inputs = this->json_->get("inputs", Json::arrayValue);
-  if (!inputs.isArray()) {
-    throw invalid_argument("'inputs' is not an array");
-  }
-  return inputs.size();
-}
-
-std::vector<size_t> DrogonWs::getInputSizes() const {
-  std::vector<size_t> sizes;
-
-  if (!this->json_->isMember("inputs")) {
-    throw invalid_argument("No 'inputs' key present in request");
-  }
-  auto inputs = this->json_->get("inputs", Json::arrayValue);
-  if (!inputs.isArray()) {
-    throw invalid_argument("'inputs' is not an array");
-  }
-
-  for (const auto &tensor : inputs) {
-    // using asCString() doesn't work -> the string is empty
-    const auto raw_type = tensor.get("datatype", "UNKNOWN").asString();
-    auto datatype = DataType(raw_type.c_str());
-
-    const auto shape = tensor.get("shape", Json::arrayValue);
-    size_t size = 1;
-    for (const auto &index : shape) {
-      size *= index.asInt64();
-    }
-
-    sizes.push_back(size * datatype.size());
-  }
-  return sizes;
-}
-
-std::shared_ptr<InferenceRequest> DrogonWs::getRequest(
-  const BufferRawPtrs &input_buffers, std::vector<size_t> &input_offset,
-  const BufferRawPtrs &output_buffers, std::vector<size_t> &output_offset) {
-#ifdef AMDINFER_ENABLE_LOGGING
-  const auto &logger = this->getLogger();
-#endif
-  try {
-    auto request = RequestBuilder::build(
-      this->json_, input_buffers, input_offset, output_buffers, output_offset);
-    Callback callback =
-      [conn = std::move(this->conn_)](const InferenceResponse &response) {
-        if (response.isError()) {
-          conn->send(response.getError());
-        } else {
-          auto outputs = response.getOutputs();
-          const auto *msg = static_cast<char *>(outputs[0].getData());
-          if (conn->connected()) {
-            conn->send(msg, outputs[0].getSize());
-          }
-        }
-      };
-    request->setCallback(std::move(callback));
-    return request;
-  } catch (const invalid_argument &e) {
-    AMDINFER_LOG_INFO(logger, e.what());
-    this->conn_->shutdown(drogon::CloseCode::kUnexpectedCondition,
-                          "Failed to create request");
-    return nullptr;
-  }
-}
-
-void DrogonWs::errorHandler(const std::exception &e) {
-  AMDINFER_LOG_INFO(this->getLogger(), e.what());
-  this->conn_->shutdown(drogon::CloseCode::kUnexpectedCondition, e.what());
-}
-
 }  // namespace amdinfer::http
diff --git a/src/amdinfer/servers/websocket_server.hpp b/src/amdinfer/servers/websocket_server.hpp
index 7f751846b..fc2e098b3 100644
--- a/src/amdinfer/servers/websocket_server.hpp
+++ b/src/amdinfer/servers/websocket_server.hpp
@@ -28,7 +28,6 @@
 #include <vector>     // for vector
 
 #include "amdinfer/build_options.hpp"        // for AMDINFER_ENABLE_LOGGING
-#include "amdinfer/core/interface.hpp"       // for Interface
 #include "amdinfer/declarations.hpp"         // for BufferRawPtrs
 #include "amdinfer/observation/logging.hpp"  // for LoggerPtr
 
@@ -43,32 +42,6 @@ class SharedState;
 
 namespace amdinfer::http {
 
-/**
- * @brief The DrogonWs Interface class encapsulates incoming requests from
- * Drogon's Websocket interface to the batcher.
- *
- */
-class DrogonWs : public Interface {
- public:
-  DrogonWs(const drogon::WebSocketConnectionPtr &conn,
-           std::shared_ptr<Json::Value> json);
-
-  std::shared_ptr<InferenceRequest> getRequest(
-    const BufferRawPtrs &input_buffers, std::vector<size_t> &input_offsets,
-    const BufferRawPtrs &output_buffers,
-    std::vector<size_t> &output_offsets) override;
-
-  size_t getInputSize() override;
-  std::vector<size_t> getInputSizes() const override;
-  void errorHandler(const std::exception &e) override;
-
- private:
-  void setJson();
-
-  std::shared_ptr<Json::Value> json_;
-  drogon::WebSocketConnectionPtr conn_;
-};
-
 /**
  * @brief The Websocket server handles incoming websocket requests. Currently,
  * this is primarily used to make inferences to video streaming workers.


Test patch:
diff --git a/tests/performance/batching/CMakeLists.txt b/tests/performance/batching/CMakeLists.txt
index 0371c8cca..071685fa8 100644
--- a/tests/performance/batching/CMakeLists.txt
+++ b/tests/performance/batching/CMakeLists.txt
@@ -18,10 +18,10 @@ list(APPEND tests soft)
 list(
   APPEND
     tests_libs
-    "$<TARGET_OBJECTS:batcher>~fake_observation~interface\
+    "$<TARGET_OBJECTS:batcher>~fake_observation\
       ~$<TARGET_OBJECTS:fake_worker_info_buffers_finite>~predict_api~parameters\
       ~data_types\~$<TARGET_OBJECTS:softBatcher>~cpuBuffer~buffer~memory_pool\
-      ~batch~$<TARGET_OBJECTS:native_internal>~timer"
+      ~batch~timer"
 )
 
 amdinfer_add_benchmarks("${tests}" "${tests_libs}")
diff --git a/tests/performance/batching/benchmark_soft.cpp b/tests/performance/batching/benchmark_soft.cpp
index 74ba9c36a..a2b86d600 100644
--- a/tests/performance/batching/benchmark_soft.cpp
+++ b/tests/performance/batching/benchmark_soft.cpp
@@ -30,15 +30,15 @@
 #include <utility>   // for move
 #include <vector>    // for vector
 
-#include "amdinfer/batching/soft.hpp"            // for BatchPtr, SoftBatcher
-#include "amdinfer/clients/native_internal.hpp"  // for CppNativeApi
-#include "amdinfer/core/data_types.hpp"          // for DataType, DataType::U...
-#include "amdinfer/core/memory_pool/pool.hpp"    // for MemoryPool
-#include "amdinfer/core/parameters.hpp"          // for ParameterMap
-#include "amdinfer/core/predict_api.hpp"         // for InferenceRequest, Req...
-#include "amdinfer/core/worker_info.hpp"         // for WorkerInfo
-#include "amdinfer/declarations.hpp"             // for BufferPtrs
-#include "amdinfer/observation/logging.hpp"      // for LogOptions, initLogger
+#include "amdinfer/batching/soft.hpp"          // for BatchPtr, SoftBatcher
+#include "amdinfer/core/data_types.hpp"        // for DataType, DataType::U...
+#include "amdinfer/core/memory_pool/pool.hpp"  // for MemoryPool
+#include "amdinfer/core/parameters.hpp"        // for ParameterMap
+#include "amdinfer/core/predict_api.hpp"       // for InferenceRequest, Req...
+#include "amdinfer/core/predict_api_internal.hpp"  // for RequestContainer
+#include "amdinfer/core/worker_info.hpp"           // for WorkerInfo
+#include "amdinfer/declarations.hpp"               // for BufferPtrs
+#include "amdinfer/observation/logging.hpp"        // for LogOptions, initLogger
 #include "gtest/gtest.h"
 
 namespace amdinfer {
@@ -49,7 +49,8 @@ class PerfSoftBatcherFixture : public ::benchmark::Fixture {
  public:
   void enqueue(int count) {
     for (auto i = 0; i < count; ++i) {
-      auto req = std::make_unique<CppNativeApi>(this->request_);
+      auto req = std::make_unique<RequestContainer>();
+      req->request = this->request_;
       batcher_->enqueue(std::move(req));
     }
   }
@@ -103,9 +104,9 @@ class PerfSoftBatcherFixture : public ::benchmark::Fixture {
       data_[i] = static_cast<uint8_t>(i);
     }
 
-    this->request_ = InferenceRequest();
-    this->request_.addInputTensor(static_cast<void*>(data_.data()), data_shape,
-                                  DataType::Uint8);
+    request_ = std::make_shared<InferenceRequest>();
+    request_->addInputTensor(static_cast<void*>(data_.data()), data_shape,
+                             DataType::Uint8);
   }
 
   void TearDown([[maybe_unused]] const ::benchmark::State& state) override {
@@ -115,7 +116,7 @@ class PerfSoftBatcherFixture : public ::benchmark::Fixture {
 
  private:
   std::vector<uint8_t> data_;
-  InferenceRequest request_;
+  InferenceRequestPtr request_;
   std::optional<WorkerInfo> worker_;
   std::optional<SoftBatcher> batcher_;
   MemoryPool pool_;
diff --git a/tests/performance/models/benchmark_resnet50.cpp b/tests/performance/models/benchmark_resnet50.cpp
index 09a2648f9..2d9bfb58a 100644
--- a/tests/performance/models/benchmark_resnet50.cpp
+++ b/tests/performance/models/benchmark_resnet50.cpp
@@ -165,19 +165,23 @@ void resnet50(benchmark::State& st, Args&&... args) {
   parameters.put("output_classes", output_classes);
   parameters.put("batch_size", batch_size);
 
-  const auto default_http_port = 8998;
-  const auto default_grpc_port = 50'051;
+  [[maybe_unused]] const auto default_http_port = 8998;
+  [[maybe_unused]] const auto default_grpc_port = 50'051;
 
   amdinfer::Server server;
   std::unique_ptr<amdinfer::Client> client;
   if (client_index == 0) {
     client = std::make_unique<amdinfer::NativeClient>(&server);
+#ifdef AMDINFER_ENABLE_HTTP
   } else if (client_index == 1) {
     server.startHttp(default_http_port);
     client = std::make_unique<amdinfer::HttpClient>("http://127.0.0.1:8998");
+#endif
+#ifdef AMDINFER_ENABLE_GRPC
   } else if (client_index == 2) {
     server.startGrpc(default_grpc_port);
     client = std::make_unique<amdinfer::GrpcClient>("127.0.0.1:50051");
+#endif
   } else {
     st.SkipWithError("Unknown client index passed");
     return;
diff --git a/tests/src/amdinfer/CMakeLists.txt b/tests/src/amdinfer/CMakeLists.txt
index 992ebb034..bfc06400f 100644
--- a/tests/src/amdinfer/CMakeLists.txt
+++ b/tests/src/amdinfer/CMakeLists.txt
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-add_subdirectory(batching)
+# add_subdirectory(batching)
 # add_subdirectory(buffers)
 # add_subdirectory(clients)
 add_subdirectory(core)
diff --git a/tests/src/amdinfer/batching/CMakeLists.txt b/tests/src/amdinfer/batching/CMakeLists.txt
deleted file mode 100644
index dad8418d5..000000000
--- a/tests/src/amdinfer/batching/CMakeLists.txt
+++ /dev/null
@@ -1,22 +0,0 @@
-# Copyright 2022 Xilinx, Inc.
-# Copyright 2022 Advanced Micro Devices, Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-add_library(fakeBatcher OBJECT batcher.cpp)
-target_include_directories(
-  fakeBatcher BEFORE PRIVATE ${AMDINFER_TEST_INCLUDE_DIRS}
-)
-target_include_directories(fakeBatcher AFTER PRIVATE ${AMDINFER_INCLUDE_DIRS})
-target_link_libraries(fakeBatcher INTERFACE fake_observation Threads::Threads)
-set_target_options(fakeBatcher)
diff --git a/tests/src/amdinfer/batching/batcher.cpp b/tests/src/amdinfer/batching/batcher.cpp
deleted file mode 100644
index ca241dfb9..000000000
--- a/tests/src/amdinfer/batching/batcher.cpp
+++ /dev/null
@@ -1,151 +0,0 @@
-// Copyright 2021 Xilinx, Inc.
-// Copyright 2022 Advanced Micro Devices, Inc.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-/**
- * @file
- * @brief Implements the fake batcher
- */
-
-#include "amdinfer/batching/batcher.hpp"
-
-#include <cstddef>    // for size_t
-#include <exception>  // for exception
-#include <future>     // for promise
-#include <memory>     // for shared_ptr, unique_ptr
-#include <string>     // for operator+, char_traits
-#include <utility>    // for move
-#include <vector>     // for vector
-
-#include "amdinfer/build_options.hpp"          // for AMDINFER_ENABLE_TRACING
-#include "amdinfer/core/fake_predict_api.hpp"  // for FakeInferenceRequest
-#include "amdinfer/core/interface.hpp"         // for Interface
-#include "amdinfer/core/predict_api.hpp"       // for InferenceResponsePromis...
-#include "amdinfer/observation/logging.hpp"    // for Logger, AMDINFER_LOG_DEBUG
-#include "amdinfer/observation/tracing.hpp"    // for Trace
-#include "amdinfer/util/queue.hpp"             // for BlockingConcurrentQueue
-#include "amdinfer/util/thread.hpp"            // for setThreadName
-
-// IWYU pragma: no_forward_declare amdinfer::Buffer
-
-namespace amdinfer {
-
-// /**
-//  * @brief For testing purposes, this fake C++ interface returns a
-//  * fakeInferenceRequest object similar to how the real one does.
-//  *
-//  */
-// class FakeInterface : public Interface {
-//  public:
-//   explicit FakeInterface(InferenceRequestInput request);
-
-//   std::shared_ptr<InferenceRequest> getRequest(
-//     const BufferRawPtrs &input_buffers, std::vector<size_t> &input_offsets,
-//     const BufferRawPtrs &output_buffers,
-//     std::vector<size_t> &output_offsets) override;
-
-//   size_t getInputSize() override;
-//   void errorHandler(const std::exception &e) override;
-//   std::promise<amdinfer::InferenceResponse> *getPromise();
-
-//  private:
-//   InferenceRequestInput request_;
-//   InferenceResponsePromisePtr promise_;
-// };
-
-// FakeInterface::FakeInterface(InferenceRequestInput request)
-//   : request_(std::move(request)) {
-//   this->promise_ =
-//     std::make_unique<std::promise<amdinfer::InferenceResponse>>();
-// }
-
-// size_t FakeInterface::getInputSize() { return 1; }
-
-// std::promise<amdinfer::InferenceResponse> *FakeInterface::getPromise() {
-//   return this->promise_.get();
-// }
-
-// void fakeCppCallback(const InferenceResponsePromisePtr &promise,
-//                      const InferenceResponse &response) {
-//   promise->set_value(response);
-// }
-
-// std::shared_ptr<InferenceRequest> FakeInterface::getRequest(
-//   const BufferRawPtrs &input_buffers, std::vector<size_t> &input_offsets,
-//   const BufferRawPtrs &output_buffers, std::vector<size_t> &output_offsets) {
-//   auto request = std::make_shared<FakeInferenceRequest>(
-//     this->request_, input_buffers, input_offsets, output_buffers,
-//     output_offsets);
-//   Callback callback = [this](const InferenceResponse &response) {
-//     fakeCppCallback(this->promise_, response);
-//   };
-//   request->setCallback(std::move(callback));
-//   return request;
-// }
-
-// void FakeInterface::errorHandler(const std::exception &e) {
-//   AMDINFER_LOG_ERROR(this->getLogger(), e.what());
-//   (void)e;  // suppress unused variable warning
-// }
-
-// void Batcher::run(WorkerInfo *worker) {
-//   auto thread_name = "batch" + this->getName();
-//   util::setThreadName(thread_name);
-//   InterfacePtr req;
-//   bool run = true;
-
-//   while (run) {
-//     auto batch = std::make_unique<Batch>(worker);
-//     auto input_buffers = batch->getRawInputBuffers();
-//     auto output_buffers = batch->getRawOutputBuffers();
-//     std::vector<size_t> input_offset(input_buffers.size(), 0);
-//     std::vector<size_t> output_offset(output_buffers.size(), 0);
-
-//     // wait for the first request
-//     this->input_queue_->wait_dequeue(req);
-//     AMDINFER_LOG_DEBUG(
-//       logger_, "Got initial request of a new batch for " + this->model_);
-
-//     if (req == nullptr) {
-//       break;
-//     }
-
-// #ifdef AMDINFER_ENABLE_TRACING
-//     auto trace = req->getTrace();
-//     trace->startSpan("fake_batcher");
-// #endif
-//     req->getInputSize();  // initialize the req object
-//     auto new_req = req->getRequest(input_buffers, input_offset,
-//     output_buffers,
-//                                    output_offset);
-
-//     batch->addRequest(new_req);
-
-// #ifdef AMDINFER_ENABLE_TRACING
-//     trace->endSpan();
-//     batch->addTrace(std::move(trace));
-// #endif
-// #ifdef AMDINFER_ENABLE_METRICS
-//     batch->addTime(req->getTime());
-// #endif
-//     // batch_size += 1;
-
-//     if (!batch->empty()) {
-//       AMDINFER_LOG_DEBUG(logger_, "Enqueuing batch for " + this->model_);
-//       this->output_queue_->enqueue(std::move(batch));
-//     }
-//   }
-// }
-
-}  // namespace amdinfer
diff --git a/tests/src/amdinfer/core/CMakeLists.txt b/tests/src/amdinfer/core/CMakeLists.txt
index 61f9120cd..e2c1166df 100644
--- a/tests/src/amdinfer/core/CMakeLists.txt
+++ b/tests/src/amdinfer/core/CMakeLists.txt
@@ -18,7 +18,7 @@ list(
          predict_api
          worker_info_buffers_infinite
          worker_info_buffers_finite
-         # data_types interface
+         # data_types
 )
 
 foreach(target ${targets})
diff --git a/tests/unit/batching/CMakeLists.txt b/tests/unit/batching/CMakeLists.txt
index 4df173f35..5e3c2a12d 100644
--- a/tests/unit/batching/CMakeLists.txt
+++ b/tests/unit/batching/CMakeLists.txt
@@ -18,13 +18,13 @@ list(APPEND tests soft soft_batching)
 list(
   APPEND
     tests_libs
-    "$<TARGET_OBJECTS:batcher>~fake_observation~interface~\
+    "$<TARGET_OBJECTS:batcher>~fake_observation~\
       $<TARGET_OBJECTS:fake_worker_info_buffers_infinite>~predict_api~\
       parameters~data_types~$<TARGET_OBJECTS:softBatcher>~timer~memory_pool~batch~cpuBuffer~buffer"
-    "$<TARGET_OBJECTS:batcher>~fake_observation~interface~\
+    "$<TARGET_OBJECTS:batcher>~fake_observation~\
       $<TARGET_OBJECTS:fake_worker_info_buffers_finite>~predict_api~data_types~\
       parameters~$<TARGET_OBJECTS:softBatcher>~cpuBuffer~buffer~\
-      $<TARGET_OBJECTS:native_internal>~timer~memory_pool~batch"
+      timer~memory_pool~batch"
 )
 
 amdinfer_add_unit_tests("${tests}" "${tests_libs}")
diff --git a/tests/unit/batching/test_soft.cpp b/tests/unit/batching/test_soft.cpp
index 9db61f55d..9b5ec86fc 100644
--- a/tests/unit/batching/test_soft.cpp
+++ b/tests/unit/batching/test_soft.cpp
@@ -17,9 +17,9 @@
 
 #include "amdinfer/batching/soft.hpp"          // for SoftBatcher
 #include "amdinfer/build_options.hpp"          // for AMDINFER_ENABLE_LOGGING
-#include "amdinfer/core/interface.hpp"         // IWYU pragma: keep
 #include "amdinfer/core/memory_pool/pool.hpp"  // for MemoryPool
-#include "amdinfer/core/worker_info.hpp"       // for WorkerInfo
+#include "amdinfer/core/predict_api_internal.hpp"  // for InferenceRequestInput
+#include "amdinfer/core/worker_info.hpp"           // for WorkerInfo
 #include "amdinfer/observation/logging.hpp"  // for initLogger, LogLevel, Log...
 #include "gtest/gtest.h"                     // for Test, SuiteApiResolver, TEST
 
diff --git a/tests/unit/batching/test_soft_batching.cpp b/tests/unit/batching/test_soft_batching.cpp
index 5c908e893..58c7ef008 100644
--- a/tests/unit/batching/test_soft_batching.cpp
+++ b/tests/unit/batching/test_soft_batching.cpp
@@ -23,17 +23,17 @@
 #include <utility>           // for move
 #include <vector>            // for vector
 
-#include "amdinfer/batching/soft.hpp"            // for BatchPtr, SoftBatcher
-#include "amdinfer/build_options.hpp"            // for AMDINFER_ENABLE_LOGGING
-#include "amdinfer/clients/native_internal.hpp"  // for CppNativeApi
-#include "amdinfer/core/data_types.hpp"          // for DataType, DataType::U...
-#include "amdinfer/core/memory_pool/pool.hpp"    // for MemoryPool
-#include "amdinfer/core/parameters.hpp"          // for ParameterMap
-#include "amdinfer/core/predict_api.hpp"         // for InferenceRequest, Req...
-#include "amdinfer/core/worker_info.hpp"         // for WorkerInfo
-#include "amdinfer/declarations.hpp"             // for BufferPtrs
-#include "amdinfer/observation/logging.hpp"      // for initLogger, LogLevel
-#include "gtest/gtest.h"                         // for ParamIteratorInterface
+#include "amdinfer/batching/soft.hpp"          // for BatchPtr, SoftBatcher
+#include "amdinfer/buffers/cpu.hpp"            // for CpuBuffer
+#include "amdinfer/build_options.hpp"          // for AMDINFER_ENABLE_LOGGING
+#include "amdinfer/core/data_types.hpp"        // for DataType, DataType::U...
+#include "amdinfer/core/memory_pool/pool.hpp"  // for MemoryPool
+#include "amdinfer/core/parameters.hpp"        // for ParameterMap
+#include "amdinfer/core/predict_api_internal.hpp"  // for InferenceRequest, Req...
+#include "amdinfer/core/worker_info.hpp"           // for WorkerInfo
+#include "amdinfer/declarations.hpp"               // for BufferPtrs
+#include "amdinfer/observation/logging.hpp"        // for initLogger, LogLevel
+#include "gtest/gtest.h"                           // for ParamIteratorInterface
 
 namespace amdinfer {
 
@@ -111,9 +111,12 @@ class UnitSoftBatcherFixture : public testing::TestWithParam<BatchConfig> {
 
     this->batcher_->start({MemoryAllocators::Cpu});
 
-    data_.resize(data_size);
+    buffer_ = pool_.get({MemoryAllocators::Cpu}, data_size);
+
+    auto* data = static_cast<uint8_t*>(buffer_->data(0));
+
     for (uint8_t i = 0; i < data_size; i++) {
-      data_[i] = i;
+      data[i] = i;
     }
   }
 
@@ -156,30 +159,24 @@ class UnitSoftBatcherFixture : public testing::TestWithParam<BatchConfig> {
           compareData(buffer.get(), j * data_size_);
         }
       }
-
-      const auto& reqs = batch->getRequests();
-      for (const auto& req : reqs) {
-        req->runCallback(InferenceResponse());
-      }
     }
   }
 
-  void enqueue(std::unique_ptr<CppNativeApi> req) {
+  void enqueue(std::unique_ptr<RequestContainer> req) {
     batcher_->enqueue(std::move(req));
   }
 
-  InferenceRequest createRequest() {
-    InferenceRequest request;
-    request.addInputTensor(static_cast<void*>(data_.data()), data_shape_,
-                           DataType::Uint8);
+  InferenceRequestPtr createRequest() {
+    auto request = std::make_shared<InferenceRequest>();
+    request->addInputTensor(buffer_->data(0), data_shape_, DataType::Uint8);
     return request;
   }
 
  private:
   int data_size_ = 0;
-  std::vector<uint8_t> data_;
+  BufferPtr buffer_;
   std::initializer_list<uint64_t> data_shape_;
-  InferenceRequest request_;
+  InferenceRequestPtr request_;
   std::optional<WorkerInfo> worker_;
   std::optional<SoftBatcher> batcher_;
   MemoryPool pool_;
@@ -191,8 +188,8 @@ TEST_P(UnitSoftBatcherFixture, BasicBatching) {  // NOLINT
 
   for (const auto& i : requests) {
     for (auto j = 0; j < i; ++j) {
-      auto request = createRequest();
-      auto req = std::make_unique<CppNativeApi>(request);
+      auto req = std::make_unique<RequestContainer>();
+      req->request = createRequest();
       this->enqueue(std::move(req));
     }
   }