Repository: Xilinx/inference-server (language: python)

Issue:
Support model chaining in modelLoad
#176 adds model chaining support using the C++ API but it's not supported through `modelLoad` using the repository format
Add model chaining
Chaining should let a user define a pipeline of steps that could include pre- and post-processing steps to run end-to-end inference or send the output of one model to another.


Discussion / hints:

Fix patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index cdddba642..10f879ed7 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -32,6 +32,7 @@ Added
 * Google Benchmark for writing performance-tracking tests (:pr:`147`)
 * Custom memory storage classes in the memory pool (:pr:`166`)
 * C++ worker for executing C++ "models" (:pr:`172`)
+* Model chaining (:pr:`176`) and loading chains from ``modelLoad`` ()
 
 Changed
 ^^^^^^^
@@ -44,6 +45,8 @@ Changed
 * Bump up to Vitis AI 3.0 (:pr:`169`)
 * Refactor inference request objects and tensors (:pr:`172`)
 * Use const references throughout for ParameterMap (:pr:`172`)
+* Update workers' ``doRun`` method signature to produce and return a batch (:pr:`176`)
+* Use TOML-based configuration files in the repository by default ()
 
 Deprecated
 ^^^^^^^^^^
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 17fc5d2b4..93a473984 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -73,6 +73,7 @@ include(CTest)
 include(SetTargetOptions)
 include(AddOption)
 include(AddTargets)
+include(TomlPlusPlus)
 
 # check requirements
 find_package(Drogon CONFIG)
diff --git a/cmake/AddTargets.cmake b/cmake/AddTargets.cmake
index 96b05dbce..0d0b44e31 100644
--- a/cmake/AddTargets.cmake
+++ b/cmake/AddTargets.cmake
@@ -26,6 +26,7 @@ function(amdinfer_add_protobuf_target target file build_grpc)
   target_include_directories(${target} PRIVATE ${CMAKE_CURRENT_BINARY_DIR})
   set_target_options(${target})
   protobuf_generate(TARGET ${target} LANGUAGE cpp)
+  target_link_libraries(${target} INTERFACE protobuf::libprotobuf)
   if(${build_grpc})
     protobuf_generate(
       TARGET ${target}
diff --git a/cmake/TomlPlusPlus.cmake b/cmake/TomlPlusPlus.cmake
new file mode 100644
index 000000000..66bcfd888
--- /dev/null
+++ b/cmake/TomlPlusPlus.cmake
@@ -0,0 +1,20 @@
+# Copyright 2023 Advanced Micro Devices, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+include(FetchContent)
+FetchContent_Declare(
+  tomlplusplus GIT_REPOSITORY https://github.com/marzer/tomlplusplus.git
+  GIT_TAG v3.3.0
+)
+FetchContent_MakeAvailable(tomlplusplus)
diff --git a/docs/kserve.rst b/docs/kserve.rst
index 246ba46b1..08e48546c 100644
--- a/docs/kserve.rst
+++ b/docs/kserve.rst
@@ -63,34 +63,32 @@ The model format for the AMD Inference Server is the following:
     /
     ├─ model_a/
     │  ├─ 1/
-    │  │  ├─ saved_model.x
-    │  ├─ config.pbtxt
+    │  │  ├─ <model>.x
+    │  ├─ <config>.toml
 
 The model name, ``model_a`` in this template, must be unique among the models loaded on a particular server.
 This name is used to name the endpoint used to make inference requests to.
-Under this directory, there must be a directory named ``1/`` containing the model file itself and a text file named ``config.pbtxt``.
-The model file must be named ``saved_model`` and the file extension depends on the type of the model.
-The ``config.pbtxt`` file contains metadata for the model.
+Under this directory, there must be a directory named ``1/`` containing the model file itself and a TOML file describing the configuration.
+This file, ``<config>.toml`` in this template, can have any name though ``config.toml`` is suggested and will be used in this documentation.
+You can also use ``.pbtxt`` format for single models as well.
+The model file can have an arbitrary name and the file extension depends on the type of the model.
+The ``config.toml`` file contains metadata for the model.
 Consider this example of an MNIST TensorFlow model:
 
-.. code-block:: text
+.. code-block:: toml
+
+    name = "mnist"
+    platform = "tensorflow_graphdef"
+
+    [[inputs]]
+    name = "images_in"
+    datatype = "FP32"
+    shape = [28, 28, 1]
 
-    name: "mnist"
-    platform: "tensorflow_graphdef"
-    inputs [
-      {
-        name: "images_in"
-        datatype: "FP32"
-        shape: [28,28,1]
-      }
-    ]
-    outputs [
-      {
-        name: "flatten/Reshape"
-        datatype: "FP32"
-        shape: [10]
-      }
-    ]
+    [[outputs]]
+    name = "flatten/Reshape"
+    datatype = "FP32"
+    shape = [10]
 
 The name must match the name of the model directory, i.e. ``model_a``.
 The platform identifies the type of the model and determines the file extension of the model file.
@@ -105,6 +103,8 @@ The supported platforms are:
     ``pytorch_torchscript``,``.pt``
     ``vitis_xmodel``,``.xmodel``
     ``onnx_onnxv1``,``.onnx``
+    ``migraphx_mxr``,``.mxr``
+    ``amdinfer_cpp``,``.so``
 
 The inputs and outputs define the list of input and output tensors for the model.
 The names of the tensors may be significant if the platform needs them to perform inference.
diff --git a/docs/quickstart_deployment.rst b/docs/quickstart_deployment.rst
index 329f3e046..7c9f923eb 100644
--- a/docs/quickstart_deployment.rst
+++ b/docs/quickstart_deployment.rst
@@ -39,16 +39,16 @@ The format of the directory is as follows:
     /
     ├─ model_a/
     │  ├─ 1/
-    │  │  ├─ saved_model.x
-    │  ├─ config.pbtxt
+    │  │  ├─ <model>.x
+    │  ├─ config.toml
     | model_b/
     |  ...
 
 The model name, ``model_a`` in this template, must be unique among the models loaded on a particular server.
 This name is used to name the endpoint used to make inference requests to.
-Under this directory, there must be a directory named ``1/`` containing the model file itself and a text file named ``config.pbtxt``.
-The model file must be named ``saved_model`` and the file extension depends on the type of the model.
-The ``config.pbtxt`` file contains metadata for the model.
+Under this directory, there must be a directory named ``1/`` containing the model file itself and a TOML file containing the metadata for the model.
+This file may be named anything but ``config.toml`` is suggested and used throughout this documentation.
+The model file can have an arbitrary name and the file extension depends on the type of the model.
 
 As an example, consider a deployment of a single ResNet50 model.
 
@@ -74,66 +74,54 @@ As an example, consider a deployment of a single ResNet50 model.
         $ mkdir -p /tmp/model_repository/resnet50/1
         $ mv ./resnet_v1_50_tf/resnet_v1_50_tf.xmodel /tmp/model_repository/resnet50/1/saved_model.xmodel
 
-For the models used here, their corresponding ``config.pbtxt`` should be placed in the chosen model repository (``/tmp/model_repository/resnet50/``):
+For the models used here, their corresponding ``config.toml`` should be placed in the chosen model repository (``/tmp/model_repository/resnet50/``):
 
 .. tabs::
 
-    .. code-tab:: text CPU
-
-        name: "resnet50"
-        platform: "tensorflow_graphdef"
-        inputs [
-            {
-                name: "input"
-                datatype: "FP32"
-                shape: [224,224,3]
-            }
-        ]
-        outputs [
-            {
-                name: "resnet_v1_50/predictions/Reshape_1"
-                datatype: "FP32"
-                shape: [1000]
-            }
-        ]
+    .. code-tab:: toml CPU
+
+        name = "resnet50"
+        platform = "tensorflow_graphdef"
+
+        [[inputs]]
+        name = "input"
+        datatype = "FP32"
+        shape = [224, 224, 3]
+
+        [[outputs]]
+        name = "resnet_v1_50/predictions/Reshape_1"
+        datatype = "FP32"
+        shape = [1000]
 
     .. code-tab:: text GPU
 
-        name: "resnet50"
-        platform: "onnx_onnxv1"
-        inputs [
-            {
-                name: "input"
-                datatype: "FP32"
-                shape: [224,224,3]
-            }
-        ]
-        outputs [
-            {
-                name: "output"
-                datatype: "FP32"
-                shape: [1000]
-            }
-        ]
+        name = "resnet50"
+        platform = "onnx_onnxv1"
+
+        [[inputs]]
+        name = "input"
+        datatype = "FP32"
+        shape = [224, 224, 3]
+
+        [[outputs]]
+        name = "output"
+        datatype = "FP32"
+        shape = [1000]
 
     .. code-tab:: console FPGA
 
-        name: "resnet50"
-        platform: "vitis_xmodel"
-        inputs [
-            {
-                name: "input"
-                datatype: "INT8"
-                shape: [224,224,3]
-            }
-        ]
-        outputs [
-            {
-                name: "output"
-                datatype: "INT8"
-                shape: [1000]
-            }
-        ]
+        name = "resnet50"
+        platform = "vitis_xmodel"
+
+        [[inputs]]
+        name = "input"
+        datatype = "INT8"
+        shape = [224, 224, 3]
+
+        [[outputs]]
+        name = "output"
+        datatype = "INT8"
+        shape = [1000]
 
 The name must match the name of the model directory.
 The platform identifies the type of the model and determines the file extension of the model file.
@@ -148,6 +136,8 @@ The supported platforms are:
     ``pytorch_torchscript``,``.pt``
     ``vitis_xmodel``,``.xmodel``
     ``onnx_onnxv1``,``.onnx``
+    ``migraphx_mxr``,``.mxr``
+    ``amdinfer_cpp``,``.so``
 
 The inputs and outputs define the list of input and output tensors for the model.
 The names of the tensors may be significant if the platform needs them to perform inference.
@@ -171,7 +161,7 @@ You can pull the deployment image with Docker if it exists or :ref:`build it you
 
     .. code-tab:: console FPGA
 
-        # this image is not currently pre-built
+        # this image is not currently pre-built but you can build it yourself
 
 Start the image
 ---------------
@@ -202,6 +192,6 @@ By default, the container will start the server executable in the container that
 The ``--publish`` flags will map ports 8998 and 50051 in the container to arbitrary free ports on the host machine for HTTP and gRPC requests, respectively.
 You can use ``docker ps`` to show the running containers and what ports on the host machine are used by the container.
 Your clients will need these port numbers to make requests to the server.
-The endpoints for each model will be the name of the model in the ``config.pbtxt``, which should match the name of the parent directory in the model repository.
+The endpoints for each model will be the name of the model in the ``config.toml``, which should match the name of the parent directory in the model repository.
 In this example, it would be "resnet50".
 Once the container is started, you and any clients can :ref:`make requests to it <quickstart_inference:Quickstart - Inference>`.
diff --git a/examples/resnet50/quickstart-setup.sh b/examples/resnet50/quickstart-setup.sh
index 0058a0673..e00e2a283 100755
--- a/examples/resnet50/quickstart-setup.sh
+++ b/examples/resnet50/quickstart-setup.sh
@@ -22,23 +22,20 @@ wget -O tensorflow.zip https://www.xilinx.com/bin/public/openDownload?filename=t
 unzip -j "tensorflow.zip" "tf_resnetv1_50_imagenet_224_224_6.97G_2.5/float/resnet_v1_50_baseline_6.96B_922.pb" -d .
 mkdir -p model_repository/resnet50/1
 mv ./resnet_v1_50_baseline_6.96B_922.pb model_repository/resnet50/1/saved_model.pb
-echo 'name: "resnet50"
-platform: "tensorflow_graphdef"
-inputs [
-    {
-        name: "input"
-        datatype: "FP32"
-        shape: [224,224,3]
-    }
-]
-outputs [
-    {
-        name: "resnet_v1_50/predictions/Reshape_1"
-        datatype: "FP32"
-        shape: [1000]
-    }
-]' > model_repository/resnet50/config.pbtxt
+cat << EOF > model_repository/resnet50/config.toml
+name = "mnist"
+platform = "tensorflow_graphdef"
 
+[[inputs]]
+name = "images_in"
+datatype = "FP32"
+shape = [28, 28, 1]
+
+[[outputs]]
+name = "flatten/Reshape"
+datatype = "FP32"
+shape = [10]
+EOF
 
 # Download the class labels for this model
 wget https://github.com/Xilinx/inference-server/raw/main/examples/resnet50/imagenet_classes.txt
diff --git a/src/amdinfer/clients/grpc_internal.cpp b/src/amdinfer/clients/grpc_internal.cpp
index caf5ed76a..a17e19bcf 100644
--- a/src/amdinfer/clients/grpc_internal.cpp
+++ b/src/amdinfer/clients/grpc_internal.cpp
@@ -182,7 +182,7 @@ struct SetOutputData {
     data.resize(bytes_to_copy);
     const auto* contents = getTensorContents<T>(tensor);
     if constexpr (std::is_same_v<T, char>) {
-      std::memcpy(data.data(), contents, size * sizeof(std::byte));
+      std::memcpy(data.data(), (*contents)->data(), size * sizeof(T));
       output->setData(std::move(data));
     } else {
       if constexpr (util::is_any_v<T, int8_t, uint8_t, int16_t, uint16_t,
diff --git a/src/amdinfer/core/CMakeLists.txt b/src/amdinfer/core/CMakeLists.txt
index 196c6e0c7..96f317306 100644
--- a/src/amdinfer/core/CMakeLists.txt
+++ b/src/amdinfer/core/CMakeLists.txt
@@ -19,6 +19,7 @@ set(base_targets
     inference_request
     inference_response
     inference_tensor
+    model_config
     tensor
     model_metadata
     endpoints
@@ -39,9 +40,18 @@ amdinfer_add_protobuf_target(lib_config model_config.proto OFF)
 target_include_directories(
   model_repository PRIVATE $<TARGET_PROPERTY:lib_config,INCLUDE_DIRECTORIES>
 )
-target_link_libraries(model_repository INTERFACE efsw lib_config)
+target_link_libraries(
+  model_repository PRIVATE tomlplusplus::tomlplusplus INTERFACE efsw lib_config
+)
 add_dependencies(model_repository lib_config)
 
+target_include_directories(
+  model_config PRIVATE $<TARGET_PROPERTY:lib_config,INCLUDE_DIRECTORIES>
+)
+target_link_libraries(
+  model_config PRIVATE tomlplusplus::tomlplusplus INTERFACE lib_config
+)
+
 target_link_libraries(inference_tensor INTERFACE $<TARGET_OBJECTS:tensor>)
 target_link_libraries(
   inference_request INTERFACE inference_tensor
diff --git a/src/amdinfer/core/model_config.cpp b/src/amdinfer/core/model_config.cpp
new file mode 100644
index 000000000..906f50a77
--- /dev/null
+++ b/src/amdinfer/core/model_config.cpp
@@ -0,0 +1,290 @@
+// Copyright 2023 Advanced Micro Devices, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/**
+ * @file
+ * @brief
+ */
+
+#include "amdinfer/core/model_config.hpp"
+
+#include <toml++/toml.h>
+
+#include <filesystem>
+
+#include "amdinfer/core/exceptions.hpp"
+#include "amdinfer/util/filesystem.hpp"
+#include "model_config.pb.h"  // for Config, InferP...
+
+namespace fs = std::filesystem;
+
+namespace amdinfer {
+
+std::string extractString(const toml::table& table, const std::string& key,
+                          bool is_required) {
+  if (!table.contains(key)) {
+    if (is_required) {
+      throw invalid_argument("The configuration must have a " + key);
+    } else {
+      return "";
+    }
+  }
+
+  const auto& node = table.at(key);
+  if (!node.is_string()) {
+    throw invalid_argument(key + " must be a string");
+  }
+
+  return node.value<std::string>().value();
+}
+
+ModelConfigTensor extractModelConfigTensor(const toml::table& table);
+ModelConfigData extractConfig(const toml::table& table, bool is_ensemble);
+
+// if we're not explicitly handling the types, use this to catch the failure at
+// compile time
+template <class...>
+constexpr std::false_type templated_false{};
+
+template <typename T>
+std::vector<T> extractArray(const toml::table& table, const std::string& key) {
+  if (!table.contains(key)) {
+    throw invalid_argument("The configuration must have a " + key);
+  }
+
+  const auto& node = table.at(key);
+  if (!node.is_array()) {
+    throw invalid_argument(key + " must be an array");
+  }
+  if (!node.is_homogeneous()) {
+    throw invalid_argument(key + " must be a homogenous array");
+  }
+  const auto& array = *node.as_array();
+
+  std::vector<T> vector;
+  for (auto&& element : array) {
+    if constexpr (std::is_integral_v<T>) {
+      const auto optional_value = element.value<T>();
+      if (optional_value) {
+        vector.push_back(optional_value.value());
+      } else {
+        throw invalid_argument(
+          "One or more values could not be converted to integer in " + key);
+      }
+    } else if constexpr (std::is_same_v<T, ModelConfigTensor>) {
+      if (!element.is_table()) {
+        throw invalid_argument(
+          "One or more values could not be converted to a table");
+      }
+      const auto& tbl = *element.as_table();
+      vector.push_back(extractModelConfigTensor(tbl));
+    } else if constexpr (std::is_same_v<T, ModelConfigData>) {
+      if (!element.is_table()) {
+        throw invalid_argument(
+          "One or more values could not be converted to a table");
+      }
+      const auto& tbl = *element.as_table();
+      vector.push_back(extractConfig(tbl, true));
+    } else {
+      static_assert(templated_false<T>);
+    }
+  }
+
+  return vector;
+}
+
+ModelConfigTensor extractModelConfigTensor(const toml::table& table) {
+  auto name = extractString(table, "name", true);
+  auto datatype_str = extractString(table, "datatype", true);
+  auto shape = extractArray<uint64_t>(table, "shape");
+  auto id = extractString(table, "id", false);
+
+  DataType datatype{datatype_str.c_str()};
+
+  return {name, shape, datatype, id};
+}
+
+ModelConfigData extractConfig(const toml::table& table, bool is_ensemble) {
+  auto name = extractString(table, "name", true);
+  auto platform = extractString(table, "platform", true);
+  auto id = extractString(table, "id", is_ensemble);
+
+  auto inputs = extractArray<ModelConfigTensor>(table, "inputs");
+  auto outputs = extractArray<ModelConfigTensor>(table, "outputs");
+
+  return {name, platform, id, inputs, outputs};
+}
+
+ModelConfigTensor::ModelConfigTensor(std::string name,
+                                     std::vector<uint64_t> shape,
+                                     DataType data_type, std::string id)
+  : Tensor(std::move(name), std::move(shape), data_type), id_(std::move(id)) {}
+
+const std::string& ModelConfigTensor::id() const& { return id_; }
+
+std::string ModelConfigTensor::id() && { return std::move(id_); }
+
+void ModelConfig::createModels() {
+  for (const auto& config : configs_) {
+    models_.emplace_back(config.name, ParameterMap{});
+    auto& parameters = std::get<1>(models_.back());
+    std::string extension;
+    if (config.platform == "tensorflow_graphdef") {
+      const auto& inputs = config.inputs;
+      if (inputs.size() != 1) {
+        throw invalid_argument("Currently, there must be one input tensor");
+      }
+      const auto& input = inputs.at(0);
+      parameters.put("input_node", input.getName());
+      const auto& input_shape = input.getShape();
+      // ZenDNN assumes square image in HWC format
+      parameters.put("input_size", static_cast<int>(input_shape.at(0)));
+      parameters.put("image_channels",
+                     static_cast<int>(input_shape.at(input_shape.size() - 1)));
+
+      const auto& outputs = config.outputs;
+      if (outputs.size() != 1) {
+        throw invalid_argument("Currently, there must be one output tensor");
+      }
+      const auto& output = outputs.at(0);
+      parameters.put("output_node", output.getName());
+      const auto& output_shape = output.getShape();
+      // ZenDNN assumes [X] classes as output
+      parameters.put("output_classes", static_cast<int>(output_shape.at(0)));
+
+      parameters.put("worker", "tfzendnn");
+    } else if (config.platform == "pytorch_torchscript") {
+      parameters.put("worker", "ptzendnn");
+    } else if (config.platform == "onnx_onnxv1") {
+      parameters.put("worker", "migraphx");
+    } else if (config.platform == "migraphx_mxr") {
+      parameters.put("worker", "migraphx");
+    } else if (config.platform == "vitis_xmodel") {
+      parameters.put("worker", "xmodel");
+    } else if (config.platform == "amdinfer_cpp") {
+      parameters.put("worker", "cplusplus");
+    } else {
+      throw invalid_argument("Unknown platform: " + config.platform);
+    }
+  }
+
+  // assuming a static chain for now so define it in reverse order
+  for (auto i = models_.size() - 1; i-- > 0;) {
+    const auto& model = std::get<0>(models_.at(i + 1));
+    auto& parameters = std::get<1>(models_.at(i));
+    parameters.put("next", model);
+  }
+}
+
+ModelConfig::ModelConfig(const toml::table& toml) {
+  if (toml.empty()) {
+    throw invalid_argument("The configuration cannot be empty");
+  }
+
+  if (toml.contains("models")) {
+    configs_ = extractArray<ModelConfigData>(toml, "models");
+  } else {
+    configs_.emplace_back(extractConfig(toml, false));
+  }
+
+  this->createModels();
+}
+
+ModelConfig::ModelConfig(const inference::Config& config) {
+  // proto does not support ensembles so id is fixed to empty
+  const std::string id;
+
+  const auto& model_name = config.name();
+  const auto& platform = config.platform();
+
+  const auto& proto_inputs = config.inputs();
+  std::vector<ModelConfigTensor> inputs;
+  for (const auto& input : proto_inputs) {
+    const auto& name = input.name();
+    const DataType datatype{input.datatype().c_str()};
+    const auto& proto_shape = input.shape();
+    std::vector<uint64_t> shape;
+    shape.reserve(proto_shape.size());
+    for (const auto& index : proto_shape) {
+      shape.push_back(index);
+    }
+    inputs.emplace_back(name, shape, datatype, id);
+  }
+
+  const auto& proto_outputs = config.outputs();
+  std::vector<ModelConfigTensor> outputs;
+  for (const auto& output : proto_outputs) {
+    const auto& name = output.name();
+    const DataType datatype{output.datatype().c_str()};
+    const auto& proto_shape = output.shape();
+    std::vector<uint64_t> shape;
+    shape.reserve(proto_shape.size());
+    for (const auto& index : proto_shape) {
+      shape.push_back(index);
+    }
+    outputs.emplace_back(name, shape, datatype, id);
+  }
+
+  configs_.emplace_back(model_name, platform, id, inputs, outputs);
+
+  this->createModels();
+}
+
+void ModelConfig::setModelFiles(const std::filesystem::path& base_path) {
+  auto i = 0;
+  for (auto& [_, parameters] : models_) {
+    const auto& config = configs_.at(i);
+    if (config.id.empty()) {
+      auto model = util::findFile(base_path, "");
+      parameters.put("model", model.string());
+    } else {
+      parameters.put("model", (base_path / config.id).string());
+    }
+    i++;
+  }
+}
+
+size_t ModelConfig::size() const { return configs_.size(); }
+
+std::pair<std::string, ParameterMap> ModelConfig::get(size_t index) {
+  return models_.at(index);
+}
+
+ModelConfig::Iterator ModelConfig::begin() { return models_.begin(); }
+ModelConfig::ConstIterator ModelConfig::begin() const {
+  return models_.cbegin();
+}
+ModelConfig::ConstIterator ModelConfig::cbegin() const {
+  return models_.cbegin();
+}
+ModelConfig::ReverseIterator ModelConfig::rbegin() { return models_.rbegin(); }
+ModelConfig::ConstReverseIterator ModelConfig::rbegin() const {
+  return models_.crbegin();
+}
+ModelConfig::ConstReverseIterator ModelConfig::crbegin() const {
+  return models_.crbegin();
+}
+
+ModelConfig::Iterator ModelConfig::end() { return models_.end(); }
+ModelConfig::ConstIterator ModelConfig::end() const { return models_.cend(); }
+ModelConfig::ConstIterator ModelConfig::cend() const { return models_.cend(); }
+ModelConfig::ReverseIterator ModelConfig::rend() { return models_.rend(); }
+ModelConfig::ConstReverseIterator ModelConfig::rend() const {
+  return models_.crend();
+}
+ModelConfig::ConstReverseIterator ModelConfig::crend() const {
+  return models_.crend();
+}
+
+}  // namespace amdinfer
diff --git a/src/amdinfer/core/model_config.hpp b/src/amdinfer/core/model_config.hpp
new file mode 100644
index 000000000..bf9924ab7
--- /dev/null
+++ b/src/amdinfer/core/model_config.hpp
@@ -0,0 +1,123 @@
+// Copyright 2023 Advanced Micro Devices, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+/**
+ * @file
+ * @brief
+ */
+
+#ifndef GUARD_AMDINFER_CORE_MODEL_CONFIG
+#define GUARD_AMDINFER_CORE_MODEL_CONFIG
+
+#include <filesystem>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include "amdinfer/core/parameters.hpp"
+#include "amdinfer/core/tensor.hpp"
+
+namespace toml {
+inline namespace v3 {
+class table;
+}  // namespace v3
+}  // namespace toml
+
+namespace inference {
+class Config;
+}  // namespace inference
+
+namespace amdinfer {
+
+class ModelConfigTensor : public Tensor {
+ public:
+  ModelConfigTensor(std::string name, std::vector<uint64_t> shape,
+                    DataType data_type, std::string id);
+
+  const std::string& id() const&;
+  std::string id() &&;
+
+ private:
+  std::string id_;
+};
+
+struct ModelConfigData {
+  ModelConfigData(std::string name, std::string platform, std::string id,
+                  std::vector<ModelConfigTensor> inputs,
+                  std::vector<ModelConfigTensor> outputs)
+    : name(std::move(name)),
+      platform(std::move(platform)),
+      id(std::move(id)),
+      inputs(std::move(inputs)),
+      outputs(std::move(outputs)) {}
+
+  std::string name;
+  std::string platform;
+  std::string id;
+  std::vector<ModelConfigTensor> inputs;
+  std::vector<ModelConfigTensor> outputs;
+};
+
+class ModelConfig {
+  using Container = std::vector<std::pair<std::string, ParameterMap>>;
+  using Iterator = Container::iterator;
+  using ConstIterator = Container::const_iterator;
+  using ReverseIterator = Container::reverse_iterator;
+  using ConstReverseIterator = Container::const_reverse_iterator;
+
+ public:
+  explicit ModelConfig(const toml::v3::table& config);
+  explicit ModelConfig(const inference::Config& config);
+
+  std::pair<std::string, ParameterMap> get(size_t index);
+  void setModelFiles(const std::filesystem::path& base_path);
+
+  size_t size() const;
+
+  /// Returns a read/write iterator to the first parameter in the object
+  Iterator begin();
+  /// Returns a read iterator to the first parameter in the object
+  [[nodiscard]] ConstIterator begin() const;
+  /// Returns a read iterator to the first parameter in the object
+  [[nodiscard]] ConstIterator cbegin() const;
+  /// Returns a read/write iterator to the first parameter in the object
+  ReverseIterator rbegin();
+  /// Returns a read iterator to the first parameter in the object
+  [[nodiscard]] ConstReverseIterator rbegin() const;
+  /// Returns a read iterator to the first parameter in the object
+  [[nodiscard]] ConstReverseIterator crbegin() const;
+
+  /// Returns a read/write iterator to one past the last parameter in the object
+  Iterator end();
+  /// Returns a read iterator to one past the last parameter in the object
+  [[nodiscard]] ConstIterator end() const;
+  /// Returns a read iterator to one past the last parameter in the object
+  [[nodiscard]] ConstIterator cend() const;
+  /// Returns a read/write iterator to the first parameter in the object
+  ReverseIterator rend();
+  /// Returns a read iterator to the first parameter in the object
+  [[nodiscard]] ConstReverseIterator rend() const;
+  /// Returns a read iterator to the first parameter in the object
+  [[nodiscard]] ConstReverseIterator crend() const;
+
+ private:
+  std::vector<ModelConfigData> configs_;
+  Container models_;
+
+  void createModels();
+};
+
+}  // namespace amdinfer
+
+#endif  // GUARD_AMDINFER_CORE_MODEL_CONFIG
diff --git a/src/amdinfer/core/model_repository.cpp b/src/amdinfer/core/model_repository.cpp
index 10e13282b..c6431097d 100644
--- a/src/amdinfer/core/model_repository.cpp
+++ b/src/amdinfer/core/model_repository.cpp
@@ -19,6 +19,7 @@
 #include <google/protobuf/io/zero_copy_stream_impl.h>  // for FileInputStream
 #include <google/protobuf/repeated_ptr_field.h>        // for RepeatedPtrField
 #include <google/protobuf/text_format.h>               // for TextFormat
+#include <toml++/toml.h>
 
 #include <chrono>      // for milliseconds
 #include <filesystem>  // for path, operator/
@@ -26,8 +27,12 @@
 
 #include "amdinfer/core/endpoints.hpp"       // for Endpoints
 #include "amdinfer/core/exceptions.hpp"      // for runtime_error
+#include "amdinfer/core/model_config.hpp"    // for ModelConfig
 #include "amdinfer/core/parameters.hpp"      // for ParameterMap
 #include "amdinfer/observation/logging.hpp"  // for AMDINFER_LOG_D...
+#include "amdinfer/util/filesystem.hpp"      // for findFile
+#include "amdinfer/util/string.hpp"          // for endsWith
+#include "model_config.hpp"                  // for ModelConfig
 #include "model_config.pb.h"                 // for Config, InferP...
 
 namespace fs = std::filesystem;
@@ -67,82 +72,69 @@ void mapProtoToParameters2(
   }
 }
 
-void parseModel(const fs::path& repository, const std::string& model,
-                ParameterMap* parameters) {
-  const fs::path config_file = "config.pbtxt";
+fs::path findConfigFile(const fs::path& model_path, const std::string& model) {
+  std::array extensions{".toml", ".pbtxt"};
+  for (const auto* extension : extensions) {
+    auto config_path = util::findFile(model_path, extension);
+    if (!config_path.empty()) {
+      return config_path;
+    }
 
-  auto model_path = repository / model;
-  auto config_path = model_path / config_file;
-
-  // KServe can sometimes create directories like model/model/config_file
-  // so if model/config_file doesn't exist, try searching a directory lower too
-  if (!fs::exists(config_path) &&
-      fs::exists(model_path / model / config_file)) {
-    model_path /= model;
-    config_path = model_path / config_file;
+    // KServe can sometimes create directories like model/model/config_file
+    // so if model/config_file doesn't exist, try searching a directory lower
+    // too
+    config_path = util::findFile(model_path / model, extension);
+    if (!config_path.empty()) {
+      return config_path;
+    }
   }
 
-  // TODO(varunsh): support other versions than 1/
-  const std::string model_base = model_path / "1/saved_model";
-
-  inference::Config config;
+  throw file_not_found_error("No configuration file could be loaded in " +
+                             model_path.string());
+}
 
-  // NOLINTNEXTLINE(cppcoreguidelines-pro-type-vararg, hicpp-vararg)
-  int file_descriptor = open(config_path.c_str(), O_RDONLY | O_CLOEXEC);
-  AMDINFER_IF_LOGGING(Logger logger{Loggers::Server};)
-  if (file_descriptor < 0) {
-    throw file_not_found_error("Config file " + config_path.string() +
-                               " could not be opened");
-  }
+ModelConfig openConfigFile(const fs::path& config_path) {
+  if (config_path.extension() == ".pbtxt") {
+    inference::Config proto_config;
 
-  google::protobuf::io::FileInputStream file_input(file_descriptor);
-  file_input.SetCloseOnDelete(true);
+    // NOLINTNEXTLINE(cppcoreguidelines-pro-type-vararg, hicpp-vararg)
+    int file_descriptor = open(config_path.c_str(), O_RDONLY | O_CLOEXEC);
+    if (file_descriptor < 0) {
+      throw file_not_found_error("Config file " + config_path.string() +
+                                 " could not be opened");
+    }
 
-  if (!google::protobuf::TextFormat::Parse(&file_input, &config)) {
-    throw file_read_error("Config file " + config_path.string() +
-                          " could not be parsed");
-  }
+    google::protobuf::io::FileInputStream file_input(file_descriptor);
+    file_input.SetCloseOnDelete(true);
 
-  if (config.platform() == "tensorflow_graphdef") {
-    const auto& inputs = config.inputs();
-    // currently supporting one input tensor
-    for (const auto& input : inputs) {
-      parameters->put("input_node", input.name());
-      const auto& shape = input.shape();
-      // ZenDNN assumes square image in HWC format
-      parameters->put("input_size", static_cast<int>(shape.at(0)));
-      parameters->put("image_channels",
-                      static_cast<int>(shape.at(shape.size() - 1)));
+    if (!google::protobuf::TextFormat::Parse(&file_input, &proto_config)) {
+      throw file_read_error("Config file " + config_path.string() +
+                            " could not be parsed");
     }
 
-    const auto& outputs = config.outputs();
-    // currently supporting one output tensor
-    for (const auto& output : outputs) {
-      parameters->put("output_node", output.name());
-      const auto& shape = output.shape();
-      // ZenDNN assumes [X] classes as output
-      parameters->put("output_classes", static_cast<int>(shape.at(0)));
-    }
+    return ModelConfig{proto_config};
+  } else if (config_path.extension() == ".toml") {
+    auto toml = toml::parse_file(config_path.string());
 
-    parameters->put("worker", "tfzendnn");
-    parameters->put("model", model_base + ".pb");
-  } else if (config.platform() == "pytorch_torchscript") {
-    parameters->put("worker", "ptzendnn");
-    parameters->put("model", model_base + ".pt");
-  } else if (config.platform() == "onnx_onnxv1") {
-    parameters->put("worker", "migraphx");
-    parameters->put("model", model_base + ".onnx");
-  } else if (config.platform() == "migraphx_mxr") {
-    parameters->put("worker", "migraphx");
-    parameters->put("model", model_base + ".mxr");
-  } else if (config.platform() == "vitis_xmodel") {
-    parameters->put("worker", "xmodel");
-    parameters->put("model", model_base + ".xmodel");
-  } else {
-    throw invalid_argument("Unknown platform: " + config.platform());
+    return ModelConfig{toml};
   }
+  throw invalid_argument(
+    "Unknown config file extension passed to openConfigFile: " +
+    config_path.extension().string());
+}
+
+ModelConfig parseModel(const fs::path& repository, const std::string& model) {
+  AMDINFER_IF_LOGGING(Logger logger{Loggers::Server};)
+  auto model_path = repository / model;
 
-  mapProtoToParameters2(config.parameters(), parameters);
+  auto config_path = findConfigFile(model_path, model);
+  auto config = openConfigFile(config_path);
+
+  // TODO(varunsh): support other versions than 1/
+  const auto model_base = config_path.parent_path() / "1";
+
+  config.setModelFiles(model_base);
+  return config;
 }
 
 void ModelRepository::setRepository(const fs::path& repository_path,
@@ -185,16 +177,17 @@ void UpdateListener::handleFileAction(
   AMDINFER_IF_LOGGING(Logger logger{Loggers::Server};)
   // arbitrary delay to make sure filesystem has settled
   const std::chrono::milliseconds delay{100};
-  if (filename == "config.pbtxt") {
+  if (filename == "config.pbtxt" || util::endsWith(filename, "toml")) {
     if (action == efsw::Actions::Add) {
       std::this_thread::sleep_for(delay);
-      auto model = fs::path(dir).parent_path().filename();
-      ParameterMap params;
+      auto model_name = fs::path(dir).parent_path().filename();
       try {
-        parseModel(repository_, model, &params);
-        endpoints_->load(model, params);
+        auto config = parseModel(repository_, model_name);
+        for (const auto& [model, parameters] : config) {
+          endpoints_->load(model, parameters);
+        }
       } catch (const runtime_error&) {
-        AMDINFER_LOG_INFO(logger, "Error loading " + model.string());
+        AMDINFER_LOG_INFO(logger, "Error loading " + model_name.string());
       }
     } else if (action == efsw::Actions::Delete) {
       // arbitrary delay to make sure filesystem has settled
diff --git a/src/amdinfer/core/model_repository.hpp b/src/amdinfer/core/model_repository.hpp
index 6255b0207..22123215e 100644
--- a/src/amdinfer/core/model_repository.hpp
+++ b/src/amdinfer/core/model_repository.hpp
@@ -25,6 +25,7 @@ namespace amdinfer {
 
 class Endpoints;
 class ParameterMap;
+class ModelConfig;
 
 class UpdateListener : public efsw::FileWatchListener {
  public:
@@ -39,8 +40,8 @@ class UpdateListener : public efsw::FileWatchListener {
   Endpoints* endpoints_;
 };
 
-void parseModel(const std::filesystem::path& repository,
-                const std::string& model, ParameterMap* parameters);
+ModelConfig parseModel(const std::filesystem::path& repository,
+                       const std::string& model);
 
 class ModelRepository {
  public:
diff --git a/src/amdinfer/core/parameters.cpp b/src/amdinfer/core/parameters.cpp
index 2ae8ab74d..fd4a69bf7 100644
--- a/src/amdinfer/core/parameters.cpp
+++ b/src/amdinfer/core/parameters.cpp
@@ -44,11 +44,14 @@ ParameterMap::ParameterMap(const std::vector<std::string> &keys,
 }
 
 void ParameterMap::put(const std::string &key, Parameter value) {
-  this->parameters_.try_emplace(key, std::move(value));
+  auto [iter, emplaced] = this->parameters_.try_emplace(key, std::move(value));
+  if (!emplaced) {
+    iter->second = value;
+  }
 }
 
 void ParameterMap::put(const std::string &key, const char *value) {
-  this->parameters_.try_emplace(key, std::string{value});
+  put(key, std::string{value});
 }
 
 void ParameterMap::erase(const std::string &key) {
diff --git a/src/amdinfer/core/shared_state.cpp b/src/amdinfer/core/shared_state.cpp
index d0ed63f32..338310a34 100644
--- a/src/amdinfer/core/shared_state.cpp
+++ b/src/amdinfer/core/shared_state.cpp
@@ -33,6 +33,7 @@
 #include "amdinfer/build_options.hpp"          // for AMDINFER_ENABLE_VITIS
 #include "amdinfer/core/endpoints.hpp"         // for Endpoints
 #include "amdinfer/core/exceptions.hpp"        // for external_error, invalid...
+#include "amdinfer/core/model_config.hpp"      // for ModelConfig
 #include "amdinfer/core/model_repository.hpp"  // for ModelRepository
 #include "amdinfer/core/parameters.hpp"        // for ParameterMap
 #include "amdinfer/core/request_container.hpp"  // for ServerMetadata, ModelMe...
@@ -75,10 +76,17 @@ void SharedState::modelLoad(const std::string& model,
                             const ParameterMap& parameters) {
   assert(util::isLower(model));
 
-  ParameterMap updated_parameters = parameters;
-
-  parseModel(repository_.getRepository(), model, &updated_parameters);
-  endpoints_.load(model, updated_parameters);
+  auto model_config = parseModel(repository_.getRepository(), model);
+  const auto end = model_config.crend();
+  for (auto it = model_config.crbegin(); it != end; ++it) {
+    const auto& [model_name, model_parameters] = *it;
+    // merge the runtime provided parameters with those from the model file
+    ParameterMap updated_parameters = model_parameters;
+    for (const auto& [key, value] : parameters) {
+      updated_parameters.put(key, value);
+    }
+    endpoints_.load(model_name, updated_parameters);
+  }
 }
 
 void SharedState::modelUnload(const std::string& model) {
diff --git a/src/amdinfer/models/CMakeLists.txt b/src/amdinfer/models/CMakeLists.txt
index b38e6cfc4..46a49bb16 100644
--- a/src/amdinfer/models/CMakeLists.txt
+++ b/src/amdinfer/models/CMakeLists.txt
@@ -18,7 +18,7 @@ set(filenames base64_encode base64_decode echo echo_multi invert_image)
 set(targets "")
 
 function(amdinfer_get_target_name target filename)
-  set(${target} ${filename}_model PARENT_SCOPE)
+  set(${target} ${filename} PARENT_SCOPE)
 endfunction()
 
 foreach(filename ${filenames})
diff --git a/src/amdinfer/models/base64_decode.cpp b/src/amdinfer/models/base64_decode.cpp
index 1efc34a9a..0930e91d6 100644
--- a/src/amdinfer/models/base64_decode.cpp
+++ b/src/amdinfer/models/base64_decode.cpp
@@ -80,7 +80,7 @@ amdinfer::BatchPtr run(amdinfer::Batch* batch) {
 
     auto new_request = req->propagate();
 
-    auto* input_data = static_cast<char*>(input.getData());
+    const auto* input_data = static_cast<char*>(input.getData());
     auto decoded_str = amdinfer::util::base64Decode(input_data, input_size);
     std::vector<char> data(decoded_str.begin(), decoded_str.end());
     cv::Mat img;
diff --git a/src/amdinfer/servers/grpc_server.cpp b/src/amdinfer/servers/grpc_server.cpp
index 6ec7cffa8..467031dab 100644
--- a/src/amdinfer/servers/grpc_server.cpp
+++ b/src/amdinfer/servers/grpc_server.cpp
@@ -206,8 +206,13 @@ struct WriteData {
   void operator()(Buffer* buffer, Tensor* tensor, size_t offset, size_t size,
                   [[maybe_unused]] const Observer& observer) const {
     auto* contents = getTensorContents<T>(tensor);
-    if constexpr (util::is_any_v<T, bool, uint32_t, uint64_t, int32_t, int64_t,
-                                 float, double, char>) {
+    if constexpr (std::is_same_v<T, char>) {
+      // gRPC stores char as a string so contents is a string**. Copy over the
+      // actual char* data from the string
+      auto* dest = static_cast<std::byte*>(buffer->data(offset));
+      std::memcpy(dest, (*contents)->data(), size * sizeof(T));
+    } else if constexpr (util::is_any_v<T, bool, uint32_t, uint64_t, int32_t,
+                                        int64_t, float, double>) {
       auto* dest = static_cast<std::byte*>(buffer->data(offset));
       std::memcpy(dest, contents, size * sizeof(T));
     } else if constexpr (util::is_any_v<T, uint8_t, uint16_t, int8_t, int16_t,
diff --git a/src/amdinfer/servers/http_server.cpp b/src/amdinfer/servers/http_server.cpp
index dbb924e0c..bcb3e350a 100644
--- a/src/amdinfer/servers/http_server.cpp
+++ b/src/amdinfer/servers/http_server.cpp
@@ -505,10 +505,11 @@ void HttpServer::modelInfer(
   } catch (const invalid_argument &e) {
     AMDINFER_LOG_INFO(logger_, e.what());
     auto resp = errorHttpResponse(e.what(), HttpStatusCode::k400BadRequest);
-#ifdef AMDINFER_ENABLE_TRACING
-    auto context = trace->propagate();
-    propagate(resp.get(), context);
-#endif
+    // TODO(varunsh): could use after move here from the try block
+    // #ifdef AMDINFER_ENABLE_TRACING
+    //     auto context = trace->propagate();
+    //     propagate(resp.get(), context);
+    // #endif
     callback(resp);
   }
 }
diff --git a/src/amdinfer/util/CMakeLists.txt b/src/amdinfer/util/CMakeLists.txt
index 352341d80..68f9908b9 100644
--- a/src/amdinfer/util/CMakeLists.txt
+++ b/src/amdinfer/util/CMakeLists.txt
@@ -17,8 +17,9 @@ set(base_targets
     base64
     compression
     ctpl
-    parse_env
     exec
+    filesystem
+    parse_env
     read_nth_line
     timer
 )
diff --git a/src/amdinfer/util/filesystem.cpp b/src/amdinfer/util/filesystem.cpp
new file mode 100644
index 000000000..a0a6a89c3
--- /dev/null
+++ b/src/amdinfer/util/filesystem.cpp
@@ -0,0 +1,61 @@
+// Copyright 2023 Advanced Micro Devices, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "amdinfer/util/filesystem.hpp"
+
+#include <fstream>
+
+#include "amdinfer/core/exceptions.hpp"
+
+namespace fs = std::filesystem;
+
+namespace amdinfer::util {
+
+fs::path findFile(const fs::path& directory, const std::string& extension) {
+  if (!fs::exists(directory)) {
+    return "";
+  }
+
+  fs::path found_path;
+  for (const auto& entry : fs::directory_iterator(directory)) {
+    if (entry.is_directory()) {
+      continue;
+    }
+    const auto& path = entry.path();
+    if (!extension.empty() && path.extension().string() != extension) {
+      continue;
+    }
+
+    return path;
+  }
+
+  return "";
+}
+
+std::string readFile(const std::string& path) {
+  const size_t block_size = 4096;  // arbitrary value for block size
+  std::ifstream stream{path};
+  // throw exception in case irrecoverable read error
+  stream.exceptions(std::ios_base::badbit);
+
+  std::string out;
+  std::string buffer(block_size, '\0');
+  while (stream.read(buffer.data(), block_size)) {
+    out.append(buffer, 0, stream.gcount());
+  }
+  out.append(buffer, 0, stream.gcount());
+  return out;
+}
+
+}  // namespace amdinfer::util
diff --git a/src/amdinfer/util/filesystem.hpp b/src/amdinfer/util/filesystem.hpp
new file mode 100644
index 000000000..12262bb2d
--- /dev/null
+++ b/src/amdinfer/util/filesystem.hpp
@@ -0,0 +1,39 @@
+// Copyright 2023 Advanced Micro Devices, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef GUARD_AMDINFER_UTIL_FILESYSTEM
+#define GUARD_AMDINFER_UTIL_FILESYSTEM
+
+#include <filesystem>
+
+namespace amdinfer::util {
+
+/**
+ * @brief Get the path to the file in a directory with the given extension. If
+ * there is more than one file in the directory that matches, only the first one
+ * found is returned. If extension is empty, then the first found file is
+ * returned. Subdirectories are not searched.
+ *
+ * @param directory path to search in
+ * @param extension extension of the file to search for
+ * @return std::filesystem::path
+ */
+std::filesystem::path findFile(const std::filesystem::path& directory,
+                               const std::string& extension);
+
+std::string readFile(const std::string& path);
+
+}  // namespace amdinfer::util
+
+#endif  // GUARD_AMDINFER_UTIL_FILESYSTEM
diff --git a/src/amdinfer/workers/c_plus_plus.cpp b/src/amdinfer/workers/c_plus_plus.cpp
index b55b84cc7..85ce3df63 100644
--- a/src/amdinfer/workers/c_plus_plus.cpp
+++ b/src/amdinfer/workers/c_plus_plus.cpp
@@ -43,6 +43,7 @@
 #include "amdinfer/observation/tracing.hpp"  // for startFollowSpan, SpanPtr
 #include "amdinfer/util/containers.hpp"      // for containerSum
 #include "amdinfer/util/queue.hpp"           // for BufferPtrsQueue
+#include "amdinfer/util/string.hpp"          // for endsWith
 #include "amdinfer/util/thread.hpp"          // for setThreadName
 #include "amdinfer/util/timer.hpp"           // for Timer
 #include "amdinfer/workers/worker.hpp"       // for Worker
@@ -149,7 +150,11 @@ void CPlusPlus::doInit(ParameterMap* parameters) {
     throw invalid_argument("No model specified");
   }
 
-  handle_ = openModel("lib" + model + "_model.so");
+  if (!util::endsWith(model, ".so")) {
+    model = "lib" + model + ".so";
+  }
+
+  handle_ = openModel(model);
 }
 
 void CPlusPlus::doAcquire([[maybe_unused]] ParameterMap* parameters) {


Test patch:
diff --git a/tests/unit/core/CMakeLists.txt b/tests/unit/core/CMakeLists.txt
index 9cbf191cd..b33a53cd1 100644
--- a/tests/unit/core/CMakeLists.txt
+++ b/tests/unit/core/CMakeLists.txt
@@ -15,10 +15,18 @@
 
 add_subdirectory(memory_pool)
 
-list(APPEND tests inference_request_input parameter_map)
+list(
+  APPEND tests
+         inference_request_input
+         model_config
+         parameter_map
+)
 
 list(APPEND tests_libs "inference_request~parameters~inference_response"
-            "parameters"
+            "model_config~tensor~data_types~parameters~util" "parameters"
 )
 
 amdinfer_add_unit_tests("${tests}" "${tests_libs}")
+
+amdinfer_get_test_target(target model_config)
+target_link_libraries(${target} PRIVATE tomlplusplus::tomlplusplus)
diff --git a/tests/unit/core/test_model_config.cpp b/tests/unit/core/test_model_config.cpp
new file mode 100644
index 000000000..b937ad0ef
--- /dev/null
+++ b/tests/unit/core/test_model_config.cpp
@@ -0,0 +1,154 @@
+// Copyright 2023 Advanced Micro Devices, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include <toml++/toml.h>
+
+#include <iostream>
+
+#include "amdinfer/core/model_config.hpp"  // for ModelConfig
+#include "gtest/gtest.h"                   // for Message, TestPartResult, Test
+
+using namespace std::string_view_literals;
+
+namespace amdinfer {
+
+// NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
+TEST(UnitModelConfig, Basic) {
+  constexpr std::string_view toml_str = R"(
+    name = "mnist"
+    platform = "tensorflow_graphdef"
+
+    [[inputs]]
+    name = "images_in"
+    datatype = "FP32"
+    shape = [28, 28, 1]
+
+    [[outputs]]
+    name = "flatten/Reshape"
+    datatype = "FP32"
+    shape = [10]
+  )"sv;
+
+  const auto toml = toml::parse(toml_str);
+  ModelConfig config{toml};
+
+  ASSERT_EQ(config.size(), 1);
+  for (const auto& [model, parameters] : config) {
+    ASSERT_EQ(model, "mnist");
+    ASSERT_EQ(parameters.get<std::string>("worker"), "tfzendnn");
+  }
+}
+
+// NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
+TEST(UnitModelConfig, SingleEnsemble) {
+  constexpr std::string_view toml_str = R"(
+    [[models]]
+    name = "mnist"
+    platform = "tensorflow_graphdef"
+    id = "bar.pb"
+
+    [[models.inputs]]
+    name = "images_in"
+    datatype = "FP32"
+    shape = [28, 28, 1]
+    id = "image"
+
+    [[models.outputs]]
+    name = "flatten/Reshape"
+    datatype = "FP32"
+    shape = [10]
+    id = "classes"
+  )"sv;
+
+  const auto toml = toml::parse(toml_str);
+  ModelConfig config{toml};
+
+  ASSERT_EQ(config.size(), 1);
+  for (const auto& [model, parameters] : config) {
+    ASSERT_EQ(model, "mnist");
+    ASSERT_EQ(parameters.get<std::string>("worker"), "tfzendnn");
+  }
+}
+
+// NOLINTNEXTLINE(cert-err58-cpp, cppcoreguidelines-owning-memory)
+TEST(UnitModelConfig, Chain) {
+  constexpr std::string_view toml_str = R"(
+    [[models]]
+    name = "invert_image"
+    platform = "amdinfer_cpp"
+    id = "base64_decode.so"
+
+    [[models.inputs]]
+    name = "image_in"
+    datatype = "STRING"
+    shape = [1048576]
+    id = ""
+
+    [[models.outputs]]
+    name = "image_out"
+    datatype = "INT8"
+    shape = [1080, 1920, 3]
+    id = "preprocessed_image"
+
+    [[models]]
+    name = "execute"
+    platform = "amdinfer_cpp"
+    id = "invert_image.so"
+
+    [[models.inputs]]
+    name = "image_in"
+    datatype = "INT8"
+    shape = [1080, 1920, 3]
+    id = "preprocessed_image"
+
+    [[models.outputs]]
+    name = "image_out"
+    datatype = "INT8"
+    shape = [1080, 1920, 3]
+    id = "inverted_image"
+
+    [[models]]
+    name = "invert_image_postprocess"
+    platform = "amdinfer_cpp"
+    id = "base64_encode.so"
+
+    [[models.inputs]]
+    name = "image_in"
+    datatype = "INT8"
+    shape = [1080, 1920, 3]
+    id = "inverted_image"
+
+    [[models.outputs]]
+    name = "image_out"
+    datatype = "STRING"
+    shape = [1048576]
+    id = "postprocessed_image"
+  )"sv;
+
+  const auto toml = toml::parse(toml_str);
+  ModelConfig config{toml};
+
+  ASSERT_EQ(config.size(), 3);
+  std::array<std::string, 3> models{"invert_image", "execute",
+                                    "invert_image_postprocess"};
+
+  auto index = 0;
+  for (const auto& [model, parameters] : config) {
+    ASSERT_EQ(model, models.at(index));
+    ASSERT_EQ(parameters.get<std::string>("worker"), "cplusplus");
+    index++;
+  }
+}
+
+}  // namespace amdinfer