Repository: onnx/onnx-mlir
Pull request #3095: Added minimal support to do some timing of OM Runtime functionality
Author: login: AlexandreEichenberger
Created: 2025-03-13T21:27:46Z | Merged: 2025-03-19T14:31:09Z

Tried to determine where some of the time is spend when invoking the python runtime. Time printed in seconds. 

Timing information is read using indentation, more indented timers are within the following less indented timers.


Test program:
```mlir
module {
  func.func @main_graph(%arg0: tensor<1x384x30522xf32>, %arg1: tensor<1x384x30522xf32>) -> (tensor<1x384x30522xf32> {onnx.name = "y"}) {
    %0 = "onnx.Add"(%arg0, %arg1) : (tensor<1x384x30522xf32>, tensor<1x384x30522xf32>) -> tensor<1x384x30522xf32>
    return %0 : tensor<1x384x30522xf32>
  }
  "onnx.EntryPoint"() {func = @main_graph} : () -> ()
}
```

Results when compiling with `-O3` for Z using `RunONNXModel.py`

```
@OM_DRIVER, process_input, 0.000003
@OM_DRIVER,   tensor_list_create, 0.000000
@OM_DRIVER, tensor_list_create, 0.000000
@OM_DRIVER, inference, 0.003231
@OM_DRIVER,   process_output_types, 0.000002
@OM_DRIVER,   process_output_pyarray, 0.004537
@OM_DRIVER, process_output, 0.004542
@OM_DRIVER,     tensor_destroy_ptr, 0.000085
@OM_DRIVER,     tensor_destroy_struct, 0.000000
@OM_DRIVER,   tensor_list_destroy, 0.000087
@OM_DRIVER, delete_out_lists, 0.000092
@OM_DRIVER,     tensor_destroy_struct, 0.000000
@OM_DRIVER,     tensor_destroy_struct, 0.000000
@OM_DRIVER,   tensor_list_destroy, 0.000003
@OM_DRIVER, delete_in_lists, 0.000005
  2nd iteration, 0.008029167998756748, seconds
```

where notably a singnificant amount of time is spent in `process_output_pyarray`, which corresponds to this in `onnx-mlir/src/Runtime/python/PyExecutionSessionBase.cpp`

```
outputPyArrays.emplace_back(
        py::array(dtype, shape, omTensorGetDataPtr(omt)));
```




Diff:
diff --git a/include/onnx-mlir/Runtime/OMTensor.h b/include/onnx-mlir/Runtime/OMTensor.h
index f25e25c5230..2cf91b326ea 100644
--- a/include/onnx-mlir/Runtime/OMTensor.h
+++ b/include/onnx-mlir/Runtime/OMTensor.h
@@ -167,6 +167,21 @@ OM_EXTERNAL_VISIBILITY void omTensorDestroy(OMTensor *tensor);
  */
 OM_EXTERNAL_VISIBILITY void *omTensorGetDataPtr(const OMTensor *tensor);
 
+/**
+ * \brief OMTensor allocated data pointer getter.
+ *
+ * @param tensor pointer to the OMTensor
+ * @return pointer to the allocated memory buffer of the OMTensor,
+ *         This should only be used when needing to create cross-language
+ *         operation to free OMTensor buffers allocated by the model and freed
+ *         in a different language environment. NULL if the numerical data
+ *         buffer is not set. Note that depending on alignment constraints, the
+ *         allocated and the data pointers are not necessarily identical. Use
+ *         data pointer to read/write values, and allocated pointer to free the
+ *         memory.
+ */
+OM_EXTERNAL_VISIBILITY void *omTensorGetAllocatedPtr(const OMTensor *tensor);
+
 /**
  * \brief OMTensor data shape getter.
  *
diff --git a/src/Runtime/OMInstrument.c b/src/Runtime/OMInstrument.c
index bdf35895585..86ca397c25d 100644
--- a/src/Runtime/OMInstrument.c
+++ b/src/Runtime/OMInstrument.c
@@ -2,7 +2,7 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-//===-------------- OMTensor.cpp - OMTensor C Implementation --------------===//
+//===--- OMInstrument.inc - C/C++ Neutral Instrumentation Implementation---===//
 //
 // Copyright 2019-2020 The IBM Research Authors.
 //
diff --git a/src/Runtime/OMInstrument.cpp b/src/Runtime/OMInstrument.cpp
index 72dfd4abeb3..a4e006aaaa4 100644
--- a/src/Runtime/OMInstrument.cpp
+++ b/src/Runtime/OMInstrument.cpp
@@ -2,7 +2,7 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-//===------------- OMTensor.cpp - OMTensor C++ Implementation -------------===//
+//===--- OMInstrument.inc - C/C++ Neutral Instrumentation Implementation---===//
 //
 // Copyright 2019-2020 The IBM Research Authors.
 //
diff --git a/src/Runtime/OMInstrument.inc b/src/Runtime/OMInstrument.inc
index cb87bc6bc7e..d368ac8f8fa 100644
--- a/src/Runtime/OMInstrument.inc
+++ b/src/Runtime/OMInstrument.inc
@@ -2,14 +2,13 @@
  * SPDX-License-Identifier: Apache-2.0
  */
 
-//===--------- OMTensor.inc - C/C++ Neutral OMTensor Implementation--------===//
+//===--- OMInstrument.inc - C/C++ Neutral Instrumentation Implementation---===//
 //
-// Copyright 2019-2020 The IBM Research Authors.
+// Copyright 2019-2025 The IBM Research Authors.
 //
 // =============================================================================
 //
-// This file contains implementations of OMTensor data structures
-// and helper functions.
+// This file contains implementations of the OMInstrument calls.
 //
 //===----------------------------------------------------------------------===//
 
@@ -67,6 +66,10 @@ static char instrumentReportOpName[INSTRUMENT_OP_NAME_MASK + 1];
 static char instrumentReportNodeName[INSTRUMENT_NODE_NAME_MASK + 1];
 static FILE *fout = 0;
 
+// Global variable to help OMInstrumentHelper.h to keep track of nesting level
+// of timing operations.
+int timing_nest_level = 0;
+
 #ifdef __MVS__
 #define timersub(a, b, result)                                                 \
   do {                                                                         \
@@ -146,7 +149,8 @@ static void ReportMemory() {
   char memOutput[200];
   FILE *memPipe;
   mypid = getpid();
-  int num_chars_written = snprintf(memCommand, sizeof(memCommand), "ps -o vsz='' -p %d", mypid);
+  int num_chars_written =
+      snprintf(memCommand, sizeof(memCommand), "ps -o vsz='' -p %d", mypid);
   assert(num_chars_written >= 0 && "snprintf write error to memCommand");
   memPipe = popen(memCommand, "r");
   if (!memPipe) {
diff --git a/src/Runtime/OMInstrumentHelper.h b/src/Runtime/OMInstrumentHelper.h
new file mode 100644
index 00000000000..f4bf1e97516
--- /dev/null
+++ b/src/Runtime/OMInstrumentHelper.h
@@ -0,0 +1,107 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+//===--------- OMInstrumentHelper.h - Helper for Instrumentation ----------===//
+//
+// Copyright 2019-2025 The IBM Research Authors.
+//
+// =============================================================================
+//
+// This file contains helpers for gathering timing, enabled by setting:
+//
+// #define OM_DRIVER_TIMING 1
+// #include "src/Runtime/OMInstrumentHelper.h"
+//
+// When not defined, all macros are empty, aka generate no code, so no
+// overheads.
+//
+// In Linux: uses gettimeofday and timersub.
+//
+// TIMING_INIT(var) defines a timing var (context must hold for all timing
+//   operations).
+// TIMING_START(var) starts the timer named var. TIMING_STOP(var)
+//   adds to the timer named var the difference between now and the last start.
+// TIMING_PRINT(var) prints the cumulative time of timer named var.
+//
+// TIMING_INIT_START does both init and start.
+// TIMING_STOP_PRINT does both stop and print.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef OM_INSTRUMENT_HELPER_H
+#define OM_INSTRUMENT_HELPER_H 1
+
+// Set to 1 to disable all timing regardless of other flags.
+#define OM_DRIVER_TIMING_DISABLE_ALL 1 /* 1 (unless when debugging perf) */
+
+//===----------------------------------------------------------------------===//
+// Timing support for MVS
+
+#ifdef __MVS__
+#define timersub(a, b, result)                                                 \
+  do {                                                                         \
+    (result)->tv_sec = (a)->tv_sec - (b)->tv_sec;                              \
+    (result)->tv_usec = (a)->tv_usec - (b)->tv_usec;                           \
+    if ((result)->tv_usec < 0) {                                               \
+      --(result)->tv_sec;                                                      \
+      (result)->tv_usec += 1000000;                                            \
+    }                                                                          \
+  } while (0);
+#endif
+
+//===----------------------------------------------------------------------===//
+// Timing functions
+
+#if OM_DRIVER_TIMING && !OM_DRIVER_TIMING_DISABLE_ALL
+#include <stdio.h>
+#include <sys/time.h>
+
+// Global variable to help OMInstrumentHelper.h to keep track of nesting level
+// of timing operations.
+extern int timing_nest_level;
+
+#define TIMING_INIT(_var_name)                                                 \
+  /* Define variable in current scope. */                                      \
+  struct timeval _var_name, _var_name##_tmp;                                   \
+  int _var_name##_nest_level = -1;                                             \
+  _var_name.tv_sec = 0;                                                        \
+  _var_name.tv_usec = 0;
+
+#define TIMING_START(_var_name)                                                \
+  _var_name##_nest_level = timing_nest_level;                                  \
+  ++timing_nest_level;                                                         \
+  gettimeofday(&_var_name##_tmp, NULL);
+
+#define TIMING_STOP(_var_name)                                                 \
+  { /* Define variables in their own scope */                                  \
+    struct timeval start_time, stop_time, diff_time;                           \
+    start_time = _var_name##_tmp;                                              \
+    gettimeofday(&stop_time, NULL);                                            \
+    timersub(&stop_time, &start_time, &diff_time);                             \
+    _var_name.tv_sec += diff_time.tv_sec;                                      \
+    _var_name.tv_usec += diff_time.tv_usec;                                    \
+    --timing_nest_level;                                                       \
+  }
+
+#define TIMING_PRINT(_var_name)                                                \
+  if (_var_name##_nest_level >= 0) { /* was started at least once */           \
+    int l = _var_name##_nest_level;                                            \
+    fprintf(stderr, "@OM_DRIVER, %*s%s, %ld.%06ld\n", l, " ", #_var_name,      \
+        (long int)_var_name.tv_sec, (long int)_var_name.tv_usec);              \
+  }
+
+#else
+#define TIMING_INIT(_var_name)
+#define TIMING_START(_var_name)
+#define TIMING_STOP(_var_name)
+#define TIMING_PRINT(_var_name)
+#endif
+
+// Combined calls.
+#define TIMING_INIT_START(_var_name)                                           \
+  TIMING_INIT(_var_name) TIMING_START(_var_name)
+#define TIMING_STOP_PRINT(_var_name)                                           \
+  TIMING_STOP(_var_name) TIMING_PRINT(_var_name)
+
+#endif
diff --git a/src/Runtime/OMTensor.inc b/src/Runtime/OMTensor.inc
index 4be4f2af187..ca72f8aed82 100644
--- a/src/Runtime/OMTensor.inc
+++ b/src/Runtime/OMTensor.inc
@@ -57,16 +57,13 @@ int *__attribute__((__weak__)) __errno_location(void) { return &errno; }
 #ifdef ENABLE_PYRUNTIME_LIGHT
 // The implementation depends on src/Utility and llvm. Will be solved in
 // another PR. Here are just dummy definitions for compilation
-float om_f16_to_f32(uint16_t a) {
-  return (float) 0;
-}
+float om_f16_to_f32(uint16_t a) { return (float)0; }
 
-uint16_t  om_f32_to_f16(uint16_t a) {
-  return (uint16_t) 0;
-}
+uint16_t om_f32_to_f16(uint16_t a) { return (uint16_t)0; }
 #endif
 
-
+#define OM_DRIVER_TIMING 1
+#include "src/Runtime/OMInstrumentHelper.h"
 
 // On some platforms LLVM generates f16 conversion code that calls some
 // of these C runtime functions.
@@ -267,11 +264,15 @@ void omTensorDestroy(OMTensor *tensor) {
   if (!tensor)
     return;
   if (tensor->_owning) {
+    TIMING_INIT_START(tensor_destroy_ptr);
     free(tensor->_allocatedPtr);
+    TIMING_STOP_PRINT(tensor_destroy_ptr);
   }
+  TIMING_INIT_START(tensor_destroy_struct);
   free(tensor->_shape);
   free(tensor->_strides);
   free(tensor);
+  TIMING_STOP_PRINT(tensor_destroy_struct);
 }
 
 /* OMTensor data getter */
@@ -496,7 +497,7 @@ static void printData(FILE *fout, const OMTensor *tensor) {
 /* Helper macros to print data for 1-4D tensors  */
 #define LOOP_1(INDEX, IV, UB)                                                  \
   fprintf(fout, "[");                                                          \
-  for (int64_t IV = 0; (IV) < (UB); ++(IV)) {                                \
+  for (int64_t IV = 0; (IV) < (UB); ++(IV)) {                                  \
     if (IV)                                                                    \
       fprintf(fout, ", ");                                                     \
     indexes[(INDEX)] = (IV);                                                   \
@@ -507,7 +508,7 @@ static void printData(FILE *fout, const OMTensor *tensor) {
 
 #define LOOP_2(INDEX, IV, UB, ...)                                             \
   fprintf(fout, "[");                                                          \
-  for (int64_t IV = 0; (IV) < (UB); ++(IV)) {                                \
+  for (int64_t IV = 0; (IV) < (UB); ++(IV)) {                                  \
     if (IV)                                                                    \
       fprintf(fout, ", ");                                                     \
     indexes[(INDEX)] = (IV);                                                   \
@@ -517,7 +518,7 @@ static void printData(FILE *fout, const OMTensor *tensor) {
 
 #define LOOP_3(INDEX, IV, UB, ...)                                             \
   fprintf(fout, "[");                                                          \
-  for (int64_t IV = 0; (IV) < (UB); ++(IV)) {                                \
+  for (int64_t IV = 0; (IV) < (UB); ++(IV)) {                                  \
     if (IV)                                                                    \
       fprintf(fout, ", ");                                                     \
     indexes[(INDEX)] = (IV);                                                   \
@@ -527,7 +528,7 @@ static void printData(FILE *fout, const OMTensor *tensor) {
 
 #define LOOP_4(INDEX, IV, UB, ...)                                             \
   fprintf(fout, "[");                                                          \
-  for (int64_t IV = 0; (IV) < (UB); ++(IV)) {                                \
+  for (int64_t IV = 0; (IV) < (UB); ++(IV)) {                                  \
     if (IV)                                                                    \
       fprintf(fout, ", ");                                                     \
     indexes[(INDEX)] = (IV);                                                   \
@@ -537,7 +538,7 @@ static void printData(FILE *fout, const OMTensor *tensor) {
 
 #define LOOP_5(INDEX, IV, UB, ...)                                             \
   fprintf(fout, "[");                                                          \
-  for (int64_t IV = 0; (IV) < (UB); ++(IV)) {                                 \
+  for (int64_t IV = 0; (IV) < (UB); ++(IV)) {                                  \
     if (IV)                                                                    \
       fprintf(fout, ", ");                                                     \
     indexes[(INDEX)] = (IV);                                                   \
@@ -547,7 +548,7 @@ static void printData(FILE *fout, const OMTensor *tensor) {
 
 #define LOOP_6(INDEX, IV, UB, ...)                                             \
   fprintf(fout, "[");                                                          \
-  for (int64_t IV = 0; (IV) < (UB); ++(IV)) {                                 \
+  for (int64_t IV = 0; (IV) < (UB); ++(IV)) {                                  \
     if (IV)                                                                    \
       fprintf(fout, ", ");                                                     \
     indexes[(INDEX)] = (IV);                                                   \
@@ -587,7 +588,8 @@ static void printData(FILE *fout, const OMTensor *tensor) {
   } break;
   case 6: {
     int64_t indexes[6];
-    LOOP_6(0, i, shape[0], j, shape[1], k, shape[2], l, shape[3],  m, shape[4],  n, shape[5])
+    LOOP_6(0, i, shape[0], j, shape[1], k, shape[2], l, shape[3], m, shape[4],
+        n, shape[5])
   } break;
   default:
     assert(false && "not implemented");
diff --git a/src/Runtime/OMTensorList.inc b/src/Runtime/OMTensorList.inc
index 3c1789ba876..993ee003159 100644
--- a/src/Runtime/OMTensorList.inc
+++ b/src/Runtime/OMTensorList.inc
@@ -28,6 +28,8 @@
 #include "onnx-mlir/Runtime/OMTensorList.h"
 #include <string.h>
 
+#define OM_DRIVER_TIMING 1
+#include "src/Runtime/OMInstrumentHelper.h"
 
 struct OMTensorList {
 #ifdef __cplusplus
@@ -74,6 +76,7 @@ OMTensorList *omTensorListCreate(OMTensor **tensors, int64_t n) {
   if (!list)
     return NULL;
 
+  TIMING_INIT_START(tensor_list_create)
   size_t omts_bytes = sizeof(OMTensor *) * n;
   list->_size = n;
   list->_omts = (OMTensor **)malloc(omts_bytes);
@@ -84,7 +87,7 @@ OMTensorList *omTensorListCreate(OMTensor **tensors, int64_t n) {
   }
   // Copy the given OMTensors pointers to an array owned by OMTensorList
   memcpy(list->_omts, tensors, omts_bytes);
-
+  TIMING_STOP_PRINT(tensor_list_create);
   return list;
 }
 
@@ -92,10 +95,12 @@ OMTensorList *omTensorListCreate(OMTensor **tensors, int64_t n) {
 void omTensorListDestroy(OMTensorList *list) {
   if (!list)
     return;
+  TIMING_INIT_START(tensor_list_destroy)
   for (int64_t i = 0; i < list->_size; i++)
     omTensorDestroy(list->_omts[i]);
   // Free the list as well as the pointers to the OMTensor array
   omTensorListDestroyShallow(list);
+  TIMING_STOP_PRINT(tensor_list_destroy)
 }
 
 /* OMTensorList destroyer which does not destroy the tensors.
diff --git a/src/Runtime/python/PyCompileAndRuntime.py b/src/Runtime/python/PyCompileAndRuntime.py
index 4c2bece2898..4222ab55510 100644
--- a/src/Runtime/python/PyCompileAndRuntime.py
+++ b/src/Runtime/python/PyCompileAndRuntime.py
@@ -5,7 +5,7 @@
 # Copyright 2021-2024 The IBM Research Authors.
 #
 ################################################################################
-# commom class `PyOMRuntime` called by python scripts
+# Common class `PyOMRuntime` called by python scripts
 ################################################################################
 import numpy as np
 
diff --git a/src/Runtime/python/PyExecutionSessionBase.cpp b/src/Runtime/python/PyExecutionSessionBase.cpp
index eeb07955747..0f9e59f0413 100644
--- a/src/Runtime/python/PyExecutionSessionBase.cpp
+++ b/src/Runtime/python/PyExecutionSessionBase.cpp
@@ -28,6 +28,9 @@ SUPPRESS_WARNINGS_POP
 
 #include "PyExecutionSessionBase.hpp"
 
+#define OM_DRIVER_TIMING 1 /* 1 for timing, 0 for no timing/overheads */
+#include "src/Runtime/OMInstrumentHelper.h"
+
 namespace pybind11 {
 namespace detail {
 
@@ -71,7 +74,7 @@ namespace onnx_mlir {
 //
 // For numerical array, pybind11 can convert multi-dimensional array into
 // one-dimensional array without manual conversion, but pybind11 cannot
-// convert multi-dimentional string array into one-dimentional array
+// convert multi-dimensional string array into one-dimensional array
 // automatically.
 // This function will be rewritten when pybind fixes the issue, or
 // better way for fixing it is found.
@@ -118,6 +121,7 @@ std::vector<py::array> PyExecutionSessionBase::pyRun(
     throw std::runtime_error(reportUndefinedEntryPointIn("run"));
 
   // 1. Process inputs.
+  TIMING_INIT_START(process_input);
   std::vector<OMTensor *> omts;
   if (inputsPyArray.size() != shapesPyArray.size())
     throw std::runtime_error(
@@ -230,16 +234,21 @@ std::vector<py::array> PyExecutionSessionBase::pyRun(
     }
     omts.emplace_back(inputOMTensor);
   }
+  TIMING_STOP_PRINT(process_input);
 
   // 2. Call entry point.
+  TIMING_INIT_START(inference);
   auto *wrappedInput = omTensorListCreate(&omts[0], omts.size());
   auto *wrappedOutput = _entryPointFunc(wrappedInput);
   if (!wrappedOutput)
     throw std::runtime_error(reportErrnoError());
+  TIMING_STOP_PRINT(inference);
 
   // 3. Process outputs.
+  TIMING_INIT_START(process_output);
   std::vector<py::array> outputPyArrays;
   for (int64_t i = 0; i < omTensorListGetSize(wrappedOutput); i++) {
+    TIMING_INIT_START(process_output_types);
     auto *omt = omTensorListGetOmtByIndex(wrappedOutput, i);
     auto shape = std::vector<int64_t>(
         omTensorGetShape(omt), omTensorGetShape(omt) + omTensorGetRank(omt));
@@ -300,12 +309,45 @@ std::vector<py::array> PyExecutionSessionBase::pyRun(
       throw std::runtime_error(reportPythonError(errStr.str()));
     }
     }
-
-    outputPyArrays.emplace_back(
-        py::array(dtype, shape, omTensorGetDataPtr(omt)));
+    TIMING_STOP_PRINT(process_output_types);
+
+    TIMING_INIT_START(process_output_pyarray);
+    // Use data pointer to indicate to the numpy array where the data is, and
+    // use allocated pointer for the custom deallocator. These two pointers may
+    // be different when trying to allocate data that must be at specific
+    // boundaries. Data pointer will be at the custom boundary, but alloc will
+    // be whatever the malloc returned.
+    void *omtAllocPtr = omTensorGetAllocatedPtr(omt);
+    void *omtDataPtr = omTensorGetDataPtr(omt);
+    // Check if the return value is a static constant, which cannot be freed and
+    // thus would have been created with the "owning" flag being false.
+    if (omTensorGetOwning(omt)) {
+      // CWe have a regular tensor which we will need to free at the right time.
+      // Create the capsule that points to the data to be freed (allocated
+      // pointer).
+      py::capsule free_data_with_allocate_ptr(
+          omtAllocPtr, [](void *ptr) { free(ptr); });
+      // Set owning to false as we migrate the ownership to python
+      omTensorSetOwning(omt, false);
+      // Pass the py::capsule to the numpy array for proper bookkeeping.
+      outputPyArrays.emplace_back(
+          py::array(dtype, shape, omtDataPtr, free_data_with_allocate_ptr));
+    } else {
+      // We have a constant, its a very rare case, just do like in the past:
+      // copy.
+      outputPyArrays.emplace_back(py::array(dtype, shape, omtDataPtr));
+    }
+    TIMING_STOP_PRINT(process_output_pyarray);
   }
+  TIMING_STOP_PRINT(process_output);
+
+  TIMING_INIT_START(delete_out_lists);
   omTensorListDestroy(wrappedOutput);
+  TIMING_STOP_PRINT(delete_out_lists);
+
+  TIMING_INIT_START(delete_in_lists);
   omTensorListDestroy(wrappedInput);
+  TIMING_STOP_PRINT(delete_in_lists);
 
   return outputPyArrays;
 }


Reviews:
user: login: chentong319
body: LGTM!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-14T20:06:34Z
user: login: tungld
body: LGTM!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-18T05:17:23Z
user: login: AlexandreEichenberger
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-18T14:58:40Z
user: login: AlexandreEichenberger
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-18T14:59:53Z

Review comments:
diff_hunk: @@ -0,0 +1,108 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+//===--------- OMInstrumentHelper.h - Helper for Instrumentation ----------===//
+//
+// Copyright 2019-2025 The IBM Research Authors.
+//
+// =============================================================================
+//
+// This file contains helpers for gathering timing, enabled by setting:
+//
+// #define OM_DRIVER_TIMING 1
+// #include "src/Runtime/OMInstrumentHelper.h"
+//
+// When not defined, all macros are empty, aka generate no code, so no
+// overheads.
+//
+// In Linux: uses gettimeofday and timersub.
+//
+// TIMING_INIT(var) defines a timing var (context must hold for all timing
+//   operations).
+// TIMING_START(var) starts the timer named var. TIMING_STOP(var)
+//   adds to the timer named var the difference between now and the last start.
+// TIMING_PRINT(var) prints the cumulative time of timer named var.
+//
+// TIMING_INIT_START does both init and start.
+// TIMING_STOP_PRINT does both stop and print.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef OM_INSTRUMENT_HELPER_H
+#define OM_INSTRUMENT_HELPER_H 1
+
+// Set to 1 to disable all timing regardless of other flags.
+#define OM_DRIVER_TIMING_DISABLE_ALL 1 /* 1 (unless when debugging perf) */
+
+//===----------------------------------------------------------------------===//
+// Timing support for MVS
+
+#ifdef __MVS__
+#define timersub(a, b, result)                                                 \
+  do {                                                                         \
+    (result)->tv_sec = (a)->tv_sec - (b)->tv_sec;                              \
+    (result)->tv_usec = (a)->tv_usec - (b)->tv_usec;                           \
+    if ((result)->tv_usec < 0) {                                               \
+      --(result)->tv_sec;                                                      \
+      (result)->tv_usec += 1000000;                                            \
+    }                                                                          \
+  } while (0);
+#endif
+
+//===----------------------------------------------------------------------===//
+// Timing functions
+
+#if OM_DRIVER_TIMING && !OM_DRIVER_TIMING_DISABLE_ALL
+#include <stdio.h>
+#include <sys/time.h>
+
+// Global variable to help OMInstrumentHelper.h to keep track of nesting level
+// of timing operations.
+extern int timing_nest_level;
+extern char timing_nest_strings[6][20];
+
+#define TIMING_INIT(_var_name)                                                 \
+  /* Define variable in current scope. */                                      \
+  struct timeval _var_name, _var_name##_tmp;                                   \
+  int _var_name##_nest_level = -1;                                             \
+  _var_name.tv_sec = 0;                                                        \
+  _var_name.tv_usec = 0;
+
+#define TIMING_START(_var_name)                                                \
+  _var_name##_nest_level = timing_nest_level;                                  \
+  ++timing_nest_level;                                                         \
+  gettimeofday(&_var_name##_tmp, NULL);
+
+#define TIMING_STOP(_var_name)                                                 \
+  { /* Define variables in their own scope */                                  \
+    struct timeval start_time, stop_time, diff_time;                           \
+    start_time = _var_name##_tmp;                                              \
+    gettimeofday(&stop_time, NULL);                                            \
+    timersub(&stop_time, &start_time, &diff_time);                             \
+    _var_name.tv_sec += diff_time.tv_sec;                                      \
+    _var_name.tv_usec += diff_time.tv_usec;                                    \
+    --timing_nest_level;                                                       \
+  }
+
+#define TIMING_PRINT(_var_name)                                                \
+  if (_var_name##_nest_level >= 0) { /* was started at least once */           \
+    int l = _var_name##_nest_level <= 5 ? _var_name##_nest_level : 5;          \
+    fprintf(stderr, "@OM_DRIVER, %s%s, %ld.%06ld\n", timing_nest_strings[l],   \
path: src/Runtime/OMInstrumentHelper.h
user: login: tungld
body: We can avoid using `timing_nest_string` by using `%*` to print out "" `l` times, e.g.
```c
fprintf(stderr, "@OM_DRIVER, %*s%s, %ld.%06ld\n", l, "",
           #_var_name, (long int)_var_name.tv_sec, (long int)_var_name.tv_usec);
```
created_at: 2025-03-18T05:15:18Z
author_association: MEMBER
diff_hunk: @@ -167,6 +167,18 @@ OM_EXTERNAL_VISIBILITY void omTensorDestroy(OMTensor *tensor);
  */
 OM_EXTERNAL_VISIBILITY void *omTensorGetDataPtr(const OMTensor *tensor);
 
+/**
+ * \brief OMTensor allocated data pointer getter.
+ *
+ * @param tensor pointer to the OMTensor
+ * @return pointer to the allocated memory buffer of the OMTensor,
+ *         This should only be used when needing needing to create
path: include/onnx-mlir/Runtime/OMTensor.h
user: login: tungld
body: Two `needing`.
created_at: 2025-03-18T05:17:13Z
author_association: MEMBER
diff_hunk: @@ -167,6 +167,18 @@ OM_EXTERNAL_VISIBILITY void omTensorDestroy(OMTensor *tensor);
  */
 OM_EXTERNAL_VISIBILITY void *omTensorGetDataPtr(const OMTensor *tensor);
 
+/**
+ * \brief OMTensor allocated data pointer getter.
+ *
+ * @param tensor pointer to the OMTensor
+ * @return pointer to the allocated memory buffer of the OMTensor,
+ *         This should only be used when needing needing to create
path: include/onnx-mlir/Runtime/OMTensor.h
user: login: AlexandreEichenberger
body: tx, added some additional comments on the difference between data and allocated pointers.
created_at: 2025-03-18T14:58:40Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,108 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+//===--------- OMInstrumentHelper.h - Helper for Instrumentation ----------===//
+//
+// Copyright 2019-2025 The IBM Research Authors.
+//
+// =============================================================================
+//
+// This file contains helpers for gathering timing, enabled by setting:
+//
+// #define OM_DRIVER_TIMING 1
+// #include "src/Runtime/OMInstrumentHelper.h"
+//
+// When not defined, all macros are empty, aka generate no code, so no
+// overheads.
+//
+// In Linux: uses gettimeofday and timersub.
+//
+// TIMING_INIT(var) defines a timing var (context must hold for all timing
+//   operations).
+// TIMING_START(var) starts the timer named var. TIMING_STOP(var)
+//   adds to the timer named var the difference between now and the last start.
+// TIMING_PRINT(var) prints the cumulative time of timer named var.
+//
+// TIMING_INIT_START does both init and start.
+// TIMING_STOP_PRINT does both stop and print.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef OM_INSTRUMENT_HELPER_H
+#define OM_INSTRUMENT_HELPER_H 1
+
+// Set to 1 to disable all timing regardless of other flags.
+#define OM_DRIVER_TIMING_DISABLE_ALL 1 /* 1 (unless when debugging perf) */
+
+//===----------------------------------------------------------------------===//
+// Timing support for MVS
+
+#ifdef __MVS__
+#define timersub(a, b, result)                                                 \
+  do {                                                                         \
+    (result)->tv_sec = (a)->tv_sec - (b)->tv_sec;                              \
+    (result)->tv_usec = (a)->tv_usec - (b)->tv_usec;                           \
+    if ((result)->tv_usec < 0) {                                               \
+      --(result)->tv_sec;                                                      \
+      (result)->tv_usec += 1000000;                                            \
+    }                                                                          \
+  } while (0);
+#endif
+
+//===----------------------------------------------------------------------===//
+// Timing functions
+
+#if OM_DRIVER_TIMING && !OM_DRIVER_TIMING_DISABLE_ALL
+#include <stdio.h>
+#include <sys/time.h>
+
+// Global variable to help OMInstrumentHelper.h to keep track of nesting level
+// of timing operations.
+extern int timing_nest_level;
+extern char timing_nest_strings[6][20];
+
+#define TIMING_INIT(_var_name)                                                 \
+  /* Define variable in current scope. */                                      \
+  struct timeval _var_name, _var_name##_tmp;                                   \
+  int _var_name##_nest_level = -1;                                             \
+  _var_name.tv_sec = 0;                                                        \
+  _var_name.tv_usec = 0;
+
+#define TIMING_START(_var_name)                                                \
+  _var_name##_nest_level = timing_nest_level;                                  \
+  ++timing_nest_level;                                                         \
+  gettimeofday(&_var_name##_tmp, NULL);
+
+#define TIMING_STOP(_var_name)                                                 \
+  { /* Define variables in their own scope */                                  \
+    struct timeval start_time, stop_time, diff_time;                           \
+    start_time = _var_name##_tmp;                                              \
+    gettimeofday(&stop_time, NULL);                                            \
+    timersub(&stop_time, &start_time, &diff_time);                             \
+    _var_name.tv_sec += diff_time.tv_sec;                                      \
+    _var_name.tv_usec += diff_time.tv_usec;                                    \
+    --timing_nest_level;                                                       \
+  }
+
+#define TIMING_PRINT(_var_name)                                                \
+  if (_var_name##_nest_level >= 0) { /* was started at least once */           \
+    int l = _var_name##_nest_level <= 5 ? _var_name##_nest_level : 5;          \
+    fprintf(stderr, "@OM_DRIVER, %s%s, %ld.%06ld\n", timing_nest_strings[l],   \
path: src/Runtime/OMInstrumentHelper.h
user: login: AlexandreEichenberger
body: I will give this pattern a try.
created_at: 2025-03-18T14:59:52Z
author_association: COLLABORATOR

PR comments:
user: login: AlexandreEichenberger
created_at: 2025-03-14T14:36:15Z
body: Measured overheads of the `pyarray` creation for 3 different output size and it seems to be linear in the data size

<img width="418" alt="image" src="https://github.com/user-attachments/assets/2ae95395-3bf8-403c-86d2-a34016a47e29" />

author_association: COLLABORATOR
user: login: chentong319
created_at: 2025-03-14T17:18:42Z
body: Did you use the docker env for your measure? Not sure whether the problem is related to the version of python or pybind11, or the combination.
author_association: COLLABORATOR
user: login: AlexandreEichenberger
created_at: 2025-03-14T18:49:51Z
body: It turns out that there is a way when "copying" a C++ data structure to a numpy object that allows the creation of a capsule object that enables python to free objects not allocated by it.

We have to save the pointer of the "allocated" data for the free to works, as sometime the buffer is padded so that the actual data might be a the correct alignment.

The trace below shows that the allocated data is at one address, the data is at another, and when data is freed, it will be effectively the allocated buffer that is freed

```
Running inference ...
@OM_DRIVER, process_input, 0.004687
@OM_DRIVER,   tensor_list_create, 0.000000
@OM_DRIVER, tensor_list_create, 0.000001
@OM_DRIVER, inference, 0.009955
@OM_DRIVER,   process_output_types, 0.000007
hi alex, preparing free for omTensor with ptr value 0x3ff9397f010 and data ptr 0x3ff93980000
@OM_DRIVER,   process_output_pyarray, 0.000007
@OM_DRIVER, process_output, 0.000020
@OM_DRIVER,     tensor_destroy_struct, 0.000000
@OM_DRIVER,   tensor_list_destroy, 0.000003
@OM_DRIVER, delete_out_lists, 0.000004
@OM_DRIVER,     tensor_destroy_ptr, 0.000095
@OM_DRIVER,     tensor_destroy_struct, 0.000000
@OM_DRIVER,     tensor_destroy_ptr, 0.000139
@OM_DRIVER,     tensor_destroy_struct, 0.000001
@OM_DRIVER,   tensor_list_destroy, 0.000242
@OM_DRIVER, delete_in_lists, 0.000243
  1st iteration, 0.015086764004081488, seconds
Reading reference outputs from check-ref ...
  - 1st output: [1x384x30522xfloat32]
  done.

Verifying value of output_0:[1, 384, 30522] using atol=0.01, rtol=0.05 ...
  correct.

hi alex, freeing omTensor with ptr value 0x3ff9397f010
```

Above, we can see the 2 different addresses for alloc and data, and we can see that the data is freed after verification (as the python driver, here `CheckONNX.py` keep the numpy array live until after the verification is performed. 
author_association: COLLABORATOR
user: login: AlexandreEichenberger
created_at: 2025-03-14T19:08:45Z
body: Results of the timing:
```
@OM_DRIVER, process_input, 0.000008
@OM_DRIVER,   tensor_list_create, 0.000000
@OM_DRIVER, tensor_list_create, 0.000001
@OM_DRIVER, inference, 0.003783
@OM_DRIVER,   process_output_types, 0.000007
@OM_DRIVER,   process_output_pyarray, 0.000005
@OM_DRIVER, process_output, 0.000020
@OM_DRIVER,     tensor_destroy_struct, 0.000000
@OM_DRIVER,   tensor_list_destroy, 0.000003
@OM_DRIVER, delete_out_lists, 0.000005
@OM_DRIVER,     tensor_destroy_struct, 0.000000
@OM_DRIVER,     tensor_destroy_struct, 0.000000
@OM_DRIVER,   tensor_list_destroy, 0.000005
@OM_DRIVER, delete_in_lists, 0.000006
  1st iteration, 0.0039923149888636544, seconds
```

we see now that inference takes  0.003783 s, and model takes 0.0039923149888636544. So now 94% of the time is spent in the inference, when driving it from python. Before it was less than 50%... obviously skewed by the fact that this was a single node operation. Nevertheless, we can now see that the "copy" is removed as the overheads are so much lowered, and the log above shows that the freeing of the data was done in the proper location within the python program using the provided lambda function by the C++ runtime.
author_association: COLLABORATOR
user: login: AlexandreEichenberger
created_at: 2025-03-14T19:10:39Z
body: > Did you use the docker env for your measure? Not sure whether the problem is related to the version of python or pybind11, or the combination.

I think it was due to the copy operation within the `py::array` as moving to a scheme that does not require the copying removed the issue. Thanks for the good suggestion.
author_association: COLLABORATOR
user: login: AlexandreEichenberger
created_at: 2025-03-17T20:01:24Z
body: @Sunny-Anand let me know if you want to merge this now or in a bit.

It removes a copy for the outputs of a model when using in a python environment. 

My tests cover usages using the `RunONNXModel.py` scripts and the backend tests. I verified the results for one model on a Z arch using NNPA, as there are alignments "tricks" that we do for NNPA buffers that we don't otherwise do.

I let you the final go ahead/wait choice.
author_association: COLLABORATOR
user: login: Sunny-Anand
created_at: 2025-03-17T20:09:05Z
body: @AlexandreEichenberger, let's merge this improvement in the main. Thanks for improving it, it's essential for those performance benchmark measurements.

author_association: COLLABORATOR
user: login: jenkins-droid
created_at: 2025-03-19T14:32:19Z
body: **Jenkins Linux s390x** Build [#16394](https://www.onnxmlir.xyz/jenkins/job/ONNX-MLIR-Pipeline-Docker-Build/16394/) [push] Added minimal support to... started at 10:32
author_association: COLLABORATOR
user: login: jenkins-droid
created_at: 2025-03-19T14:32:19Z
body: **Jenkins Linux amd64** Build [#16392](https://www.onnxmlir.xyz/jenkinx/job/ONNX-MLIR-Pipeline-Docker-Build/16392/) [push] Added minimal support to... started at 09:32
author_association: COLLABORATOR
user: login: jenkins-droid
created_at: 2025-03-19T14:32:22Z
body: **Jenkins Linux ppc64le** Build [#15375](https://www.onnxmlir.xyz/jenkinp/job/ONNX-MLIR-Pipeline-Docker-Build/15375/) [push] Added minimal support to... started at 10:44
author_association: COLLABORATOR
user: login: jenkins-droid
created_at: 2025-03-19T15:53:10Z
body: **Jenkins Linux amd64** Build [#16392](https://www.onnxmlir.xyz/jenkinx/job/ONNX-MLIR-Pipeline-Docker-Build/16392/) [push] Added minimal support to... passed after 1 hr 20 min
author_association: COLLABORATOR
user: login: jenkins-droid
created_at: 2025-03-19T16:03:04Z
body: **Jenkins Linux s390x** Build [#16394](https://www.onnxmlir.xyz/jenkins/job/ONNX-MLIR-Pipeline-Docker-Build/16394/) [push] Added minimal support to... passed after 1 hr 30 min
author_association: COLLABORATOR
user: login: jenkins-droid
created_at: 2025-03-19T16:57:20Z
body: **Jenkins Linux ppc64le** Build [#15375](https://www.onnxmlir.xyz/jenkinp/job/ONNX-MLIR-Pipeline-Docker-Build/15375/) [push] Added minimal support to... passed after 2 hr 25 min
author_association: COLLABORATOR