Repository: NREL/OpenStudio (language: python)

Issue:
C++ CLI: Calling a non existing method in a measure ends up with a crash and stack trace
Issue overview
--------------



## Current Behavior

For example, if you put in a measure something like:

```
  def run(model, runner, user_arguments)
    super(model, runner, user_arguments)

    runner.nonExistingMethod()
```

You end up with a hard crash.

## Expected Behavior

It should gracefully fail and report the error.

## Steps to Reproduce

See Current behavior

## Possible Solution

The issue is that we do `ScriptObject->run(args)` whereas the ruby CLI was all under a `rb_protect` block.

We need to find a way to do things like RubyInterpreter::makeProtectedCall or RubyEval's `evalString` (the Ruby Engine has exec / eval that call evalString).

## Details

### Environment

Some additional details about your environment for this issue (if relevant):
 - Platform (Operating system, version): all
 - Version of OpenStudio (if using an intermediate build, include SHA): 3.7.0-rc2

### Context

- #5016
C++ CLI: Calling a non existing method in a measure ends up with a crash and stack trace
Issue overview
--------------



## Current Behavior

For example, if you put in a measure something like:

```
  def run(model, runner, user_arguments)
    super(model, runner, user_arguments)

    runner.nonExistingMethod()
```

You end up with a hard crash.

## Expected Behavior

It should gracefully fail and report the error.

## Steps to Reproduce

See Current behavior

## Possible Solution

The issue is that we do `ScriptObject->run(args)` whereas the ruby CLI was all under a `rb_protect` block.

We need to find a way to do things like RubyInterpreter::makeProtectedCall or RubyEval's `evalString` (the Ruby Engine has exec / eval that call evalString).

## Details

### Environment

Some additional details about your environment for this issue (if relevant):
 - Platform (Operating system, version): all
 - Version of OpenStudio (if using an intermediate build, include SHA): 3.7.0-rc2

### Context

- #5016
C++ CLI: Calling a non existing method in a measure ends up with a crash and stack trace
Issue overview
--------------



## Current Behavior

For example, if you put in a measure something like:

```
  def run(model, runner, user_arguments)
    super(model, runner, user_arguments)

    runner.nonExistingMethod()
```

You end up with a hard crash.

## Expected Behavior

It should gracefully fail and report the error.

## Steps to Reproduce

See Current behavior

## Possible Solution

The issue is that we do `ScriptObject->run(args)` whereas the ruby CLI was all under a `rb_protect` block.

We need to find a way to do things like RubyInterpreter::makeProtectedCall or RubyEval's `evalString` (the Ruby Engine has exec / eval that call evalString).

## Details

### Environment

Some additional details about your environment for this issue (if relevant):
 - Platform (Operating system, version): all
 - Version of OpenStudio (if using an intermediate build, include SHA): 3.7.0-rc2

### Context

- #5016


Discussion / hints:

Fix patch:
diff --git a/python/engine/CMakeLists.txt b/python/engine/CMakeLists.txt
index 27d5cd211d0..1343becdea3 100644
--- a/python/engine/CMakeLists.txt
+++ b/python/engine/CMakeLists.txt
@@ -44,6 +44,20 @@ target_compile_options(pythonengine PRIVATE
 
 target_compile_definitions(pythonengine PRIVATE openstudio_scriptengine_EXPORTS SHARED_OS_LIBS)
 
+if(BUILD_TESTING)
+  set(pythonengine_test_depends
+    openstudio_scriptengine
+    openstudiolib
+    CONAN_PKG::fmt
+  )
+
+  set(pythonengine_test_src
+    test/PythonEngine_GTest.cpp
+  )
+
+  CREATE_TEST_TARGETS(pythonengine "${pythonengine_test_src}" "${pythonengine_test_depends}")
+endif()
+
 install(TARGETS pythonengine EXPORT openstudio DESTINATION ${LIB_DESTINATION_DIR} COMPONENT "CLI")
 
 # it goes into lib/ and we want to find:
diff --git a/ruby/engine/CMakeLists.txt b/ruby/engine/CMakeLists.txt
index a5d3dc3e5f2..601be0496cd 100644
--- a/ruby/engine/CMakeLists.txt
+++ b/ruby/engine/CMakeLists.txt
@@ -287,4 +287,18 @@ elseif(UNIX)
   )
 endif()
 
+if(BUILD_TESTING)
+  set(rubyengine_test_depends
+    openstudio_scriptengine
+    openstudiolib
+    CONAN_PKG::fmt
+  )
+
+  set(rubyengine_test_src
+    test/RubyEngine_GTest.cpp
+  )
+
+  CREATE_TEST_TARGETS(rubyengine "${rubyengine_test_src}" "${rubyengine_test_depends}")
+endif()
+
 install(TARGETS rubyengine EXPORT openstudio DESTINATION ${LIB_DESTINATION_DIR} COMPONENT "CLI")
diff --git a/src/measure/Measure.i b/src/measure/Measure.i
index ea239940b8a..a3cedd922cd 100644
--- a/src/measure/Measure.i
+++ b/src/measure/Measure.i
@@ -96,6 +96,71 @@
 };
 
 #if defined SWIGRUBY
+  %feature("director:except") {
+    // Mimic openstudio::evalString for a possible approach to reporting more info or a callstack.
+    VALUE errinfo = rb_errinfo();
+    VALUE exception_class = rb_obj_class(errinfo);
+    VALUE classNameValue = rb_class_name(exception_class);
+    std::string className(StringValuePtr(classNameValue));
+
+    VALUE errstr = rb_obj_as_string(errinfo);
+    std::string errMessage(StringValuePtr(errstr));
+
+    std::string totalErr = className + ": " + errMessage;
+
+    // Generally speaking, the backtrace is there, but not for the case where it's a stack too deep error
+    std::string loc;
+    const ID ID_backtrace = rb_intern_const("backtrace");
+    if (exception_class != rb_eSysStackError && rb_respond_to(errinfo, ID_backtrace)) {
+      /*volatile*/ VALUE backtrace;
+      if (!NIL_P(backtrace = rb_funcall(errinfo, ID_backtrace, 0))) {
+        VALUE backtracejoin = rb_ary_join(backtrace, rb_str_new2("\n"));
+        const std::string btlines = StringValuePtr(backtracejoin);
+        if (!btlines.empty()) {
+          loc += "\nTraceback (most recent call last):\n" + btlines;
+        }
+      }
+    }
+    if (loc.empty()) {
+      // In case we couldn't produce the backtrace (SystemStackError: stack level too deep), fall back on this:
+      // I just **cannot** figure out a way to get the error location in C, without calling $@.to_s. Seems nothing is available here
+      // $@ is an array, seems to be already ordered from most recent to older
+      int locationError;
+      VALUE btArray = rb_eval_string_protect("$@", &locationError); // "$@.reverse"
+      if (locationError == 0) {
+        // Get the backing C array of the ruby array
+        VALUE* elements = RARRAY_PTR(btArray);
+        long arrayLen = RARRAY_LEN(btArray);
+        const long back_trace_limit = 20;
+        loc += "\nTraceback (most recent call last):";
+
+        if (arrayLen > back_trace_limit) {
+          long omit = arrayLen - back_trace_limit;
+
+          const long initial_back_trace_limit_when_too_long = 8;
+          for (long c = arrayLen - 1 ; c > arrayLen - initial_back_trace_limit_when_too_long; --c) {
+            VALUE entry = elements[c];
+            std::string backtrace_line = RSTRING_PTR(entry);
+            loc += "\n" + std::string(backtrace_line);
+          }
+          loc += "\n\t... " + std::to_string(omit) + " levels...";
+
+          arrayLen = back_trace_limit - initial_back_trace_limit_when_too_long;
+        }
+        for (long c = 0; c < arrayLen; ++c) {
+          VALUE entry = elements[c];
+          std::string backtrace_line = RSTRING_PTR(entry);
+          loc += "\n" + std::string(backtrace_line);
+        }
+
+      } else {
+        loc = "Failed to get VM location";
+      }
+    }
+
+    totalErr += "\n" + loc;
+    throw Swig::DirectorMethodException(totalErr.c_str());
+  }
 
   %ignore OSMeasureInfoGetter;
 
@@ -107,4 +172,76 @@
 
 #endif
 
+#if defined SWIGPYTHON
+  %feature("director:except") {
+
+    if ($error != nullptr) {
+
+      PyObject *exc_type = nullptr;
+      PyObject *exc_value = nullptr;
+      PyObject *exc_tb = nullptr;
+      PyErr_Fetch(&exc_type, &exc_value, &exc_tb);
+      PyErr_NormalizeException(&exc_type, &exc_value, &exc_tb);
+      PyObject *str_exc_value = PyObject_Repr(exc_value); // Now a unicode object
+      PyObject *pyStr2 = PyUnicode_AsEncodedString(str_exc_value, "utf-8", "Error ~");
+      Py_DECREF(str_exc_value);
+      char *strExcValue = PyBytes_AsString(pyStr2); // NOLINT(hicpp-signed-bitwise)
+      Py_DECREF(pyStr2);
+
+      std::string err_msg = "In method '$symname': `" + std::string{strExcValue} + "`";
+
+      // See if we can get a full traceback.
+      // Calls into python, and does the same as capturing the exception in `e`
+      // then `print(traceback.format_exception(e.type, e.value, e.tb))`
+      PyObject *pModuleName = PyUnicode_DecodeFSDefault("traceback");
+      PyObject *pyth_module = PyImport_Import(pModuleName);
+      Py_DECREF(pModuleName);
+
+      if (pyth_module == nullptr) {
+          err_msg += "\nCannot find 'traceback' module, this should not happen";
+          throw Swig::DirectorMethodException(err_msg.c_str());
+      }
+
+      PyObject *pyth_func = PyObject_GetAttrString(pyth_module, "format_exception");
+      Py_DECREF(pyth_module); // PyImport_Import returns a new reference, decrement it
+
+      if (pyth_func || PyCallable_Check(pyth_func)) {
+
+          PyObject *pyth_val = PyObject_CallFunction(pyth_func, "OOO", exc_type, exc_value, exc_tb);
+
+          // traceback.format_exception returns a list, so iterate on that
+          if (!pyth_val || !PyList_Check(pyth_val)) { // NOLINT(hicpp-signed-bitwise)
+              err_msg += "\ntraceback.format_exception did not return a list.";
+          } else {
+            Py_ssize_t numVals = PyList_Size(pyth_val);
+            if (numVals == 0) {
+                err_msg += "\nNo traceback available";
+            } else {
+              // err_msg += "\nPython traceback follows:\n```";
+              err_msg += '\n';
+              for (Py_ssize_t itemNum = 0; itemNum < numVals; itemNum++) {
+                  PyObject *item = PyList_GetItem(pyth_val, itemNum);
+                  if (PyUnicode_Check(item)) { // NOLINT(hicpp-signed-bitwise) -- something inside Python code causes warning
+                      std::string traceback_line = PyUnicode_AsUTF8(item);
+                      if (!traceback_line.empty() && traceback_line[traceback_line.length() - 1] == '\n') {
+                          traceback_line.erase(traceback_line.length() - 1);
+                      }
+                      err_msg += "\n" + traceback_line;
+                  }
+                  // PyList_GetItem returns a borrowed reference, do not decrement
+              }
+
+              // err_msg += "\n```";
+            }
+          }
+
+          // PyList_Size returns a borrowed reference, do not decrement
+          Py_DECREF(pyth_val); // PyObject_CallFunction returns new reference, decrement
+      }
+      Py_DECREF(pyth_func); // PyObject_GetAttrString returns a new reference, decrement it
+      throw Swig::DirectorMethodException(err_msg.c_str());
+    }
+  }
+#endif
+
 #endif // MEASURE_I


Test patch:
diff --git a/python/engine/test/BadMeasure/measure.py b/python/engine/test/BadMeasure/measure.py
new file mode 100644
index 00000000000..67471e8588f
--- /dev/null
+++ b/python/engine/test/BadMeasure/measure.py
@@ -0,0 +1,33 @@
+import typing
+
+import openstudio
+
+
+class BadMeasure(openstudio.measure.ModelMeasure):
+    def name(self):
+        return "Bad Measure"
+
+    def modeler_description(self):
+        return "The arguments method calls another_method which does a raise ValueError"
+
+    def another_method(self):
+        raise ValueError("oops")
+
+    def arguments(self, model: typing.Optional[openstudio.model.Model] = None):
+        self.another_method()
+        args = openstudio.measure.OSArgumentVector()
+
+        return args
+
+    def run(
+        self,
+        model: openstudio.model.Model,
+        runner: openstudio.measure.OSRunner,
+        user_arguments: openstudio.measure.OSArgumentMap,
+    ):
+        """
+        define what happens when the measure is run
+        """
+        super().run(model, runner, user_arguments)  # Do **NOT** remove this line
+
+BadMeasure().registerWithApplication()
diff --git a/python/engine/test/BadMeasure/measure.xml b/python/engine/test/BadMeasure/measure.xml
new file mode 100644
index 00000000000..3b0e0be4172
--- /dev/null
+++ b/python/engine/test/BadMeasure/measure.xml
@@ -0,0 +1,59 @@
+<?xml version="1.0"?>
+<measure>
+  <schema_version>3.1</schema_version>
+  <name>bad_measure</name>
+  <uid>812d3ebf-c89b-4b93-b400-110ca060b2bb</uid>
+  <version_id>25ad8ea8-b28b-4f45-93a6-76f056c28ca8</version_id>
+  <version_modified>2023-11-10T10:47:04Z</version_modified>
+  <xml_checksum>33A29C78</xml_checksum>
+  <class_name>BadMeasure</class_name>
+  <display_name>Bad Measure</display_name>
+  <description></description>
+  <modeler_description>The arguments method calls another_method which does a raise ValueError</modeler_description>
+  <arguments />
+  <outputs />
+  <provenances />
+  <tags>
+    <tag>Envelope.Opaque</tag>
+  </tags>
+  <attributes>
+    <attribute>
+      <name>Measure Function</name>
+      <value>Measure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Requires EnergyPlus Results</name>
+      <value>false</value>
+      <datatype>boolean</datatype>
+    </attribute>
+    <attribute>
+      <name>Measure Type</name>
+      <value>ModelMeasure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Measure Language</name>
+      <value>Python</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Uses SketchUp API</name>
+      <value>false</value>
+      <datatype>boolean</datatype>
+    </attribute>
+  </attributes>
+  <files>
+    <file>
+      <version>
+        <software_program>OpenStudio</software_program>
+        <identifier>3.4.1</identifier>
+        <min_compatible>3.4.1</min_compatible>
+      </version>
+      <filename>measure.py</filename>
+      <filetype>py</filetype>
+      <usage_type>script</usage_type>
+      <checksum>E787E0E0</checksum>
+    </file>
+  </files>
+</measure>
diff --git a/python/engine/test/PythonEngine_GTest.cpp b/python/engine/test/PythonEngine_GTest.cpp
new file mode 100644
index 00000000000..6c44bf739bb
--- /dev/null
+++ b/python/engine/test/PythonEngine_GTest.cpp
@@ -0,0 +1,136 @@
+/***********************************************************************************************************************
+*  OpenStudio(R), Copyright (c) Alliance for Sustainable Energy, LLC.
+*  See also https://openstudio.net/license
+***********************************************************************************************************************/
+
+#include <gtest/gtest.h>
+
+#include "measure/ModelMeasure.hpp"
+#include "measure/OSArgument.hpp"
+#include "measure/OSMeasure.hpp"
+#include "model/Model.hpp"
+#include "scriptengine/ScriptEngine.hpp"
+
+#include <fmt/format.h>
+
+class PythonEngineFixture : public testing::Test
+{
+ public:
+  static openstudio::path getScriptPath(const std::string& classAndDirName) {
+    openstudio::path scriptPath = openstudio::getApplicationSourceDirectory() / fmt::format("python/engine/test/{}/measure.py", classAndDirName);
+    OS_ASSERT(openstudio::filesystem::is_regular_file(scriptPath));
+    return scriptPath;
+  }
+
+ protected:
+  // initialize for each test
+  virtual void SetUp() override {
+    std::vector<std::string> args;
+    thisEngine = std::make_unique<openstudio::ScriptEngineInstance>("pythonengine", args);
+
+    (*thisEngine)->registerType<openstudio::measure::OSMeasure*>("openstudio::measure::OSMeasure *");
+    (*thisEngine)->registerType<openstudio::measure::ModelMeasure*>("openstudio::measure::ModelMeasure *");
+  }
+  // tear down after each test
+  virtual void TearDown() override {
+    // Want to ensure the engine is reset regardless of the test outcome, or it may throw a segfault
+    thisEngine->reset();
+  }
+
+  std::unique_ptr<openstudio::ScriptEngineInstance> thisEngine;
+};
+
+TEST_F(PythonEngineFixture, BadMeasure) {
+
+  const std::string classAndDirName = "BadMeasure";
+
+  const auto scriptPath = getScriptPath(classAndDirName);
+  auto measureScriptObject = (*thisEngine)->loadMeasure(scriptPath, classAndDirName);
+  auto* measurePtr = (*thisEngine)->getAs<openstudio::measure::ModelMeasure*>(measureScriptObject);
+
+  ASSERT_EQ(measurePtr->name(), "Bad Measure");
+
+  std::string expected_exception = fmt::format(R"(SWIG director method error. In method 'arguments': `ValueError('oops')`
+
+Traceback (most recent call last):
+  File "{0}", line 17, in arguments
+    self.another_method()
+  File "{0}", line 14, in another_method
+    raise ValueError("oops")
+ValueError: oops)",
+                                               scriptPath.generic_string());
+
+  openstudio::model::Model model;
+  try {
+    measurePtr->arguments(model);
+    ASSERT_FALSE(true) << "Expected measure arguments(model) to throw";
+  } catch (std::exception& e) {
+    std::string error = e.what();
+    EXPECT_EQ(expected_exception, error);
+  }
+}
+
+TEST_F(PythonEngineFixture, WrongMethodMeasure) {
+
+  const std::string classAndDirName = "WrongMethodMeasure";
+
+  const auto scriptPath = getScriptPath(classAndDirName);
+  auto measureScriptObject = (*thisEngine)->loadMeasure(scriptPath, classAndDirName);
+  auto* measurePtr = (*thisEngine)->getAs<openstudio::measure::ModelMeasure*>(measureScriptObject);
+
+  ASSERT_EQ(measurePtr->name(), "Wrong Method Measure");
+
+  std::string expected_exception =
+    fmt::format(R"(SWIG director method error. In method 'arguments': `AttributeError("'Model' object has no attribute 'nonExistingMethod'")`
+
+Traceback (most recent call last):
+  File "{}", line 14, in arguments
+    model.nonExistingMethod()
+AttributeError: 'Model' object has no attribute 'nonExistingMethod')",
+                scriptPath.generic_string());
+
+  openstudio::model::Model model;
+  try {
+    measurePtr->arguments(model);
+    ASSERT_FALSE(true) << "Expected measure arguments(model) to throw";
+  } catch (std::exception& e) {
+    std::string error = e.what();
+    EXPECT_EQ(expected_exception, error);
+  }
+}
+
+TEST_F(PythonEngineFixture, StackLevelTooDeepMeasure) {
+
+  const std::string classAndDirName = "StackLevelTooDeepMeasure";
+
+  const auto scriptPath = getScriptPath(classAndDirName);
+  auto measureScriptObject = (*thisEngine)->loadMeasure(scriptPath, classAndDirName);
+  auto* measurePtr = (*thisEngine)->getAs<openstudio::measure::ModelMeasure*>(measureScriptObject);
+
+  ASSERT_EQ(measurePtr->name(), "Stack Level Too Deep Measure");
+
+  std::string expected_exception =
+    fmt::format(R"(SWIG director method error. In method 'arguments': `RecursionError('maximum recursion depth exceeded')`
+
+Traceback (most recent call last):
+  File "{0}", line 16, in arguments
+    s(10)
+  File "{0}", line 6, in s
+    return s(x)
+  File "{0}", line 6, in s
+    return s(x)
+  File "{0}", line 6, in s
+    return s(x)
+  [Previous line repeated 996 more times]
+RecursionError: maximum recursion depth exceeded)",
+                scriptPath.generic_string());
+
+  openstudio::model::Model model;
+  try {
+    measurePtr->arguments(model);
+    ASSERT_FALSE(true) << "Expected measure arguments(model) to throw";
+  } catch (std::exception& e) {
+    std::string error = e.what();
+    EXPECT_EQ(expected_exception, error);
+  }
+}
diff --git a/python/engine/test/StackLevelTooDeepMeasure/measure.py b/python/engine/test/StackLevelTooDeepMeasure/measure.py
new file mode 100644
index 00000000000..164eedb84fb
--- /dev/null
+++ b/python/engine/test/StackLevelTooDeepMeasure/measure.py
@@ -0,0 +1,32 @@
+import typing
+
+import openstudio
+
+def s(x):
+    return s(x)
+
+class StackLevelTooDeepMeasure(openstudio.measure.ModelMeasure):
+    def name(self):
+        return "Stack Level Too Deep Measure"
+
+    def modeler_description(self):
+        return "The arguments method calls an infinitely recursive function"
+
+    def arguments(self, model: typing.Optional[openstudio.model.Model] = None):
+        s(10)
+        args = openstudio.measure.OSArgumentVector()
+
+        return args
+
+    def run(
+        self,
+        model: openstudio.model.Model,
+        runner: openstudio.measure.OSRunner,
+        user_arguments: openstudio.measure.OSArgumentMap,
+    ):
+        """
+        define what happens when the measure is run
+        """
+        super().run(model, runner, user_arguments)  # Do **NOT** remove this line
+
+StackLevelTooDeepMeasure().registerWithApplication()
diff --git a/python/engine/test/StackLevelTooDeepMeasure/measure.xml b/python/engine/test/StackLevelTooDeepMeasure/measure.xml
new file mode 100644
index 00000000000..27ce36c5bcc
--- /dev/null
+++ b/python/engine/test/StackLevelTooDeepMeasure/measure.xml
@@ -0,0 +1,59 @@
+<?xml version="1.0"?>
+<measure>
+  <schema_version>3.1</schema_version>
+  <name>stack_level_too_deep_measure</name>
+  <uid>22222222-c89b-4b93-b400-220ca060b2bb</uid>
+  <version_id>c2e1967b-0672-4e80-84fd-9204e701f10a</version_id>
+  <version_modified>2023-11-10T14:04:04Z</version_modified>
+  <xml_checksum>4EDA3A53</xml_checksum>
+  <class_name>StackLevelTooDeepMeasure</class_name>
+  <display_name>Stack Level Too Deep Measure</display_name>
+  <description></description>
+  <modeler_description>The arguments method calls an infinitely recursive function</modeler_description>
+  <arguments />
+  <outputs />
+  <provenances />
+  <tags>
+    <tag>Envelope.Opaque</tag>
+  </tags>
+  <attributes>
+    <attribute>
+      <name>Measure Function</name>
+      <value>Measure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Requires EnergyPlus Results</name>
+      <value>false</value>
+      <datatype>boolean</datatype>
+    </attribute>
+    <attribute>
+      <name>Measure Type</name>
+      <value>ModelMeasure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Uses SketchUp API</name>
+      <value>false</value>
+      <datatype>boolean</datatype>
+    </attribute>
+    <attribute>
+      <name>Measure Language</name>
+      <value>Python</value>
+      <datatype>string</datatype>
+    </attribute>
+  </attributes>
+  <files>
+    <file>
+      <version>
+        <software_program>OpenStudio</software_program>
+        <identifier>3.4.1</identifier>
+        <min_compatible>3.4.1</min_compatible>
+      </version>
+      <filename>measure.py</filename>
+      <filetype>py</filetype>
+      <usage_type>script</usage_type>
+      <checksum>C952AC5B</checksum>
+    </file>
+  </files>
+</measure>
diff --git a/python/engine/test/WrongMethodMeasure/measure.py b/python/engine/test/WrongMethodMeasure/measure.py
new file mode 100644
index 00000000000..a2f524afe3d
--- /dev/null
+++ b/python/engine/test/WrongMethodMeasure/measure.py
@@ -0,0 +1,30 @@
+import typing
+
+import openstudio
+
+
+class WrongMethodMeasure(openstudio.measure.ModelMeasure):
+    def name(self):
+        return "Wrong Method Measure"
+
+    def modeler_description(self):
+        return "The arguments method calls a non existing method on the model passed as argument"
+
+    def arguments(self, model: typing.Optional[openstudio.model.Model] = None):
+        model.nonExistingMethod()
+        args = openstudio.measure.OSArgumentVector()
+
+        return args
+
+    def run(
+        self,
+        model: openstudio.model.Model,
+        runner: openstudio.measure.OSRunner,
+        user_arguments: openstudio.measure.OSArgumentMap,
+    ):
+        """
+        define what happens when the measure is run
+        """
+        super().run(model, runner, user_arguments)  # Do **NOT** remove this line
+
+WrongMethodMeasure().registerWithApplication()
diff --git a/python/engine/test/WrongMethodMeasure/measure.xml b/python/engine/test/WrongMethodMeasure/measure.xml
new file mode 100644
index 00000000000..47d90331a24
--- /dev/null
+++ b/python/engine/test/WrongMethodMeasure/measure.xml
@@ -0,0 +1,59 @@
+<?xml version="1.0"?>
+<measure>
+  <schema_version>3.1</schema_version>
+  <name>wrong_method_measure</name>
+  <uid>11111111-c89b-4b93-b400-110ca060b2bb</uid>
+  <version_id>998cdb81-addf-4fae-809d-175c6475c592</version_id>
+  <version_modified>2023-11-10T10:37:53Z</version_modified>
+  <xml_checksum>33A29C78</xml_checksum>
+  <class_name>WrongMethodMeasure</class_name>
+  <display_name>Wrong Method Measure</display_name>
+  <description></description>
+  <modeler_description>The arguments method calls a non existing method on the model passed as argument</modeler_description>
+  <arguments />
+  <outputs />
+  <provenances />
+  <tags>
+    <tag>Envelope.Opaque</tag>
+  </tags>
+  <attributes>
+    <attribute>
+      <name>Measure Function</name>
+      <value>Measure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Requires EnergyPlus Results</name>
+      <value>false</value>
+      <datatype>boolean</datatype>
+    </attribute>
+    <attribute>
+      <name>Measure Type</name>
+      <value>ModelMeasure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Measure Language</name>
+      <value>Python</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Uses SketchUp API</name>
+      <value>false</value>
+      <datatype>boolean</datatype>
+    </attribute>
+  </attributes>
+  <files>
+    <file>
+      <version>
+        <software_program>OpenStudio</software_program>
+        <identifier>3.4.1</identifier>
+        <min_compatible>3.4.1</min_compatible>
+      </version>
+      <filename>measure.py</filename>
+      <filetype>py</filetype>
+      <usage_type>script</usage_type>
+      <checksum>7DCE622C</checksum>
+    </file>
+  </files>
+</measure>
diff --git a/ruby/engine/test/BadMeasure/measure.rb b/ruby/engine/test/BadMeasure/measure.rb
new file mode 100644
index 00000000000..2e1681b4e8c
--- /dev/null
+++ b/ruby/engine/test/BadMeasure/measure.rb
@@ -0,0 +1,30 @@
+class BadMeasure < OpenStudio::Measure::ModelMeasure
+
+  def name
+    return "Bad Measure"
+  end
+
+  def modeler_description
+     return "The arguments method calls another_method which does a raise ValueError"
+  end
+
+  def another_method
+    raise "oops"
+  end
+
+  def arguments(model)
+    another_method
+    args = OpenStudio::Measure::OSArgumentVector.new
+
+    return args
+  end
+
+  def run(model, runner, user_arguments)
+    super(model, runner, user_arguments)
+
+    return true
+  end
+
+end
+
+BadMeasure.new.registerWithApplication
diff --git a/ruby/engine/test/BadMeasure/measure.xml b/ruby/engine/test/BadMeasure/measure.xml
new file mode 100644
index 00000000000..f28b1c73009
--- /dev/null
+++ b/ruby/engine/test/BadMeasure/measure.xml
@@ -0,0 +1,54 @@
+<?xml version="1.0"?>
+<measure>
+  <schema_version>3.1</schema_version>
+  <name>bad_measure</name>
+  <uid>812d3ebf-c89b-4b93-b400-110ca060b2bb</uid>
+  <version_id>4b3097c0-9874-4478-8e7f-45fa952a91eb</version_id>
+  <version_modified>2023-11-10T10:59:53Z</version_modified>
+  <xml_checksum>EB1A0C08</xml_checksum>
+  <class_name>BadMeasure</class_name>
+  <display_name>Bad Measure</display_name>
+  <description></description>
+  <modeler_description>The arguments method calls another_method which does a raise ValueError</modeler_description>
+  <arguments />
+  <outputs />
+  <provenances />
+  <tags>
+    <tag>Envelope.Opaque</tag>
+  </tags>
+  <attributes>
+    <attribute>
+      <name>Measure Function</name>
+      <value>Measure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Requires EnergyPlus Results</name>
+      <value>false</value>
+      <datatype>boolean</datatype>
+    </attribute>
+    <attribute>
+      <name>Measure Type</name>
+      <value>ModelMeasure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Uses SketchUp API</name>
+      <value>false</value>
+      <datatype>boolean</datatype>
+    </attribute>
+  </attributes>
+  <files>
+    <file>
+      <version>
+        <software_program>OpenStudio</software_program>
+        <identifier>0.11.5</identifier>
+        <min_compatible>0.11.5</min_compatible>
+      </version>
+      <filename>measure.rb</filename>
+      <filetype>rb</filetype>
+      <usage_type>script</usage_type>
+      <checksum>D22AC6AC</checksum>
+    </file>
+  </files>
+</measure>
diff --git a/ruby/engine/test/RubyEngine_GTest.cpp b/ruby/engine/test/RubyEngine_GTest.cpp
new file mode 100644
index 00000000000..a83529eeb8f
--- /dev/null
+++ b/ruby/engine/test/RubyEngine_GTest.cpp
@@ -0,0 +1,150 @@
+/***********************************************************************************************************************
+*  OpenStudio(R), Copyright (c) Alliance for Sustainable Energy, LLC.
+*  See also https://openstudio.net/license
+***********************************************************************************************************************/
+
+#include <gtest/gtest.h>
+
+#include "measure/ModelMeasure.hpp"
+#include "measure/OSArgument.hpp"
+#include "measure/OSMeasure.hpp"
+#include "model/Model.hpp"
+#include "scriptengine/ScriptEngine.hpp"
+
+#include <fmt/format.h>
+
+#include <regex>
+
+class RubyEngineFixture : public testing::Test
+{
+ public:
+  static openstudio::path getScriptPath(const std::string& classAndDirName) {
+    openstudio::path scriptPath = openstudio::getApplicationSourceDirectory() / fmt::format("ruby/engine/test/{}/measure.rb", classAndDirName);
+    OS_ASSERT(openstudio::filesystem::is_regular_file(scriptPath));
+    return scriptPath;
+  }
+
+  static std::string stripAddressFromErrorMessage(const std::string& error_message) {
+    static std::regex object_address_re("0x[[:alnum:]]*>");
+
+    return std::regex_replace(error_message, object_address_re, "ADDRESS>");
+  }
+
+ protected:
+  // initialize for each test
+  virtual void SetUp() override {
+    std::vector<std::string> args;
+    thisEngine = std::make_unique<openstudio::ScriptEngineInstance>("rubyengine", args);
+
+    (*thisEngine)->registerType<openstudio::measure::OSMeasure*>("openstudio::measure::OSMeasure *");
+    (*thisEngine)->registerType<openstudio::measure::ModelMeasure*>("openstudio::measure::ModelMeasure *");
+    (*thisEngine)->exec("OpenStudio::init_rest_of_openstudio()");
+  }
+  // tear down after each test
+  virtual void TearDown() override {
+    // Want to ensure the engine is reset regardless of the test outcome, or it may throw a segfault
+    thisEngine->reset();
+  }
+
+  std::unique_ptr<openstudio::ScriptEngineInstance> thisEngine;
+};
+
+TEST_F(RubyEngineFixture, BadMeasure) {
+
+  const std::string classAndDirName = "BadMeasure";
+
+  const auto scriptPath = getScriptPath(classAndDirName);
+  auto measureScriptObject = (*thisEngine)->loadMeasure(scriptPath, classAndDirName);
+  auto* measurePtr = (*thisEngine)->getAs<openstudio::measure::ModelMeasure*>(measureScriptObject);
+
+  ASSERT_EQ(measurePtr->name(), "Bad Measure");
+
+  std::string expected_exception = fmt::format(R"(SWIG director method error. RuntimeError: oops
+
+Traceback (most recent call last):
+{0}:12:in `another_method'
+{0}:16:in `arguments')",
+                                               scriptPath.generic_string());
+
+  openstudio::model::Model model;
+  try {
+    measurePtr->arguments(model);
+    ASSERT_FALSE(true) << "Expected measure arguments(model) to throw";
+  } catch (std::exception& e) {
+    std::string error = e.what();
+    EXPECT_EQ(expected_exception, error);
+  }
+}
+
+TEST_F(RubyEngineFixture, WrongMethodMeasure) {
+
+  const std::string classAndDirName = "WrongMethodMeasure";
+
+  const auto scriptPath = getScriptPath(classAndDirName);
+  auto measureScriptObject = (*thisEngine)->loadMeasure(scriptPath, classAndDirName);
+  auto* measurePtr = (*thisEngine)->getAs<openstudio::measure::ModelMeasure*>(measureScriptObject);
+
+  ASSERT_EQ(measurePtr->name(), "Wrong Method Measure");
+
+  std::string expected_exception =
+    fmt::format(R"(SWIG director method error. NoMethodError: undefined method `nonExistingMethod' for #<OpenStudio::Model::Model:ADDRESS>
+
+Traceback (most recent call last):
+{0}:12:in `arguments')",
+                scriptPath.generic_string());
+
+  openstudio::model::Model model;
+  try {
+    measurePtr->arguments(model);
+    ASSERT_FALSE(true) << "Expected measure arguments(model) to throw";
+  } catch (std::exception& e) {
+    std::string error = e.what();
+    EXPECT_EQ(expected_exception, stripAddressFromErrorMessage(error));
+  }
+}
+
+TEST_F(RubyEngineFixture, StackLevelTooDeepMeasure) {
+
+  const std::string classAndDirName = "StackLevelTooDeepMeasure";
+
+  const auto scriptPath = getScriptPath(classAndDirName);
+  auto measureScriptObject = (*thisEngine)->loadMeasure(scriptPath, classAndDirName);
+  auto* measurePtr = (*thisEngine)->getAs<openstudio::measure::ModelMeasure*>(measureScriptObject);
+
+  ASSERT_EQ(measurePtr->name(), "Stack Level Too Deep Measure");
+
+  std::string expected_exception = fmt::format(R"(SWIG director method error. SystemStackError: stack level too deep
+
+Traceback (most recent call last):
+{0}:16:in `arguments'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+	... 10061 levels...
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s'
+{0}:12:in `s')",
+                                               scriptPath.generic_string());
+
+  openstudio::model::Model model;
+  try {
+    measurePtr->arguments(model);
+    ASSERT_FALSE(true) << "Expected measure arguments(model) to throw";
+  } catch (std::exception& e) {
+    std::string error = e.what();
+    EXPECT_EQ(expected_exception, stripAddressFromErrorMessage(error));
+  }
+}
+
diff --git a/ruby/engine/test/StackLevelTooDeepMeasure/measure.rb b/ruby/engine/test/StackLevelTooDeepMeasure/measure.rb
new file mode 100644
index 00000000000..23c9dc5c339
--- /dev/null
+++ b/ruby/engine/test/StackLevelTooDeepMeasure/measure.rb
@@ -0,0 +1,30 @@
+class StackLevelTooDeepMeasure < OpenStudio::Measure::ModelMeasure
+
+  def name
+    return "Stack Level Too Deep Measure"
+  end
+
+  def modeler_description
+     return "The arguments method calls an infinitely recursive function"
+  end
+
+  def s(x)
+    return x * s(x-1)
+  end
+
+  def arguments(model)
+    s(10)
+    args = OpenStudio::Measure::OSArgumentVector.new
+
+    return args
+  end
+
+  def run(model, runner, user_arguments)
+    super(model, runner, user_arguments)
+
+    return true
+  end
+
+end
+
+StackLevelTooDeepMeasure.new.registerWithApplication
diff --git a/ruby/engine/test/StackLevelTooDeepMeasure/measure.xml b/ruby/engine/test/StackLevelTooDeepMeasure/measure.xml
new file mode 100644
index 00000000000..7fe301f57e8
--- /dev/null
+++ b/ruby/engine/test/StackLevelTooDeepMeasure/measure.xml
@@ -0,0 +1,59 @@
+<?xml version="1.0"?>
+<measure>
+  <schema_version>3.1</schema_version>
+  <name>stack_level_too_deep_measure</name>
+  <uid>22222222-c89b-4b93-b400-220ca060b2bb</uid>
+  <version_id>b9722f6a-e289-4915-8297-6dad9bf08556</version_id>
+  <version_modified>2023-11-10T13:43:11Z</version_modified>
+  <xml_checksum>5872977E</xml_checksum>
+  <class_name>StackLevelTooDeepMeasure</class_name>
+  <display_name>Stack Level Too Deep Measure</display_name>
+  <description></description>
+  <modeler_description>The arguments method calls an infinitely recursive function</modeler_description>
+  <arguments />
+  <outputs />
+  <provenances />
+  <tags>
+    <tag>Envelope.Opaque</tag>
+  </tags>
+  <attributes>
+    <attribute>
+      <name>Measure Function</name>
+      <value>Measure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Requires EnergyPlus Results</name>
+      <value>false</value>
+      <datatype>boolean</datatype>
+    </attribute>
+    <attribute>
+      <name>Measure Type</name>
+      <value>ModelMeasure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Uses SketchUp API</name>
+      <value>false</value>
+      <datatype>boolean</datatype>
+    </attribute>
+    <attribute>
+      <name>Measure Language</name>
+      <value>Ruby</value>
+      <datatype>string</datatype>
+    </attribute>
+  </attributes>
+  <files>
+    <file>
+      <version>
+        <software_program>OpenStudio</software_program>
+        <identifier>3.4.1</identifier>
+        <min_compatible>3.4.1</min_compatible>
+      </version>
+      <filename>measure.rb</filename>
+      <filetype>rb</filetype>
+      <usage_type>script</usage_type>
+      <checksum>9AB518E3</checksum>
+    </file>
+  </files>
+</measure>
diff --git a/ruby/engine/test/WrongMethodMeasure/measure.rb b/ruby/engine/test/WrongMethodMeasure/measure.rb
new file mode 100644
index 00000000000..6f9eb26c22a
--- /dev/null
+++ b/ruby/engine/test/WrongMethodMeasure/measure.rb
@@ -0,0 +1,26 @@
+class WrongMethodMeasure < OpenStudio::Measure::ModelMeasure
+
+  def name
+    return "Wrong Method Measure"
+  end
+
+  def modeler_description
+     return "The arguments method calls a non existing method on the model passed as argument"
+  end
+
+  def arguments(model)
+    model.nonExistingMethod()
+    args = OpenStudio::Measure::OSArgumentVector.new
+
+    return args
+  end
+
+  def run(model, runner, user_arguments)
+    super(model, runner, user_arguments)
+
+    return true
+  end
+
+end
+
+WrongMethodMeasure.new.registerWithApplication
diff --git a/ruby/engine/test/WrongMethodMeasure/measure.xml b/ruby/engine/test/WrongMethodMeasure/measure.xml
new file mode 100644
index 00000000000..812f4c995d2
--- /dev/null
+++ b/ruby/engine/test/WrongMethodMeasure/measure.xml
@@ -0,0 +1,59 @@
+<?xml version="1.0"?>
+<measure>
+  <schema_version>3.1</schema_version>
+  <name>wrong_method_measure</name>
+  <uid>11111111-c89b-4b93-b400-110ca060b2bb</uid>
+  <version_id>8c8f5697-b6e0-4ae0-90f9-cace0f872d41</version_id>
+  <version_modified>2023-11-10T11:02:27Z</version_modified>
+  <xml_checksum>5872977E</xml_checksum>
+  <class_name>WrongMethodMeasure</class_name>
+  <display_name>Wrong Method Measure</display_name>
+  <description></description>
+  <modeler_description>The arguments method calls a non existing method on the model passed as argument</modeler_description>
+  <arguments />
+  <outputs />
+  <provenances />
+  <tags>
+    <tag>Envelope.Opaque</tag>
+  </tags>
+  <attributes>
+    <attribute>
+      <name>Measure Function</name>
+      <value>Measure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Requires EnergyPlus Results</name>
+      <value>false</value>
+      <datatype>boolean</datatype>
+    </attribute>
+    <attribute>
+      <name>Measure Type</name>
+      <value>ModelMeasure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Uses SketchUp API</name>
+      <value>false</value>
+      <datatype>boolean</datatype>
+    </attribute>
+    <attribute>
+      <name>Measure Language</name>
+      <value>Ruby</value>
+      <datatype>string</datatype>
+    </attribute>
+  </attributes>
+  <files>
+    <file>
+      <version>
+        <software_program>OpenStudio</software_program>
+        <identifier>3.4.1</identifier>
+        <min_compatible>3.4.1</min_compatible>
+      </version>
+      <filename>measure.rb</filename>
+      <filetype>rb</filetype>
+      <usage_type>script</usage_type>
+      <checksum>BB98326D</checksum>
+    </file>
+  </files>
+</measure>