Repository: NREL/OpenStudio (language: python)

Issue:
Python Import Error on macOS after upgrading to E+ 24.2.0a
Issue overview
--------------

The `Run_RubyPythonPlugin` test is failing. This is a codesigning issue, so mac only. It happens when you try to run the "PythonPlugin" feature of E+.


            

## Current Behavior

https://ci.openstudio.net/blue/organizations/jenkins/openstudio-incremental-osx/detail/PR-5252/2/pipeline#step-84-log-42

## Expected Behavior

It should work.

## Possible Solution

Basically this is because E+ has codesigning on macOS, and it does not have a specific entitlement that allows to load unsigned (or differently signed) library at runtime (via dlopen).

The E+ exe is signed, the libpython is signed, all with the EnergyPlus codesigning certificate.
We pip install some dependencies that have a native extension, at least numpy and pandas. these are NOT signed. So it fails at runtime

Possible solutions:

* Rebuild an E+ package that adds an entitlements to allow this specific use case. I think it's the `com.apple.security.cs.disable-library-validation` one based on some preliminary testing 
* Implement #5235 and override the signature with OpenStudio's
* Remove the E+ package signing altogether (via CmakeLists, after downloading it)


## Details

### Environment

Some additional details about your environment for this issue (if relevant):
 - Platform (Operating system, version): macOS
 - Version of OpenStudio (if using an intermediate build, include SHA): develop

### Context

_Originally posted by @jmarrec in https://github.com/NREL/OpenStudio/issues/5242#issuecomment-2398059563_

Found again in https://github.com/NREL/OpenStudio/pull/5279

Fix #5233 - Validate OSW measures before running
Pull request overview
---------------------

 - Fixes #5233 


### Pull Request Author

<!--- Add to this list or remove from it as applicable.  This is a simple templated set of guidelines. -->

 - [ ] Model API Changes / Additions
 - [ ] Any new or modified fields have been implemented in the EnergyPlus ForwardTranslator (and ReverseTranslator as appropriate)
 - [ ] Model API methods are tested (in `src/model/test`)
 - [ ] EnergyPlus ForwardTranslator Tests (in `src/energyplus/Test`)
 - [ ] If a new object or method, added a test in NREL/OpenStudio-resources: Add Link
 - [ ] If needed, added VersionTranslation rules for the objects (`src/osversion/VersionTranslator.cpp`)
 - [ ] Verified that C# bindings built fine on Windows, partial classes used as needed, etc.
 - [ ] All new and existing tests passes
 - [ ] If methods have been deprecated, update rest of code to use the new methods

**Labels:**

 - [ ] If change to an IDD file, add the label `IDDChange`
 - [ ] If breaking existing API, add the label `APIChange`
 - [ ] If deemed ready, add label `Pull Request - Ready for CI` so that CI builds your PR

### Review Checklist

This will not be exhaustively relevant to every PR.
 - [ ] Perform a Code Review on GitHub
 - [ ] Code Style, strip trailing whitespace, etc.
 - [ ] All related changes have been implemented: model changes, model tests, FT changes, FT tests, VersionTranslation, OS App
 - [ ] Labeling is ok
 - [ ] If defect, verify by running develop branch and reproducing defect, then running PR and reproducing fix
 - [ ] If feature, test running new feature, try creative ways to break it
 - [ ] CI status: all green or justified



Discussion / hints:
I like option 2, implement code signing. That said, we have a lot of binaries, can we track them all down to get them signed? @wenyikuang this is your department. 
I don't have a preference, although at some point in future, I'm hoping we can have more integrated CI through UO > OS > E+ where we can work off of develop of each tool vs. waiting for a release. Hoping to catch things before they get into develop. This may be irrelevant. to which approach we take, but I thought I'd point it out incase it wasn't.
Codesigning is the logical choice really. Problem is, I estimate about 20-40 hours of work (based on my experience setting it up for E+ and OSApp), assuming we realdy have an Apple certificate for OpenStudio available. So that's going to be problematic to get it into 3.9.0 probably.
I'll tag this as 3.9 so it is in project board and we can talk about it internally on iteration call tomorrow.


Fix patch:
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 05fe5faf2f9..2d833b0f85c 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -161,7 +161,7 @@ get_directory_property(hasParent PARENT_DIRECTORY)
 
 # TODO: Modify the more specific variables as needed to indicate prerelease, etc
 # Keep in beta in-between release cycles. Set to empty string (or comment out) for official)
-set(PROJECT_VERSION_PRERELEASE "rc2")
+set(PROJECT_VERSION_PRERELEASE "rc3")
 
 # OpenStudio version: Only include Major.Minor.Patch, eg "3.0.0", even if you have a prerelease tag
 set(OPENSTUDIO_VERSION "${PROJECT_VERSION_MAJOR}.${PROJECT_VERSION_MINOR}.${PROJECT_VERSION_PATCH}")
@@ -638,11 +638,15 @@ endif()
 
 if(UNIX)
   if(APPLE)
+    # TODO: temp for #5281 - Resigned with a different entitlements to avoid an issue when we pip install stuff into the E+ dir with native comps (numpy for eg)
+    set(ENERGYPLUS_REPO "jmarrec")
+    set(ENERGYPLUS_RELEASE_NAME "v24.2.0a-entitlements")
+
     if (ARCH MATCHES "arm64")
-      set(ENERGYPLUS_EXPECTED_HASH f36afc055a675ae95523d6f41ec478c0)
+      set(ENERGYPLUS_EXPECTED_HASH 29616d6aa23e2fc0f71362da9794ef08)
       set(ENERGYPLUS_PLATFORM "Darwin-macOS13-arm64")
     else()
-      set(ENERGYPLUS_EXPECTED_HASH b2003c461277c0bd4e91a4c003b350ac)
+      set(ENERGYPLUS_EXPECTED_HASH 21bdad40fbc560b4b382469ef9f96936)
       set(ENERGYPLUS_PLATFORM "Darwin-macOS12.1-x86_64")
     endif()
   elseif(LSB_RELEASE_ID_SHORT MATCHES "CentOS")
@@ -843,7 +847,7 @@ if(BUILD_CLI)
   set(OPENSTUDIO_GEMS_BASEURL "http://openstudio-resources.s3.amazonaws.com/dependencies")
 
   # TODO: temp
-  set(OPENSTUDIO_GEMS_BASEURL "https://github.com/NREL/openstudio-gems/releases/download/v3.9.0-RC2")
+  set(OPENSTUDIO_GEMS_BASEURL "https://github.com/NREL/openstudio-gems/releases/download/v3.9.0-RC3")
 
   # To use the package produced by a PR to https://github.com/NREL/openstudio-gems
   set(USE_OPENSTUDIO_GEMS_PR FALSE)
@@ -855,19 +859,19 @@ if(BUILD_CLI)
   if(UNIX)
      if(APPLE)
        if (ARCH MATCHES arm64)
-         set(OPENSTUDIO_GEMS_ZIP_FILENAME "openstudio3-gems-20241101-darwin_arm64-3.2.2.tar.gz")
-         set(OPENSTUDIO_GEMS_ZIP_EXPECTED_MD5 "c4cf7754444937e5df496f23af6d76ad")
+         set(OPENSTUDIO_GEMS_ZIP_FILENAME "openstudio3-gems-20241112-darwin_arm64-3.2.2.tar.gz")
+         set(OPENSTUDIO_GEMS_ZIP_EXPECTED_MD5 "cb6a884366780c8ba5da2852ca60e98a")
        else()
-         set(OPENSTUDIO_GEMS_ZIP_FILENAME "openstudio3-gems-20241101-darwin-3.2.2.tar.gz")
-         set(OPENSTUDIO_GEMS_ZIP_EXPECTED_MD5 "5754b6ba7cff435d33f6c4dcdcd6bfc4")
+         set(OPENSTUDIO_GEMS_ZIP_FILENAME "openstudio3-gems-20241112-darwin-3.2.2.tar.gz")
+         set(OPENSTUDIO_GEMS_ZIP_EXPECTED_MD5 "ce81fcf1bf707281082cde3d87dac114")
        endif()
      else()
        if (ARCH MATCHES "arm64")
          set(OPENSTUDIO_GEMS_ZIP_FILENAME "openstudio3-gems-20240517-linux_arm64-3.2.2.tar.gz")
          set(OPENSTUDIO_GEMS_ZIP_EXPECTED_MD5 "f6d094a3bdaf1476ce7911e74276993b")
        else()
-         set(OPENSTUDIO_GEMS_ZIP_FILENAME "openstudio3-gems-20241101-linux-3.2.2.tar.gz")
-         set(OPENSTUDIO_GEMS_ZIP_EXPECTED_MD5 "972a9246fd2a6d43bc7eb23e2ad2b715")
+         set(OPENSTUDIO_GEMS_ZIP_FILENAME "openstudio3-gems-20241112-linux-3.2.2.tar.gz")
+         set(OPENSTUDIO_GEMS_ZIP_EXPECTED_MD5 "7551691c842a86cc8198959b5d0f1ee9")
        endif()
        if (USE_OPENSTUDIO_GEMS_PR)
          set(OPENSTUDIO_GEMS_BASEURL "${OPENSTUDIO_GEMS_BASEURL}/openstudio-gems-linux/${OPENSTUDIO_GEMS_PR_NUMBER}")
@@ -875,8 +879,8 @@ if(BUILD_CLI)
      endif()
   elseif(WIN32)
     # OpenStudio gems are only supported on 64 bit windows
-    set(OPENSTUDIO_GEMS_ZIP_FILENAME "openstudio3-gems-20241101-windows-3.2.2.tar.gz")
-    set(OPENSTUDIO_GEMS_ZIP_EXPECTED_MD5 "a5f1fde7ab2555e736d3d21ba5beb683")
+    set(OPENSTUDIO_GEMS_ZIP_FILENAME "openstudio3-gems-20241112-windows-3.2.2.tar.gz")
+    set(OPENSTUDIO_GEMS_ZIP_EXPECTED_MD5 "8f1cdcca664a1d3d80908230a00981ef")
     if (USE_OPENSTUDIO_GEMS_PR)
       set(OPENSTUDIO_GEMS_BASEURL "${OPENSTUDIO_GEMS_BASEURL}/openstudio-gems-windows/${OPENSTUDIO_GEMS_PR_NUMBER}")
     endif()
diff --git a/resources/workflow/invalid_measures/missing_a_measure.osw b/resources/workflow/invalid_measures/missing_a_measure.osw
new file mode 100644
index 00000000000..2f78c0633ba
--- /dev/null
+++ b/resources/workflow/invalid_measures/missing_a_measure.osw
@@ -0,0 +1,19 @@
+{
+	"weather_file": "../../Examples/compact_osw/files/srrl_2013_amy.epw",
+	"seed_file": "../example_model.osm",
+	"measure_paths": ["../measures/"],
+	"steps": [
+		{
+			"measure_dir_name": "FakeModelMeasure",
+			"arguments": {}
+		},
+		{
+			"measure_dir_name": "NON_EXISTING_MEASURE_THIS_SHOULD_BE_CAUGHT",
+			"arguments": {}
+		},
+		{
+			"measure_dir_name": "FakeReport",
+			"arguments": {}
+		}
+	]
+}
diff --git a/resources/workflow/invalid_measures/unloadable_measure.osw b/resources/workflow/invalid_measures/unloadable_measure.osw
new file mode 100644
index 00000000000..cb5d1c88178
--- /dev/null
+++ b/resources/workflow/invalid_measures/unloadable_measure.osw
@@ -0,0 +1,19 @@
+{
+	"weather_file": "../../Examples/compact_osw/files/srrl_2013_amy.epw",
+	"seed_file": "../example_model.osm",
+	"measure_paths": ["../measures/"],
+	"steps": [
+		{
+			"measure_dir_name": "FakeModelMeasure",
+			"arguments": {}
+		},
+		{
+			"measure_dir_name": "UnloadableMeasure",
+			"arguments": {}
+		},
+		{
+			"measure_dir_name": "FakeReport",
+			"arguments": {}
+		}
+	]
+}
diff --git a/resources/workflow/invalid_measures/wrong_measure_type_order.osw b/resources/workflow/invalid_measures/wrong_measure_type_order.osw
new file mode 100644
index 00000000000..01b28e49177
--- /dev/null
+++ b/resources/workflow/invalid_measures/wrong_measure_type_order.osw
@@ -0,0 +1,15 @@
+{
+	"weather_file": "../../Examples/compact_osw/files/srrl_2013_amy.epw",
+	"seed_file": "../example_model.osm",
+	"measure_paths": ["../measures/"],
+	"steps": [
+		{
+			"measure_dir_name": "FakeReport",
+			"arguments": {}
+		},
+		{
+			"measure_dir_name": "FakeModelMeasure",
+			"arguments": {}
+		}
+	]
+}
diff --git a/resources/workflow/measures/FakeModelMeasure/measure.rb b/resources/workflow/measures/FakeModelMeasure/measure.rb
new file mode 100644
index 00000000000..68a978a9096
--- /dev/null
+++ b/resources/workflow/measures/FakeModelMeasure/measure.rb
@@ -0,0 +1,42 @@
+class FakeModelMeasure < OpenStudio::Measure::ModelMeasure
+  # human readable name
+  def name
+    # Measure name should be the title case of the class name.
+    return 'A dumb ModelMeasure'
+  end
+
+  # human readable description
+  def description
+    return 'Does nothing'
+  end
+
+  # human readable description of modeling approach
+  def modeler_description
+    return 'Just for testing'
+  end
+
+  # define the arguments that the user will input
+  def arguments(model)
+    args = OpenStudio::Measure::OSArgumentVector.new
+
+    return args
+  end
+
+  # define what happens when the measure is run
+  def run(model, runner, user_arguments)
+    super(model, runner, user_arguments)  # Do **NOT** remove this line
+
+    # use the built-in error checking
+    if !runner.validateUserArguments(arguments(model), user_arguments)
+      return false
+    end
+
+    # report final condition of model
+    runner.registerFinalCondition("The FakeModelMeasure run.")
+
+    return true
+  end
+end
+
+# register the measure to be used by the application
+FakeModelMeasure.new.registerWithApplication
diff --git a/resources/workflow/measures/FakeModelMeasure/measure.xml b/resources/workflow/measures/FakeModelMeasure/measure.xml
new file mode 100644
index 00000000000..a1fd30f66f0
--- /dev/null
+++ b/resources/workflow/measures/FakeModelMeasure/measure.xml
@@ -0,0 +1,44 @@
+<?xml version="1.0"?>
+<measure>
+  <schema_version>3.1</schema_version>
+  <name>fake_model_measure</name>
+  <uid>677b8fd3-2627-4516-b090-f6e47dc99fea</uid>
+  <version_id>162465e5-b6f6-419f-ab5f-c2fc4bd549e0</version_id>
+  <version_modified>2024-11-07T11:55:10Z</version_modified>
+  <xml_checksum>82D8F881</xml_checksum>
+  <class_name>FakeModelMeasure</class_name>
+  <display_name>A dumb ModelMeasure</display_name>
+  <description>Does nothing</description>
+  <modeler_description>Just for testing</modeler_description>
+  <arguments />
+  <outputs />
+  <provenances />
+  <tags>
+    <tag>Envelope.Form</tag>
+  </tags>
+  <attributes>
+    <attribute>
+      <name>Measure Type</name>
+      <value>ModelMeasure</value>
+      <datatype>string</datatype>
+    </attribute>
+    <attribute>
+      <name>Measure Language</name>
+      <value>Ruby</value>
+      <datatype>string</datatype>
+    </attribute>
+  </attributes>
+  <files>
+    <file>
+      <version>
+        <software_program>OpenStudio</software_program>
+        <identifier>3.9.0</identifier>
+        <min_compatible>3.9.0</min_compatible>
+      </version>
+      <filename>measure.rb</filename>
+      <filetype>rb</filetype>
+      <usage_type>script</usage_type>
+      <checksum>DDA977B0</checksum>
+    </file>
+  </files>
+</measure>
diff --git a/resources/workflow/measures/UnloadableMeasure/README.md b/resources/workflow/measures/UnloadableMeasure/README.md
new file mode 100644
index 00000000000..4459d7e1aa3
--- /dev/null
+++ b/resources/workflow/measures/UnloadableMeasure/README.md
@@ -0,0 +1,1 @@
+This doesnt have a xml
diff --git a/resources/workflow/measures/UnloadableMeasure/measure.rb b/resources/workflow/measures/UnloadableMeasure/measure.rb
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/src/alfalfa/AlfalfaJSON.cpp b/src/alfalfa/AlfalfaJSON.cpp
index 518b09946a7..9fa41f5324e 100644
--- a/src/alfalfa/AlfalfaJSON.cpp
+++ b/src/alfalfa/AlfalfaJSON.cpp
@@ -83,9 +83,9 @@ namespace alfalfa {
 
     Json::Value AlfalfaJSON_Impl::toJSON() const {
       Json::Value root;
-      for (const auto& point : m_points) {
+      for (Json::ArrayIndex i = 0; const auto& point : points()) {
         // No guard here as the toJSON call will throw an exception if the id does not exist.
-        root[point.id().get()] = point.toJSON();
+        root[i++] = point.toJSON();
       }
       return root;
     }
diff --git a/src/alfalfa/AlfalfaPoint.cpp b/src/alfalfa/AlfalfaPoint.cpp
index b3dfb301187..9ffd9a03c59 100644
--- a/src/alfalfa/AlfalfaPoint.cpp
+++ b/src/alfalfa/AlfalfaPoint.cpp
@@ -7,12 +7,11 @@
 
 #include <memory>
 #include <string_view>
+#include <algorithm>
 
 namespace openstudio {
 namespace alfalfa {
 
-  static constexpr std::string_view ID_VALID_CHARS_MSG = "IDs can only contain letters, numbers, and the following special characters _-[]():";
-
   namespace detail {
 
     AlfalfaPoint_Impl::AlfalfaPoint_Impl(const std::string& display_name) {
@@ -22,11 +21,7 @@ namespace alfalfa {
     Json::Value AlfalfaPoint_Impl::toJSON() const {
       Json::Value point;
 
-      if (auto id_ = id()) {
-        point["id"] = *id_;
-      } else {
-        throw std::runtime_error(fmt::format("Point requires a valid ID for export. {}", ID_VALID_CHARS_MSG));
-      }
+      point["id"] = id();
       point["name"] = displayName();
       if (auto input_ = input()) {
         point["input"]["type"] = input_->typeName();
@@ -80,24 +75,17 @@ namespace alfalfa {
     }
 
     void AlfalfaPoint_Impl::setId(const std::string& id) {
-      if (isValidId(id)) {
-        m_id = id;
-      } else {
-        throw std::runtime_error(ID_VALID_CHARS_MSG.data());
+      if (id.empty()) {
+        throw std::runtime_error("Id must have non-zero length");
       }
+      m_id = toIdString(id);
     }
 
-    boost::optional<std::string> AlfalfaPoint_Impl::id() const {
-      boost::optional<std::string> result = m_id;
-      if (!result.is_initialized()) {
-        std::string id = toIdString(displayName());
-        if (isValidId(id)) {
-          result = id;
-        } else {
-          LOG(Warn, fmt::format("Display name '{}' does not produce a valid point ID. Manually set a valid ID or export will fail.", displayName()));
-        }
+    std::string AlfalfaPoint_Impl::id() const {
+      if (m_id.empty()) {
+        return toIdString(displayName());
       }
-      return result;
+      return m_id;
     }
 
     void AlfalfaPoint_Impl::setDisplayName(const std::string& display_name) {
@@ -105,12 +93,6 @@ namespace alfalfa {
         throw std::runtime_error("Display name must have non-zero length");
       }
       m_display_name = display_name;
-      if (!m_id.is_initialized()) {
-        const std::string id = toIdString(display_name);
-        if (!isValidId(id)) {
-          LOG(Warn, fmt::format("Display name '{}' does not produce a valid point ID. Manually set a valid ID or export will fail.", display_name));
-        }
-      }
     }
 
     std::string AlfalfaPoint_Impl::displayName() const {
@@ -125,17 +107,13 @@ namespace alfalfa {
       return m_optional;
     }
 
-    bool AlfalfaPoint_Impl::isValidId(const std::string& id) {
-      return !id.empty() && boost::regex_match(id, boost::regex(R"(^[A-Za-z0-9_\-\[\]:()]*$)"));
-    }
-
     std::string AlfalfaPoint_Impl::toIdString(const std::string& str) {
-      return boost::regex_replace(str, boost::regex(" "), "_");
+      std::string id_string = str;
+      std::replace(id_string.begin(), id_string.end(), ' ', '_');
+      return id_string;
     }
   }  // namespace detail
 
-  // AlfalfaPoint::AlfalfaPoint(const AlfalfaPoint& point) : m_impl(point.m_impl) {}
-
   AlfalfaPoint::AlfalfaPoint(const std::string& display_name) : m_impl(std::make_shared<detail::AlfalfaPoint_Impl>(display_name)) {}
 
   void AlfalfaPoint::setInput(const AlfalfaComponent& component) {
@@ -162,7 +140,7 @@ namespace alfalfa {
     return m_impl->units();
   }
 
-  boost::optional<std::string> AlfalfaPoint::id() const {
+  std::string AlfalfaPoint::id() const {
     return m_impl->id();
   }
 
diff --git a/src/alfalfa/AlfalfaPoint.hpp b/src/alfalfa/AlfalfaPoint.hpp
index 3ed3ee51589..1ca63e380a4 100644
--- a/src/alfalfa/AlfalfaPoint.hpp
+++ b/src/alfalfa/AlfalfaPoint.hpp
@@ -69,7 +69,7 @@ namespace alfalfa {
     /**
      * Get id of point. By default this will be a version of the display name with spaces removed.
      */
-    boost::optional<std::string> id() const;
+    std::string id() const;
 
     /**
      * Set id of point. This is the component which will uniquely identify the point in the API.
diff --git a/src/alfalfa/AlfalfaPoint_Impl.hpp b/src/alfalfa/AlfalfaPoint_Impl.hpp
index f6d0810a65c..bc311434e5f 100644
--- a/src/alfalfa/AlfalfaPoint_Impl.hpp
+++ b/src/alfalfa/AlfalfaPoint_Impl.hpp
@@ -32,7 +32,7 @@ namespace alfalfa {
 
       boost::optional<std::string> units() const;
 
-      boost::optional<std::string> id() const;
+      std::string id() const;
 
       void setId(const std::string& id);
 
@@ -47,13 +47,11 @@ namespace alfalfa {
      private:
       static std::string toIdString(const std::string& str);
 
-      static bool isValidId(const std::string& id);
-
       boost::optional<AlfalfaComponent> m_input;
       boost::optional<AlfalfaComponent> m_output;
       std::string m_display_name;
       boost::optional<std::string> m_units;
-      boost::optional<std::string> m_id;
+      std::string m_id;
       bool m_optional = true;
 
       // configure logging
diff --git a/src/cli/CMakeLists.txt b/src/cli/CMakeLists.txt
index 179c35f7a04..e79684fcdfc 100644
--- a/src/cli/CMakeLists.txt
+++ b/src/cli/CMakeLists.txt
@@ -270,6 +270,63 @@ if(BUILD_TESTING)
     PASS_REGULAR_EXPRESSION "HI FROM ERB PYTHON PLUGIN[\r\n\t ]*HI FROM JINJA PYTHON PLUGIN"
   )
 
+  # ======================== Workflows should fail ========================
+  add_test(NAME OpenStudioCLI.Run_Validate.MissingAMeasure
+    COMMAND $<TARGET_FILE:openstudio> run --show-stdout -w missing_a_measure.osw
+    WORKING_DIRECTORY "${PROJECT_BINARY_DIR}/resources/workflow/invalid_measures/"
+  )
+  set_tests_properties(OpenStudioCLI.Run_Validate.MissingAMeasure PROPERTIES
+    WILL_FAIL TRUE
+    RESOURCE_LOCK "invalid_measures"
+  )
+
+  add_test(NAME OpenStudioCLI.Run_Validate.UnloadableMeasure
+    COMMAND $<TARGET_FILE:openstudio> run --show-stdout -w unloadable_measure.osw
+    WORKING_DIRECTORY "${PROJECT_BINARY_DIR}/resources/workflow/invalid_measures/"
+  )
+  set_tests_properties(OpenStudioCLI.Run_Validate.UnloadableMeasure PROPERTIES
+    WILL_FAIL TRUE
+    RESOURCE_LOCK "invalid_measures"
+  )
+
+  add_test(NAME OpenStudioCLI.Run_Validate.WrongMeasureTypeOrder
+    COMMAND $<TARGET_FILE:openstudio> run --show-stdout -w wrong_measure_type_order.osw
+    WORKING_DIRECTORY "${PROJECT_BINARY_DIR}/resources/workflow/invalid_measures/"
+  )
+  set_tests_properties(OpenStudioCLI.Run_Validate.WrongMeasureTypeOrder PROPERTIES
+    WILL_FAIL TRUE
+    RESOURCE_LOCK "invalid_measures"
+  )
+
+  # Classic
+  add_test(NAME OpenStudioCLI.Classic.Run_Validate.MissingAMeasure
+    COMMAND $<TARGET_FILE:openstudio> classic run --show-stdout -w missing_a_measure.osw
+    WORKING_DIRECTORY "${PROJECT_BINARY_DIR}/resources/workflow/invalid_measures/"
+  )
+  set_tests_properties(OpenStudioCLI.Classic.Run_Validate.MissingAMeasure PROPERTIES
+    WILL_FAIL TRUE
+    RESOURCE_LOCK "invalid_measures"
+  )
+
+  add_test(NAME OpenStudioCLI.Classic.Run_Validate.UnloadableMeasure
+    COMMAND $<TARGET_FILE:openstudio> classic run --show-stdout -w unloadable_measure.osw
+    WORKING_DIRECTORY "${PROJECT_BINARY_DIR}/resources/workflow/invalid_measures/"
+  )
+  set_tests_properties(OpenStudioCLI.Classic.Run_Validate.UnloadableMeasure PROPERTIES
+    WILL_FAIL TRUE
+    RESOURCE_LOCK "invalid_measures"
+  )
+
+  add_test(NAME OpenStudioCLI.Classic.Run_Validate.WrongMeasureTypeOrder
+    COMMAND $<TARGET_FILE:openstudio> classic run --show-stdout -w wrong_measure_type_order.osw
+    WORKING_DIRECTORY "${PROJECT_BINARY_DIR}/resources/workflow/invalid_measures/"
+  )
+  set_tests_properties(OpenStudioCLI.Classic.Run_Validate.WrongMeasureTypeOrder PROPERTIES
+    WILL_FAIL TRUE
+    RESOURCE_LOCK "invalid_measures"
+  )
+  # ====================== End Workflows should fail ======================
+
   if (Pytest_AVAILABLE)
     add_test(NAME OpenStudioCLI.test_loglevel
       COMMAND ${Python_EXECUTABLE} -m pytest --verbose ${Pytest_XDIST_OPTS} --os-cli-path $<TARGET_FILE:openstudio> "${CMAKE_CURRENT_SOURCE_DIR}/test/test_loglevel.py"
diff --git a/src/utilities/filetypes/WorkflowJSON.cpp b/src/utilities/filetypes/WorkflowJSON.cpp
index 1ba116ceea1..4bbc8a60393 100644
--- a/src/utilities/filetypes/WorkflowJSON.cpp
+++ b/src/utilities/filetypes/WorkflowJSON.cpp
@@ -46,10 +46,12 @@ namespace detail {
     std::string formattedErrors;
     bool parsingSuccessful = Json::parseFromStream(rbuilder, ss, &m_value, &formattedErrors);
 
+    openstudio::path p;
+
     if (!parsingSuccessful) {
 
       // see if this is a path
-      openstudio::path p = toPath(s);
+      p = toPath(s);
       if (boost::filesystem::exists(p) && boost::filesystem::is_regular_file(p)) {
         // open file
         std::ifstream ifs(openstudio::toSystemFilename(p));
@@ -65,6 +67,10 @@ namespace detail {
 
     parseSteps();
     parseRunOptions();
+
+    if (!p.empty()) {
+      setOswPath(p, false);
+    }
   }
 
   WorkflowJSON_Impl::WorkflowJSON_Impl(const openstudio::path& p) {
@@ -851,6 +857,65 @@ namespace detail {
     }
   }
 
+  bool WorkflowJSON_Impl::validateMeasures() const {
+    // TODO: should we exit early, or return all problems found?
+
+    bool result = true;
+    MeasureType state = MeasureType::ModelMeasure;
+
+    for (size_t i = 0; const auto& step : m_steps) {
+      LOG(Debug, "Validating step " << i);
+      if (auto step_ = step.optionalCast<MeasureStep>()) {
+        // Not calling getBCLMeasure because I want to mimic workflow-gem and be as explicit as possible about what went wrong
+        const auto measureDirName = step_->measureDirName();
+        auto measurePath_ = findMeasure(measureDirName);
+        if (!measurePath_) {
+          LOG(Error, "Cannot find measure '" << measureDirName << "'");
+          result = false;
+          continue;
+        }
+        auto bclMeasure_ = BCLMeasure::load(*measurePath_);
+        if (!bclMeasure_) {
+          LOG(Error, "Cannot load measure '" << measureDirName << "' at '" << *measurePath_ << "'");
+          result = false;
+          continue;
+        }
+
+        // Ensure that measures are in order, i.e. no OS after E+, E+ or OS after Reporting
+        const auto measureType = bclMeasure_->measureType();
+
+        if (measureType == MeasureType::ModelMeasure) {
+          if (state == MeasureType::EnergyPlusMeasure) {
+            LOG(Error, "OpenStudio measure '" << measureDirName << "' called after transition to EnergyPlus.");
+            result = false;
+          }
+          if (state == MeasureType::ReportingMeasure) {
+            LOG(Error, "OpenStudio measure '" << measureDirName << "' called after Energyplus simulation.");
+            result = false;
+          }
+
+        } else if (measureType == MeasureType::EnergyPlusMeasure) {
+          if (state == MeasureType::ReportingMeasure) {
+            LOG(Error, "EnergyPlus measure '" << measureDirName << "' called after Energyplus simulation.");
+            result = false;
+          }
+          if (state == MeasureType::ModelMeasure) {
+            state = MeasureType::EnergyPlusMeasure;
+          }
+
+        } else if (measureType == MeasureType::ReportingMeasure) {
+          state = MeasureType::ReportingMeasure;
+
+        } else {
+          LOG(Error, "MeasureType " << measureType.valueName() << " of measure '" << measureDirName << "' is not supported");
+          result = false;
+        }
+      }
+      ++i;
+    }
+
+    return result;
+  }
 }  // namespace detail
 
 WorkflowJSON::WorkflowJSON() : m_impl(std::shared_ptr<detail::WorkflowJSON_Impl>(new detail::WorkflowJSON_Impl())) {}
@@ -1123,6 +1188,10 @@ void WorkflowJSON::resetRunOptions() {
   getImpl<detail::WorkflowJSON_Impl>()->resetRunOptions();
 }
 
+bool WorkflowJSON::validateMeasures() const {
+  return getImpl<detail::WorkflowJSON_Impl>()->validateMeasures();
+}
+
 std::ostream& operator<<(std::ostream& os, const WorkflowJSON& workflowJSON) {
   os << workflowJSON.string();
   return os;
diff --git a/src/utilities/filetypes/WorkflowJSON.hpp b/src/utilities/filetypes/WorkflowJSON.hpp
index 1ac097baa7f..5d6c4dd693b 100644
--- a/src/utilities/filetypes/WorkflowJSON.hpp
+++ b/src/utilities/filetypes/WorkflowJSON.hpp
@@ -233,6 +233,9 @@ class UTILITIES_API WorkflowJSON
   /** Reset RunOptions for this workflow. */
   void resetRunOptions();
 
+  /** Checks that all measures in the Workflow can be found, and are in the correct order (ModelMeasure > EnergyPlusMeasure > ReportingMeasure) */
+  bool validateMeasures() const;
+
  protected:
   // get the impl
   template <typename T>
diff --git a/src/utilities/filetypes/WorkflowJSON_Impl.hpp b/src/utilities/filetypes/WorkflowJSON_Impl.hpp
index 65d65c52e83..03be7a07339 100644
--- a/src/utilities/filetypes/WorkflowJSON_Impl.hpp
+++ b/src/utilities/filetypes/WorkflowJSON_Impl.hpp
@@ -153,6 +153,8 @@ namespace detail {
     // Emitted on any change
     Nano::Signal<void()> onChange;
 
+    bool validateMeasures() const;
+
    private:
     REGISTER_LOGGER("openstudio.WorkflowJSON");
 
diff --git a/src/workflow/RunInitialization.cpp b/src/workflow/RunInitialization.cpp
index a35ba499a64..0d691b2dba5 100644
--- a/src/workflow/RunInitialization.cpp
+++ b/src/workflow/RunInitialization.cpp
@@ -54,10 +54,17 @@ void OSWorkflow::runInitialization() {
     }
   });
 
-  // TODO: create the runner with our WorkflowJSON (workflow gem uses datapoint/analysis too?!)
-
   // TODO: Validate the OSW measures if the flag is set to true, (the default state)
-  // Note JM 2022-11-07: Is it better to try and load all measures once, instead of crashing later?
+  // There isn't a 'verify_osw' key in the RunOptions, so always do it for now. Maybe don't if `fast`?
+  {
+    LOG(Info, "Attempting to validate the measure workflow");
+
+    if (!workflowJSON.validateMeasures()) {
+      LOG_AND_THROW("Workflow is invalid");
+    }
+
+    LOG(Info, "Validated the measure workflow");
+  }
 
   LOG(Debug, "Finding and loading the seed file");
   auto seedPath_ = workflowJSON.seedFile();


Test patch:
diff --git a/src/alfalfa/test/AlfalfaJSON_GTest.cpp b/src/alfalfa/test/AlfalfaJSON_GTest.cpp
index eca8a69338a..6a623d7c409 100644
--- a/src/alfalfa/test/AlfalfaJSON_GTest.cpp
+++ b/src/alfalfa/test/AlfalfaJSON_GTest.cpp
@@ -1,5 +1,6 @@
 #include <fmt/core.h>
 #include <gtest/gtest.h>
+#include <stdexcept>
 #include <vector>
 
 #include "../AlfalfaJSON.hpp"
@@ -194,75 +195,46 @@ TEST(AlfalfaJSON, json_serialization) {
   const bool parsing_success = Json::parseFromStream(r_builder, ifs, &root, &formatted_errors);
   EXPECT_TRUE(parsing_success);
   EXPECT_EQ(alfalfa.toJSON(), root);
-  for (const AlfalfaPoint& point : alfalfa.points()) {
-    EXPECT_EQ(root[point.id().get()], point.toJSON());
+  for (Json::ArrayIndex i = 0; const auto& point : alfalfa.points()) {
+    EXPECT_EQ(root[i++], point.toJSON());
   }
 }
 
 TEST(AlfalfaJSON, point_exceptions_logging) {
-  const std::string ID_VALID_CHARS_MSG = "IDs can only contain letters, numbers, and the following special characters _-[]():";
-  const std::string DISPLAY_NAME_VALID_CHARS_MSG =
-    "Display name '{}' does not produce a valid point ID. Manually set a valid ID or export will fail.";
+  const std::string DISPLAY_NAME_EMPTY_ERROR = "Display name must have non-zero length";
+  const std::string ID_EMPTY_ERROR = "Id must have non-zero length";
   const std::string LOG_CHANNEL = "openstudio.AlfalfaPoint";
   StringStreamLogSink ss;
   ss.setLogLevel(Warn);
 
-  const AlfalfaPoint point("Point");
+  AlfalfaPoint point("Point");
   ASSERT_EQ(0, ss.logMessages().size());
   point.id();
   ASSERT_EQ(0, ss.logMessages().size());
 
   //Test logging in constructor
-  AlfalfaPoint invalid_point("Point$$$");
-  ASSERT_EQ(1, ss.logMessages().size());
-  LogMessage invalid_id_msg = ss.logMessages().at(0);
-  EXPECT_EQ(invalid_id_msg.logMessage(), fmt::format(fmt::runtime(DISPLAY_NAME_VALID_CHARS_MSG), "Point$$$"));
-  EXPECT_EQ(invalid_id_msg.logLevel(), Warn);
-  EXPECT_EQ(invalid_id_msg.logChannel(), LOG_CHANNEL);
-  ss.resetStringStream();
-  ASSERT_EQ(ss.logMessages().size(), 0);
-
-  // Test logging when getting id()
-  const boost::optional<std::string> invalid_point_id = invalid_point.id();
-  EXPECT_FALSE(invalid_point_id.is_initialized());
-  ASSERT_EQ(ss.logMessages().size(), 1);
-  invalid_id_msg = ss.logMessages().at(0);
-  EXPECT_EQ(invalid_id_msg.logMessage(), fmt::format(fmt::runtime(DISPLAY_NAME_VALID_CHARS_MSG), "Point$$$"));
-  EXPECT_EQ(invalid_id_msg.logLevel(), Warn);
-  EXPECT_EQ(invalid_id_msg.logChannel(), LOG_CHANNEL);
-  ss.resetStringStream();
-  ASSERT_EQ(ss.logMessages().size(), 0);
-
-  // Test exception handling in setId()
   EXPECT_THROW(
     {
       try {
-        invalid_point.setId("Point_123_$$$");
+        const AlfalfaPoint invalid_point("");
       } catch (const std::runtime_error& error) {
-        EXPECT_EQ(error.what(), ID_VALID_CHARS_MSG);
+        EXPECT_EQ(error.what(), DISPLAY_NAME_EMPTY_ERROR);
         throw;
       }
     },
     std::runtime_error);
-  ASSERT_EQ(ss.logMessages().size(), 0);
 
-  // Test exception handling in toJSON()
+  // Test exception handling in setId()
   EXPECT_THROW(
     {
       try {
-        Json::Value root = invalid_point.toJSON();
+        point.setId("");
       } catch (const std::runtime_error& error) {
-        EXPECT_EQ(error.what(), "Point requires a valid ID for export. " + ID_VALID_CHARS_MSG);
+        EXPECT_EQ(error.what(), ID_EMPTY_ERROR);
         throw;
       }
     },
     std::runtime_error);
-  ASSERT_EQ(ss.logMessages().size(), 1);
-  invalid_id_msg = ss.logMessages().at(0);
-  EXPECT_EQ(invalid_id_msg.logMessage(), fmt::format(fmt::runtime(DISPLAY_NAME_VALID_CHARS_MSG), "Point$$$"));
-  EXPECT_EQ(invalid_id_msg.logLevel(), Warn);
-  EXPECT_EQ(invalid_id_msg.logChannel(), LOG_CHANNEL);
-  ss.resetStringStream();
   ASSERT_EQ(ss.logMessages().size(), 0);
 
   //Test Calls work when provided with legal input
@@ -278,8 +250,7 @@ TEST(AlfalfaJSON, point_exceptions_logging) {
   // Test that changing display name changes the ID if an ID has not been set.
   valid_point.setDisplayName("Another Good Point");
   ASSERT_EQ(ss.logMessages().size(), 0);
-  ASSERT_TRUE(valid_point.id().is_initialized());
-  ASSERT_EQ(valid_point.id().get(), "Another_Good_Point");
+  ASSERT_EQ(valid_point.id(), "Another_Good_Point");
   ASSERT_EQ(ss.logMessages().size(), 0);
 
   Json::Value valid_json = valid_point.toJSON();
@@ -289,8 +260,7 @@ TEST(AlfalfaJSON, point_exceptions_logging) {
   const std::string new_id = "Another_Valid_Point(123)";
   valid_point.setId(new_id);
   ASSERT_EQ(ss.logMessages().size(), 0);
-  ASSERT_TRUE(valid_point.id().is_initialized());
-  ASSERT_EQ(valid_point.id().get(), new_id);
+  ASSERT_EQ(valid_point.id(), new_id);
 
   valid_json = valid_point.toJSON();
   ASSERT_EQ(ss.logMessages().size(), 0);
@@ -298,7 +268,7 @@ TEST(AlfalfaJSON, point_exceptions_logging) {
   // Test that once an ID is set, setting a new display name won't throw a warning.
   valid_point.setDisplayName("Valid Name, but Invalid Id $$$");
   ASSERT_EQ(ss.logMessages().size(), 0);
-  ASSERT_EQ(valid_point.id().get(), new_id);
+  ASSERT_EQ(valid_point.id(), new_id);
 
   valid_json = valid_point.toJSON();
   ASSERT_EQ(ss.logMessages().size(), 0);
diff --git a/src/utilities/filetypes/test/WorkflowJSON_GTest.cpp b/src/utilities/filetypes/test/WorkflowJSON_GTest.cpp
index d88b5b8247a..48e2eb5a070 100644
--- a/src/utilities/filetypes/test/WorkflowJSON_GTest.cpp
+++ b/src/utilities/filetypes/test/WorkflowJSON_GTest.cpp
@@ -15,9 +15,11 @@
 
 #include "../../time/DateTime.hpp"
 
+#include "../../core/Filesystem.hpp"
 #include "../../core/Exception.hpp"
 #include "../../core/System.hpp"
 #include "../../core/Checksum.hpp"
+#include "../../core/StringStreamLogSink.hpp"
 
 #include <resources.hxx>
 
@@ -1436,3 +1438,48 @@ TEST(Filetypes, RunOptions_overrideValuesWith) {
   ASSERT_FALSE(ftOptions.excludeSpaceTranslation());
   ASSERT_TRUE(ftOptions.isExcludeSpaceTranslationDefaulted());
 }
+
+TEST(Filetypes, WorkflowJSON_ValidateMeasures_Ok) {
+  auto p = resourcesPath() / toPath("utilities/Filetypes/full.osw");
+  WorkflowJSON w(p);
+  EXPECT_TRUE(w.validateMeasures());
+}
+
+TEST(Filetypes, WorkflowJSON_ValidateMeasures_Missing) {
+  auto p = resourcesPath() / toPath("workflow/invalid_measures/missing_a_measure.osw");
+  ASSERT_TRUE(boost::filesystem::is_regular_file(p));
+  WorkflowJSON w(p);
+  StringStreamLogSink sink;
+  sink.setLogLevel(Error);
+  EXPECT_FALSE(w.validateMeasures());
+  ASSERT_EQ(1, sink.logMessages().size());
+  EXPECT_EQ("Cannot find measure 'NON_EXISTING_MEASURE_THIS_SHOULD_BE_CAUGHT'", sink.logMessages()[0].logMessage());
+}
+
+TEST(Filetypes, WorkflowJSON_ValidateMeasures_Unloadable) {
+  auto p = resourcesPath() / toPath("workflow/invalid_measures/unloadable_measure.osw");
+  ASSERT_TRUE(boost::filesystem::is_regular_file(p));
+  WorkflowJSON w(p);
+  StringStreamLogSink sink;
+  sink.setLogLevel(Error);
+  EXPECT_FALSE(w.validateMeasures());
+  auto logMessages = sink.logMessages();
+  ASSERT_EQ(3, logMessages.size());
+  EXPECT_EQ("utilities.bcl.BCLXML", logMessages.at(0).logChannel());
+  EXPECT_EQ("utilities.bcl.BCLMeasure", logMessages.at(1).logChannel());
+  EXPECT_EQ("openstudio.WorkflowJSON", logMessages.at(2).logChannel());
+  auto logMessage = sink.logMessages()[2].logMessage();
+  EXPECT_TRUE(logMessage.find("Cannot load measure 'UnloadableMeasure' at '") != std::string::npos) << logMessage;
+}
+
+TEST(Filetypes, WorkflowJSON_ValidateMeasures_WrongOrder) {
+  auto p = resourcesPath() / toPath("workflow/invalid_measures/wrong_measure_type_order.osw");
+  ASSERT_TRUE(boost::filesystem::is_regular_file(p));
+  WorkflowJSON w(p);
+  StringStreamLogSink sink;
+  sink.setLogLevel(Error);
+  EXPECT_FALSE(w.validateMeasures());
+  ASSERT_EQ(1, sink.logMessages().size());
+
+  EXPECT_EQ("OpenStudio measure 'FakeModelMeasure' called after Energyplus simulation.", sink.logMessages()[0].logMessage());
+}