Repository: NREL/OpenStudio (language: python)

Issue:
CLI issue when using bundle and it has a native gem listed as a dependency
Issue overview
--------------

This is **exactly** the same issue as https://github.com/NREL/OpenStudio/issues/5091 which was specifically for `racc`.

This popped up again yesterday when @wenyikuang and @DavidGoldwasser tried to upgrade rubocop to > 1.31.

Rubocop 1.31 added `json, ~> 2.3` as a dependency, and that is a native extension. We do have a json 2.6.3 in the CLI, but it would still fails because

```
Source locally installed gems is ignoring #<Bundler::StubSpecification name=json version=2.7.2 platform=ruby> because it is missing extensions
terminate called after throwing an instance of 'RubyException'
  what():  Bundler::GemNotFound: Could not find json-2.7.2 in locally installed gems
```



## Current Behavior
<!--- Tell us what happens instead of the expected behavior -->

## Expected Behavior
<!--- Tell us what should happen -->

## Steps to Reproduce

checkout https://github.com/NREL/openstudio-ee-gem/pull/52/commits/e0f9fdc4877cd8868164a710fdffb61ab2f045a4

using system ruby 3.2.2 with bundler 2.4.10

```
bundle _2.4.10_ install --path=$(pwd)/.bundle
```

Example command would be `bundle exec rake openstudio:test_with_openstudio`.

Or directly and more succinctly

```
$os_build_rel2/Products/openstudio --loglevel Trace --bundle '/home/julien/Software/Others/openstudio_gems/openstudio-ee-gem/Gemfile' --bundle_path '/home/julien/Software/Others/openstudio_gems/openstudio-ee-gem/.bundle' -e 'puts "hello"'
```

## Possible Solution

We should just use the embedded native gems when they are **both**:

* Preset
* Compatible with the requirements

eg rubocop requires json ~> 2.3

`bundle _2.4.10_ install --path=$(pwd)/.bundle` would resolve it as json-2.7.2.

But our CLI embeds json 2.6.3, which is compatible.

So we should just use the CLI one.

@kbenne FYI.

## Details

### Environment

Some additional details about your environment for this issue (if relevant):
 - Platform (Operating system, version): All
 - Version of OpenStudio (if using an intermediate build, include SHA): 3.8.0-rc1

### Context

CLI issue when using bundle and it has a native gem listed as a dependency
Issue overview
--------------

This is **exactly** the same issue as https://github.com/NREL/OpenStudio/issues/5091 which was specifically for `racc`.

This popped up again yesterday when @wenyikuang and @DavidGoldwasser tried to upgrade rubocop to > 1.31.

Rubocop 1.31 added `json, ~> 2.3` as a dependency, and that is a native extension. We do have a json 2.6.3 in the CLI, but it would still fails because

```
Source locally installed gems is ignoring #<Bundler::StubSpecification name=json version=2.7.2 platform=ruby> because it is missing extensions
terminate called after throwing an instance of 'RubyException'
  what():  Bundler::GemNotFound: Could not find json-2.7.2 in locally installed gems
```



## Current Behavior
<!--- Tell us what happens instead of the expected behavior -->

## Expected Behavior
<!--- Tell us what should happen -->

## Steps to Reproduce

checkout https://github.com/NREL/openstudio-ee-gem/pull/52/commits/e0f9fdc4877cd8868164a710fdffb61ab2f045a4

using system ruby 3.2.2 with bundler 2.4.10

```
bundle _2.4.10_ install --path=$(pwd)/.bundle
```

Example command would be `bundle exec rake openstudio:test_with_openstudio`.

Or directly and more succinctly

```
$os_build_rel2/Products/openstudio --loglevel Trace --bundle '/home/julien/Software/Others/openstudio_gems/openstudio-ee-gem/Gemfile' --bundle_path '/home/julien/Software/Others/openstudio_gems/openstudio-ee-gem/.bundle' -e 'puts "hello"'
```

## Possible Solution

We should just use the embedded native gems when they are **both**:

* Preset
* Compatible with the requirements

eg rubocop requires json ~> 2.3

`bundle _2.4.10_ install --path=$(pwd)/.bundle` would resolve it as json-2.7.2.

But our CLI embeds json 2.6.3, which is compatible.

So we should just use the CLI one.

@kbenne FYI.

## Details

### Environment

Some additional details about your environment for this issue (if relevant):
 - Platform (Operating system, version): All
 - Version of OpenStudio (if using an intermediate build, include SHA): 3.8.0-rc1

### Context

CLI issue when using bundle and it has a native gem listed as a dependency
Issue overview
--------------

This is **exactly** the same issue as https://github.com/NREL/OpenStudio/issues/5091 which was specifically for `racc`.

This popped up again yesterday when @wenyikuang and @DavidGoldwasser tried to upgrade rubocop to > 1.31.

Rubocop 1.31 added `json, ~> 2.3` as a dependency, and that is a native extension. We do have a json 2.6.3 in the CLI, but it would still fails because

```
Source locally installed gems is ignoring #<Bundler::StubSpecification name=json version=2.7.2 platform=ruby> because it is missing extensions
terminate called after throwing an instance of 'RubyException'
  what():  Bundler::GemNotFound: Could not find json-2.7.2 in locally installed gems
```



## Current Behavior
<!--- Tell us what happens instead of the expected behavior -->

## Expected Behavior
<!--- Tell us what should happen -->

## Steps to Reproduce

checkout https://github.com/NREL/openstudio-ee-gem/pull/52/commits/e0f9fdc4877cd8868164a710fdffb61ab2f045a4

using system ruby 3.2.2 with bundler 2.4.10

```
bundle _2.4.10_ install --path=$(pwd)/.bundle
```

Example command would be `bundle exec rake openstudio:test_with_openstudio`.

Or directly and more succinctly

```
$os_build_rel2/Products/openstudio --loglevel Trace --bundle '/home/julien/Software/Others/openstudio_gems/openstudio-ee-gem/Gemfile' --bundle_path '/home/julien/Software/Others/openstudio_gems/openstudio-ee-gem/.bundle' -e 'puts "hello"'
```

## Possible Solution

We should just use the embedded native gems when they are **both**:

* Preset
* Compatible with the requirements

eg rubocop requires json ~> 2.3

`bundle _2.4.10_ install --path=$(pwd)/.bundle` would resolve it as json-2.7.2.

But our CLI embeds json 2.6.3, which is compatible.

So we should just use the CLI one.

@kbenne FYI.

## Details

### Environment

Some additional details about your environment for this issue (if relevant):
 - Platform (Operating system, version): All
 - Version of OpenStudio (if using an intermediate build, include SHA): 3.8.0-rc1

### Context



Discussion / hints:

Fix patch:
diff --git a/ruby/bindings/InitRubyBindings.cpp b/ruby/bindings/InitRubyBindings.cpp
index dcfd9a4923d..f67a8af8e1e 100644
--- a/ruby/bindings/InitRubyBindings.cpp
+++ b/ruby/bindings/InitRubyBindings.cpp
@@ -489,6 +489,10 @@ class ConvenienceLogger
     @level <= OpenStudio::Trace
   end
 
+  def trace(msg)
+    OpenStudio::logFree(OpenStudio::Trace, "ruby", msg)
+  end
+
   def debug(msg)
     OpenStudio::logFree(OpenStudio::Debug, "ruby", msg)
   end
@@ -838,9 +842,9 @@ void locateEmbeddedGems(bool use_bundler) {
   embedded_gems_to_activate.each do |spec|
     if spec.name == "bundler"
       $logger.debug "Activating Bundler gem #{spec.spec_file}"
-      if $logger.trace?
-        pp spec
-      end
+      # if $logger.trace?
+      #   pp spec
+      # end
 
       begin
         # Activate will manipulate the $LOAD_PATH to include the gem
@@ -874,6 +878,27 @@ void locateEmbeddedGems(bool use_bundler) {
     require 'bundler/plugin'
     #require 'bundler/rubygems_ext'
     require 'bundler/rubygems_integration'
+
+    # Global list, to be populated below
+    $ignore_native_gem_names = []
+
+    module Bundler
+      class RubygemsIntegration
+
+        alias :ori_spec_missing_extensions? :spec_missing_extensions?
+
+        def spec_missing_extensions?(spec, default = true)
+
+          # This avoids getting an annoying message for no reason
+          if $ignore_native_gem_names.any? {|name| name == spec.name }
+            return false
+          end
+
+          return ori_spec_missing_extensions?(spec, default)
+        end
+      end
+    end
+
     require 'bundler/version'
     require 'bundler/ruby_version'
     #require 'bundler/constants'
@@ -920,6 +945,37 @@ void locateEmbeddedGems(bool use_bundler) {
 
     $logger.info "Bundling with groups [#{keep_groups.join(',')}]"
 
+    # Filter out dependencies with native extensions that we already have embedded in the CLI and that satisfy the requirements
+    locked_specs = Bundler.definition.instance_variable_get(:@locked_specs)
+
+    embedded_gems_to_activate.each do |spec|
+      next if spec.extensions.empty?
+
+      next unless locked_specs.any? {|locked_spec| locked_spec.name == spec.name}
+
+      dep_to_requirements = {}
+      locked_specs.each do |locked_spec|
+        deleted_deps = locked_spec.dependencies.select { |dep| dep.name == spec.name && dep.requirement.satisfied_by?(spec.version) }
+        locked_spec.dependencies.reject! { |dep| dep.name == spec.name && dep.requirement.satisfied_by?(spec.version) }
+        deleted_deps&.each do |dep|
+          dep_to_requirements[locked_spec.name] = dep.requirement.to_s
+        end
+      end
+
+
+      if locked_specs.none? { |locked_spec| locked_spec.dependencies.any? {|x| x.name == spec.name} }
+        locked_spec_v = locked_specs.select{|locked_spec| locked_spec.name == spec.name}.first.version
+        $logger.debug("Removing native gem #{spec.name} from Bundler locked_specs (version #{locked_spec_v}), using the embedded dependency (CLI one has version #{spec.version})")
+        $logger.trace("Gems having #{spec.name} as a dependency: #{dep_to_requirements}")
+        raise "Failed to activate #{spec.name}" unless spec.activate
+        $ignore_native_gem_names << spec.name
+        locked_specs.delete_by_name(spec.name)
+      end
+    end
+
+    # I am pretty sure sure we don't need this. Because internally Bundler.definition.filter_specs will ignore it from the dependencies if it's there as well
+    # Bundler.definition.instance_variable_set(:@dependencies,  Bundler.definition.dependencies.select { |dep| $ignore_native_gem_names.include?(dep.name) } )
+
     remaining_specs = []
     Bundler.definition.specs_for(keep_groups).each {|s| remaining_specs << s.name}
 
@@ -934,8 +990,11 @@ void locateEmbeddedGems(bool use_bundler) {
       RbConfig::CONFIG['arch'] = original_arch
     end
 
-      Dir.chdir(current_dir)
-    end
+    Dir.chdir(current_dir)
+
+    # clean up the global
+    $ignore_native_gem_names = nil
+  end
   )ruby";
   }
 


Test patch:
diff --git a/src/cli/test/bundle_native_embedded/Gemfile b/src/cli/test/bundle_native_embedded/Gemfile
new file mode 100644
index 00000000000..255f4209400
--- /dev/null
+++ b/src/cli/test/bundle_native_embedded/Gemfile
@@ -0,0 +1,7 @@
+source 'http://rubygems.org'
+
+# Rubocop requires 'json' '~> 2.3'
+# It also indirectly, via parser, require 'racc'
+gem 'rubocop', '1.63.4'
+# Our CLI has json 2.6.3, which satisfies this version
+# racc is also in our CLI
diff --git a/src/cli/test/bundle_native_embedded/test.rb b/src/cli/test/bundle_native_embedded/test.rb
new file mode 100644
index 00000000000..2d181f4b8c6
--- /dev/null
+++ b/src/cli/test/bundle_native_embedded/test.rb
@@ -0,0 +1,14 @@
+require 'rubygems'
+
+def local_gems
+   Gem::Specification.sort_by{ |g| [g.name.downcase, g.version] }.group_by{ |g| g.name }
+end
+
+# list installed gems
+puts local_gems.map{ |name, specs|
+  [name, specs.map{ |spec| spec.version.to_s }.join(',')].join(' ')
+}
+# test a gem that required native extensions
+require 'json'
+puts JSON::VERSION
+raise "json version does not match" unless JSON::VERSION == '2.6.3'
diff --git a/src/cli/test/test_bundle.rb b/src/cli/test/test_bundle.rb
index be2c8843e10..31c1d7f2eec 100644
--- a/src/cli/test/test_bundle.rb
+++ b/src/cli/test/test_bundle.rb
@@ -52,7 +52,7 @@ def test_bundle_native
     if /mingw/.match(RUBY_PLATFORM) || /mswin/.match(RUBY_PLATFORM)
       skip("Native gems not supported on Windows")
     else
-	  skip("Native gems not supported on Unix or Mac")
+      skip("Native gems not supported on Unix or Mac")
     end
 
     rm_if_exist('Gemfile.lock')
@@ -70,6 +70,25 @@ def test_bundle_native
     Dir.chdir(original_dir)
   end
 
+
+  # Test for #5181 - This adds a bundle dependency on a native gem we DO have it the CLI
+  def test_bundle_native_embedded
+    Dir.chdir(File.join(File.dirname(__FILE__), 'bundle_native_embedded')) do
+
+      rm_if_exist('Gemfile.lock')
+      rm_if_exist('./test_gems')
+      rm_if_exist('./bundle')
+
+      assert(system('bundle install --path ./test_gems'))
+      #assert(system('bundle lock --add_platform ruby'))
+      if /mingw/.match(RUBY_PLATFORM) || /mswin/.match(RUBY_PLATFORM)
+        assert(system('bundle lock --add_platform mswin64'))
+      end
+      assert(system("'#{OpenStudio::getOpenStudioCLI}' --bundle Gemfile --bundle_path './test_gems' --verbose test.rb"))
+
+    end
+  end
+
   def test_bundle_no_install
     original_dir = Dir.pwd
     Dir.chdir(File.join(File.dirname(__FILE__), 'bundle_no_install'))