Repository: newrelic/newrelic-ruby-agent (Ruby)

Issue:
Update usage of MD5 in the agent to a FIPS compliant alternative
Problem:
"The NewRelic ruby agent utilizes MD5 digests in a few places and is causing our app to crash when the OS is in FIPS mode. md5_dgst.c(82): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!"

The agent uses MD5 in a couple of places:
https://github.com/newrelic/newrelic-ruby-agent/blob/dev/lib/new_relic/agent/sql_sampler.rb#L362
https://github.com/newrelic/newrelic-ruby-agent/blob/dev/lib/new_relic/agent/sql_sampler.rb#L365
https://github.com/newrelic/newrelic-ruby-agent/blob/dev/lib/new_relic/agent/monitors/cross_app_monitor.rb#L105

MD5 is not FIPS compliant, causing applications on FIPS servers to crash upon encountering this error. 

`md5_dgst.c(82): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!`

Other agent teams, such as the Node agent, have updated this to use SHA1. 

[Internal JIRA](https://newrelic.atlassian.net/browse/GTSE-7735)


Discussion:
Migrating suggestion from another source:

Thanks for the update! In the meantime, we've patched the NewRelic agent methods to use Digest::SHA256, we haven't seen any issues so far but wanted to check if there was anything we should be looking out for.”

```
if defined?(NewRelic)
  require 'digest'

  class NewRelic::Agent::DistributedTracing::CrossAppMonitor
    private

    def hash_transaction_name(identifier)
      Digest::SHA256.digest(identifier).unpack1('@12N') & 0xffffffff
    end
  end

  class NewRelic::Agent::SqlTrace
    private

    def consistent_hash(string)
      if NewRelic::Agent.config[:'slow_sql.use_longer_sql_id']
        Digest::SHA256.hexdigest(string).hex.modulo(2**63 - 1)
      else
        # from when sql_id needed to fit in an INT(11)
        Digest::SHA256.hexdigest(string).hex.modulo(2**31 - 1)
      end
    end
  end
end 
```




Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 71d7bb4a92..7311075710 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,14 +2,20 @@
 
   ## v7.1.0
 
+  * **Removed MD5 use in the SQL sampler**
+
+    In order to allow the agent to run in FIPS compliant environments, the usage of MD5 for aggregating slow sql traces has been replaced with SHA1. 
+
   * **Enable server-side configuration of distributed tracing**
 
     `distributed_tracing.enabled` may now be set in server-side application configuration.
 
   * **Bugfix: Fix for missing part of a previous bugfix**
+
     Our previous fix of "nil Middlewares injection now prevented and gracefully handled in Sinatra" released in 7.0.0 was partially overwritten by some of the other changes in that release. This release adds back those missing sections of the bugfix, and should resolve the issue for sinatra users. 
 
   * **Update known conflicts with use of Module#Prepend**
+
     With our release of v7.0.0, we updated our instrumentation to use Module#Prepend by default, instead of method chaining. We have received reports of conflicts and added a check for these known conflicts. If a known conflict with prepend is detected while using the default value of 'auto' for gem instrumentation, the agent will instead install method chaining instrumentation in order to avoid this conflict. This check can be bypassed by setting the instrumentation method for the gem to 'prepend'.
 
   * **Bugfix: Updated support for ActiveRecord 6.1+ instrumentation**
diff --git a/lib/new_relic/agent/sql_sampler.rb b/lib/new_relic/agent/sql_sampler.rb
index 698d32a70c..f5d0482af4 100644
--- a/lib/new_relic/agent/sql_sampler.rb
+++ b/lib/new_relic/agent/sql_sampler.rb
@@ -359,10 +359,10 @@ def to_collector_array(encoder)
       # need to hash the same way in every process, to be able to aggregate slow SQL traces
       def consistent_hash(string)
         if NewRelic::Agent.config[:'slow_sql.use_longer_sql_id']
-          Digest::MD5.hexdigest(string).hex.modulo(2**63-1)
+          Digest::SHA1.hexdigest(string).hex.modulo(2**63-1)
         else
           # from when sql_id needed to fit in an INT(11)
-          Digest::MD5.hexdigest(string).hex.modulo(2**31-1)
+          Digest::SHA1.hexdigest(string).hex.modulo(2**31-1)
         end
       end
     end


Test patch:
diff --git a/test/new_relic/agent/sql_sampler_test.rb b/test/new_relic/agent/sql_sampler_test.rb
index 86efe3d28d..de0f75879d 100644
--- a/test/new_relic/agent/sql_sampler_test.rb
+++ b/test/new_relic/agent/sql_sampler_test.rb
@@ -344,7 +344,7 @@ def test_to_collector_array
       marshaller = NewRelic::Agent::NewRelicService::JsonMarshaller.new
       params = "eJyrrgUAAXUA+Q==\n"
 
-      expected = [ 'WebTransaction/Controller/c/a', '/c/a', 526336943,
+      expected = [ 'WebTransaction/Controller/c/a', '/c/a', 1644443586,
                    'select * from test', 'Database/test/select',
                    1, 1500, 1500, 1500, params ]