Repository: newrelic/newrelic-ruby-agent (Ruby)

Issue:
Failed to read or parse configuration file at config/newrelic.yml (Psych 4.0.0+)
`newrelic_rpm` can't parse configuration file with `psych` 4.0.0+

## Description

Psych 4.0.0 makes `YAML.load` default to safe mode (ruby/psych#487).
This cause error on parsing `config/newrelic.yml`.

## Expected Behavior

`newrelic_rpm` can parse configuration file with `psych` 4.0.0+


## [Troubleshooting](https://discuss.newrelic.com/t/troubleshooting-frameworks/108787) or [NR Diag]

```
$ cat log/newrelic_agent.log

# Logfile created on 2021-08-30 11:14:57 +0900 by logger.rb/v1.4.2
[2021-08-30 11:14:57 +0900 *LOCAL* (41089)] INFO : Starting the New Relic agent version 7.2.0 in "development" environment.
[2021-08-30 11:14:57 +0900 *LOCAL* (41089)] INFO : To prevent agent startup add a NEW_RELIC_AGENT_ENABLED=false environment variable or modify the "development" section of your newrelic.yml.
[2021-08-30 11:14:57 +0900 *LOCAL* (41089)] INFO : Reading configuration from config/newrelic.yml (/***/***/fukayatsu/newrelic-with-psych-4)
[2021-08-30 11:14:57 +0900 *LOCAL* (41089)] ERROR : Failed to read or parse configuration file at config/newrelic.yml
[2021-08-30 11:14:57 +0900 *LOCAL* (41089)] ERROR : Psych::BadAlias: Unknown alias: default_settings
[2021-08-30 11:14:57 +0900 *LOCAL* (41089)] INFO : Environment: development
[2021-08-30 11:14:57 +0900 *LOCAL* (41089)] INFO : No known dispatcher detected.
[2021-08-30 11:14:57 +0900 *LOCAL* (41089)] INFO : Application: development
[2021-08-30 11:14:57 +0900 *LOCAL* (41089)] WARN : No license key found. This often means your newrelic.yml file was not found, or it lacks a section for the running environment, 'development'. You may also want to try linting your newrelic.yml to ensure it is valid YML.
[2021-08-30 11:14:58 +0900 *LOCAL* (41089)] INFO : Installing Net:HTTP Wrappers
[2021-08-30 11:14:58 +0900 *LOCAL* (41089)] INFO : Installing New Relic supported NetHTTP instrumentation using Prepend
[2021-08-30 11:14:58 +0900 *LOCAL* (41089)] INFO : Finished instrumentation
```

## Steps to Reproduce

- Clone https://github.com/fukayatsu/newrelic-with-psych-4
- ```
  $ bundle install
  $ bundle exec ruby app.rb
  ```

## Your Environment
Include as many relevant details about your environment as possible including the running version of the Ruby agent, the Ruby version being used and any other relevant environment information

```
$ ruby -v
ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-darwin20]
```

```
$ cat Gemfile.lock
GEM
  remote: https://rubygems.org/
  specs:
    newrelic_rpm (7.2.0)
    psych (4.0.1)

PLATFORMS
  x86_64-darwin-20

DEPENDENCIES
  newrelic_rpm
  psych (~> 4)

BUNDLED WITH
   2.2.22
```

## Additional context

- ruby/psych#487
- [Fix ruby-master test suite (Psych 4.0.0) by casperisfine · Pull Request #42257 · rails/rails](https://github.com/rails/rails/pull/42257)





Discussion:
Additional Notes:

- Current Workaround: Use `gem 'psych', '~> 3'` instead of `gem 'psych'`
- I added `psych` to my Gemfile for https://github.com/ruby/psych/pull/438

Thank you for bringing this issue to our attention, @fukayatsu. 

This is definitely something we will want to address as soon as we can, so we will be including a change to support this in our next release. 

In the meantime, [adding psych 3 to the gemfile](https://github.com/newrelic/newrelic-ruby-agent/issues/759#issuecomment-907968890) is a good workaround for anyone on ruby versions that ship with the new version of psych.
Thank you for fixing the issue quickly!
You're most welcome, @fukayatsu! This is merged into the `dev` branch, a work-in-progress for our upcoming 8.0.0 major release. Feel free to point your Gemfile to that branch if you'd like to incorporate the changes now. Otherwise, the release is planned for the end of September. 



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a6507e9740..1fc265aa08 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,9 @@
 
   ## v8.0.0
 
+  * **Bugfix: Psych 4.0 causes errors when loading newrelic.yml**
+    Psych 4.0 now uses safe load behavior when using `YAML.load` which by default doesn't allow aliases, causing errors when the agent loads the config file. We have updated how we load the config file to avoid these errors. 
+
   * **Deprecate cross application tracing**
     Cross application tracing is deprecated in favor of [distributed tracing](https://docs.newrelic.com/docs/distributed-tracing/enable-configure/language-agents-enable-distributed-tracing/) and is off by default.
 
diff --git a/lib/new_relic/agent/configuration/yaml_source.rb b/lib/new_relic/agent/configuration/yaml_source.rb
index cb79e8603c..5148bb7798 100644
--- a/lib/new_relic/agent/configuration/yaml_source.rb
+++ b/lib/new_relic/agent/configuration/yaml_source.rb
@@ -99,7 +99,12 @@ def process_erb(file)
 
         def process_yaml(file, env, config, path)
           if file
-            confighash = YAML.load(file)
+            confighash = if YAML.respond_to?(:unsafe_load)
+                           YAML.unsafe_load(file)
+                         else 
+                           YAML.load(file)
+                         end
+
             unless confighash.key?(env)
               log_failure("Config file at #{path} doesn't include a '#{env}' section!")
             end
diff --git a/lib/new_relic/agent/instrumentation/sidekiq.rb b/lib/new_relic/agent/instrumentation/sidekiq.rb
index 3788185121..7c8438d586 100644
--- a/lib/new_relic/agent/instrumentation/sidekiq.rb
+++ b/lib/new_relic/agent/instrumentation/sidekiq.rb
@@ -60,7 +60,12 @@ def distributed_tracing_headers
 
     class Sidekiq::Extensions::DelayedClass
       def newrelic_trace_args(msg, queue)
-        (target, method_name, _args) = YAML.load(msg['args'][0])
+        (target, method_name, _args) = if YAML.respond_to?(:unsafe_load)
+                                         YAML.unsafe_load(msg['args'][0])
+                                       else 
+                                         YAML.load(msg['args'][0])
+                                       end
+
         {
           :name => method_name,
           :class_name => target.name,


Test patch:
diff --git a/test/multiverse/suites/active_record/config/database.rb b/test/multiverse/suites/active_record/config/database.rb
index dbabbbb794..eaaa68e0ce 100644
--- a/test/multiverse/suites/active_record/config/database.rb
+++ b/test/multiverse/suites/active_record/config/database.rb
@@ -19,7 +19,11 @@
 
 config_raw = File.read(File.join(config_dir, 'database.yml'))
 config_erb = ERB.new(config_raw).result
-config_yml = YAML.load(config_erb)
+config_yml = if YAML.respond_to?(:unsafe_load)
+               YAML.unsafe_load(config_erb)
+             else 
+               YAML.load(config_erb)
+             end
 
 # Rails 2.x didn't keep the Rails out of ActiveRecord much...
 RAILS_ENV  = "test"
diff --git a/test/multiverse/suites/config_file_loading/Envfile b/test/multiverse/suites/config_file_loading/Envfile
index dede848f4a..fff09b3c6f 100644
--- a/test/multiverse/suites/config_file_loading/Envfile
+++ b/test/multiverse/suites/config_file_loading/Envfile
@@ -8,6 +8,26 @@ gemfile <<-RB
   # Because we delay the agent, order of jruby-openssl matters
   gem 'jruby-openssl', '~> 0.9.10', :platforms => [:jruby]
 
+  if RUBY_VERSION >= '2.4.0'
+    gem 'psych', '4.0.0'
+  end
+
+  # don't start the agent
+  gem 'newrelic_rpm', :require => false, :path => File.expand_path('../../../../')
+RB
+
+gemfile <<-RB
+  # stub file system so we can test that newrelic.yml can be loaded from
+  # various places.
+  gem 'fakefs', '0.5.4', :require => 'fakefs/safe' # 0.5.4 required for 1.8.7 compatibility.
+
+  # Because we delay the agent, order of jruby-openssl matters
+  gem 'jruby-openssl', '~> 0.9.10', :platforms => [:jruby]
+
+  if RUBY_VERSION >= '2.4.0'
+    gem 'psych', '3.3.0'
+  end
+
   # don't start the agent
   gem 'newrelic_rpm', :require => false, :path => File.expand_path('../../../../')
 RB
diff --git a/test/multiverse/suites/config_file_loading/config_file_loading_test.rb b/test/multiverse/suites/config_file_loading/config_file_loading_test.rb
index 525a59f26b..babfe5c3fa 100644
--- a/test/multiverse/suites/config_file_loading/config_file_loading_test.rb
+++ b/test/multiverse/suites/config_file_loading/config_file_loading_test.rb
@@ -199,6 +199,34 @@ def test_config_will_load_settings_for_environment_passed_manual_start
     assert_equal 'bazbangbarn', NewRelic::Agent.config[:i_am], "Agent.config did not load bazbangbarn config as requested"
   end
 
+  def test_parses_default_settings_correctly
+    config_contents = <<-YAML
+common: &default_settings
+  license_key: 47ae68e8919929c1d99fa3a1a36a0aa8c06ba690
+  app_name: playground
+  foo: success!!
+
+development:
+  <<: *default_settings
+  app_name: playground (Development)
+
+test:
+  <<: *default_settings
+  monitor_mode: false
+
+bazbangbarn:
+  <<: *default_settings
+  app_name: playground (Bazbangbarn)
+    YAML
+
+    path = File.join(@cwd, 'config', 'newrelic.yml')
+    setup_config(path, {}, config_contents)
+    setup_agent
+
+    log = with_array_logger { NewRelic::Agent.manual_start }
+    assert_equal 'success!!', NewRelic::Agent.config[:foo]
+  end
+
   def refute_log_contains(log, message)
     lines = log.array
     failure_message = "Found unexpected '#{message}' in log. Log contained:\n#{lines.join('')}"