Repository: newrelic/newrelic-ruby-agent (Ruby)

Issue:
Errors that are expected by way of expected_status_codes should not impact Apdex
We have reports that errors with status codes matching specified `expected_status_codes` values are correctly appearing in the Errors Inbox as "expected", but are negatively impacting Apdex.

This relevant specification text confirms that expected errors should not impact Apdex.

> Expected errors MUST create traced errors and error events, but MUST NOT increment any of the Errors/* metrics in order to prevent error rate and Apdex modifications.

We'll need to determine where in the agent or back-end the denotation of an error as expected fails to prevent that error from impacting Apdex.

NOTE: The use of `ignore_status_codes` works as a workaround. Ignored errors are seen as not impacting Apdex. The drawback of the workaround is that ignored errors are not delivered to the Errors Inbox.


Discussion:
https://new-relic.atlassian.net/browse/NR-262162
Thank you so much for taking the time and digging into the problem. :) While the linked PR #2619 would be good enough for what we need, as far as I can see this would still let `expected_classes` and `expected_messages` impact the Apdex score -- right? As the PR has added a check specifically for `expected_status_codes` but not the other "expected" configuration options. See https://github.com/newrelic/newrelic-ruby-agent/blob/dev/newrelic.yml#L309-L322
> Thank you so much for taking the time and digging into the problem. :) While the linked PR #2619 would be good enough for what we need, as far as I can see this would still let `expected_classes` and `expected_messages` impact the Apdex score -- right? As the PR has added a check specifically for `expected_status_codes` but not the other "expected" configuration options. See https://github.com/newrelic/newrelic-ruby-agent/blob/dev/newrelic.yml#L309-L322

Hi @florianpilz! I'm very glad to hear that the linked PR will be helpful!

As for expected error classes and error messages, I believe those are now covered as well. I think the code changes cover everything, but my text summary was insufficient.

Previously, the only way an expected error would not impact Apdex would be if the `notice_error` API was used with an `expected: true` key/value pair argument. With PR #2619, that check for an `expected: true` attribute on the error is still being checked, but we are now additionally performing an `ErrorCollector#expected?` check on the exception and the HTTP response code (which can be `nil`). The exception and optional HTTP response code are sent to `ErrorFilter#expected?` which looks like this:

```ruby
def expected?(ex, status_code = nil)
  @expected_classes.include?(ex.class.name) ||
    (@expected_messages.key?(ex.class.name) &&
    @expected_messages[ex.class.name].any? { |m| ex.message.include?(m) }) ||
    @expected_status_codes.include?(status_code.to_i)
end
```

That method will take care of checking for configured expected classes and messages by looking at the exception argument, and then check for an expected HTTP status code by looking at the optional status code argument.

The inclusion of these `#expected?` methods is the big change the PR introduced.

So I think what happened is that the code in the PR does everything we want, but my description was limited by only mentioning the status codes. If after taking another look you disagree, please let me know and I'd be happy to investigate farther.

We'll keep you posted via comments on this issue for when these changes make it into a new agent gem release.

@florianpilz I have updated the PR description text, the CHANGELOG, and the source code comments. Thanks very much for pointing out that the error classes and error messages are also possible ways to expect errors! See [acb8297](https://github.com/newrelic/newrelic-ruby-agent/pull/2619/commits/acb829766351c8f7f1fa2bb618da2f8a980420bf)
Hi @florianpilz. The PR has now been approved and merged, so the bugfix should go out in the next release of the New Relic Ruby agent. Thank you again for bringing this issue to our attention!
Really nice! You are right, I only got distracted by HTTP codes being explicitly passed. Makes all sense now and looking forward to the next release. 🎉 ❤️ 



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5f2d70a89d..d7835fe946 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,14 @@
 # New Relic Ruby Agent Release Notes
 
+## dev
+
+Version <dev> fixes a bug that would cause an expected error to negatively impact a transaction's Apdex.
+
+- **Bugfix: Expected errors related to HTTP status code, class, and message won't impact Apdex
+
+  The agent is supposed to prevent observed application errors from negatively impacting Apdex if the errors are either ignored or expected. There are two ways for the agent to expect an error: via the `notice_error` API receiving an `expected: true` argument or via matches made against user-configured lists for expected HTTP status codes (`:'error_collector.expected_status_codes'`), expected error classes (`:'error_collector.expected_classes'`), or expected error messages (`:'error_collector.expected_messages'`). Previously, only errors expected via the `notice_error` API were correctly prevented from impacting Apdex. Expected errors set by configuration incorrectly impacted Apdex. This behavior has been fixed and now both types of expected errors will correctly not impact Apdex. Thanks very much to [@florianpilz](https://github.com/florianpilz) for bringing this issue to our attention. [PR#2619](https://github.com/newrelic/newrelic-ruby-agent/pull/2619)
+
+
 ## v9.9.0
 
 Version 9.9.0 introduces support for AWS Lambda serverless function observability, adds support for Elasticsearch 8.13.0, and adds the 'request.temperature' attribute to chat completion summaries in ruby-openai instrumentation.
diff --git a/lib/new_relic/agent/error_collector.rb b/lib/new_relic/agent/error_collector.rb
index 41af8e761b..896ef91d32 100644
--- a/lib/new_relic/agent/error_collector.rb
+++ b/lib/new_relic/agent/error_collector.rb
@@ -110,6 +110,29 @@ def error_is_ignored?(error, status_code = nil)
         false
       end
 
+      # Neither ignored nor expected errors impact apdex.
+      #
+      # Ignored errors are checked via `#error_is_ignored?`
+      # Expected errors are checked in 2 separate ways:
+      #   1. The presence of an `expected: true` attribute key/value pair in the
+      #      options hash, which will be set if that key/value pair was used in
+      #      the `notice_error` public API.
+      #   2. By calling `#expected?` which in turn calls `ErrorFilter#expected?`
+      #      which checks for 3 things:
+      #        - A match for user-defined HTTP status codes to expect
+      #        - A match for user-defined error classes to expect
+      #        - A match for user-defined error messages to expect
+      def error_affects_apdex?(error, options)
+        return false if error_is_ignored?(error)
+        return false if options[:expected]
+
+        !expected?(error, ::NewRelic::Agent::Tracer.state.current_transaction&.http_response_code)
+      rescue => e
+        NewRelic::Agent.logger.error("Could not determine if error '#{error}' should impact Apdex - " \
+                                     "#{e.class}: #{e.message}. Defaulting to 'true' (it should impact Apdex).")
+        true
+      end
+
       # Calling instance_variable_set on a wrapped Java object in JRuby will
       # generate a warning unless that object's class has already been marked
       # as persistent, so we skip tagging of exception objects that are actually
diff --git a/lib/new_relic/agent/transaction.rb b/lib/new_relic/agent/transaction.rb
index 7ca7ec8152..bcbf87d3da 100644
--- a/lib/new_relic/agent/transaction.rb
+++ b/lib/new_relic/agent/transaction.rb
@@ -816,13 +816,9 @@ def record_queue_time
       end
 
       def had_error_affecting_apdex?
-        @exceptions.each do |exception, options|
-          ignored = NewRelic::Agent.instance.error_collector.error_is_ignored?(exception)
-          expected = options[:expected]
-
-          return true unless ignored || expected
+        @exceptions.each.any? do |exception, options|
+          NewRelic::Agent.instance.error_collector.error_affects_apdex?(exception, options)
         end
-        false
       end
 
       def apdex_bucket(duration, current_apdex_t)


Test patch:
diff --git a/test/agent_helper.rb b/test/agent_helper.rb
index 5d12ed5e17..9615253cf9 100644
--- a/test/agent_helper.rb
+++ b/test/agent_helper.rb
@@ -228,7 +228,9 @@ def assert_metrics_recorded(expected)
   expected.each do |specish, expected_attrs|
     expected_spec = metric_spec_from_specish(specish)
     actual_stats = NewRelic::Agent.instance.stats_engine.to_h[expected_spec]
-    if !actual_stats
+    if actual_stats
+      assert(actual_stats)
+    else
       all_specs = NewRelic::Agent.instance.stats_engine.to_h.keys.sort
       matches = all_specs.select { |spec| spec.name == expected_spec.name }
       matches.map! { |m| "  #{m.inspect}" }
diff --git a/test/new_relic/agent/transaction_test.rb b/test/new_relic/agent/transaction_test.rb
index 172ba399c5..e7ff5d4a2c 100644
--- a/test/new_relic/agent/transaction_test.rb
+++ b/test/new_relic/agent/transaction_test.rb
@@ -324,10 +324,13 @@ def test_apdex_success_with_ignored_error
 
       with_ignore_error_filter(filter) do
         txn_name = 'Controller/whatever'
+
+        # this one doesn't impact Apdex
         in_web_transaction(txn_name) do
           Transaction.notice_error(SillyError.new)
         end
 
+        # this one does impact Apdex
         in_web_transaction(txn_name) do
           Transaction.notice_error(RuntimeError.new)
         end
@@ -339,6 +342,49 @@ def test_apdex_success_with_ignored_error
       )
     end
 
+    def test_noticed_errors_that_are_expected_do_not_impact_apdex
+      txn_name = 'Controller/BigDweebEnergy'
+      # this one doesn't impact Apdex
+      in_web_transaction(txn_name) do
+        Transaction.notice_error(RuntimeError.new, expected: true)
+      end
+
+      # this one does impact Apdex
+      in_web_transaction(txn_name) do
+        Transaction.notice_error(RuntimeError.new)
+      end
+
+      assert_metrics_recorded(
+        txn_name => {},
+        'Apdex' => {:apdex_s => 1, :apdex_t => 0, :apdex_f => 1},
+        'Apdex/BigDweebEnergy' => {:apdex_s => 1, :apdex_t => 0, :apdex_f => 1}
+      )
+    end
+
+    def test_expected_status_code_based_errors_do_not_impact_apdex
+      expected_status_code = 418
+      txn_name = 'Controller/BillAmend'
+
+      with_config(:'error_collector.expected_status_codes' => "11,38,#{expected_status_code}") do
+        # this one doesn't impact Apdex
+        in_web_transaction(txn_name) do |txn|
+          txn.http_response_code = expected_status_code
+          Transaction.notice_error(RuntimeError.new)
+        end
+
+        # this one does impact Apdex
+        in_web_transaction(txn_name) do
+          Transaction.notice_error(RuntimeError.new)
+        end
+      end
+
+      assert_metrics_recorded(
+        txn_name => {},
+        'Apdex' => {:apdex_s => 1, :apdex_t => 0, :apdex_f => 1},
+        'Apdex/BillAmend' => {:apdex_s => 1, :apdex_t => 0, :apdex_f => 1}
+      )
+    end
+
     def test_apdex_success_with_config_ignored_error
       txn_name = 'Controller/whatever'
       with_config(:'error_collector.ignore_classes' => [SillyError.name]) do