Repository: phac-nml/irida-next
Pull request #1026: Opentelemetry
Author: login: JeffreyThiessen
Created: 2025-04-10T18:35:26Z | Merged: 2025-04-29T13:22:22Z

## What does this PR do and why?
_Describe in detail what your merge request does and why._

Adds metrics telemetry via opentelemetry.
Tracks number of jobs queued per queue, and lowest job latency.
Also adds Traces to activejob so errors can easily be diagnosed in azure diagnostics

Configuration
* `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` to enable job queue metrics
* `OTEL_METRICS_SEND_INTERVAL` to configure seconds between sending updated metrics (default 10)
* `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT ` to enable tracing from activejob
 
## Screenshots or screen recordings
_Screenshots are required for UI changes, and strongly recommended for all other pull requests._

## How to set up and validate locally
_Numbered steps to set up and validate the change are strongly suggested._

Run open telemetry in a docker container, passing in the following yaml config
```yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:
    verbosity: detailed

processors:
  batch:

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]
```

```bash
docker run --rm -v $(pwd)/config.yaml:/etc/otel/config.yaml -p 4317:4317 -p 4318:4318 otel/opentelemetry-collector-contrib:0.123.0 --config /etc/otel/config.yaml 2>&1
```

Add the following env var to enable and direct the telemetry output to docker

Via bash
```bash
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="http://localhost:4318/v1/metrics"
```

Via launch.json
```json
            "env": {
                "RAILS_ENABLE_WEB_CONSOLE": true,
                "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT": "http://localhost:4318/v1/metrics"
            }
```

A thread should be started alongside irida next which periodically sends data to the docker container, it's spans should be visible in the docker console output

## PR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

- [x] I have evaluated the [PR acceptance checklist](https://phac-nml.github.io/irida-next/docs/development/development_processes/code_review#acceptance-checklist) for this PR.



Diff:
diff --git a/Gemfile b/Gemfile
index 23e6216f44..d590fd1d33 100644
--- a/Gemfile
+++ b/Gemfile
@@ -136,6 +136,16 @@ gem 'caxlsx'
 # renders client's local time zone
 gem 'local_time', '~> 3.0', '>= 3.0.2'
 
+# OpenTelemetry
+gem 'opentelemetry-api'
+gem 'opentelemetry-common'
+gem 'opentelemetry-exporter-otlp'
+gem 'opentelemetry-exporter-otlp-metrics'
+gem 'opentelemetry-instrumentation-active_job'
+gem 'opentelemetry-metrics-api'
+gem 'opentelemetry-metrics-sdk'
+gem 'opentelemetry-sdk'
+
 # feature flags
 flipper_version = '~> 1.3.2'
 gem 'flipper-active_record', flipper_version
diff --git a/Gemfile.lock b/Gemfile.lock
index cb82694f5f..8875d965c9 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -273,6 +273,14 @@ GEM
       googleauth (~> 1.9)
       mini_mime (~> 1.0)
     google-logging-utils (0.1.0)
+    google-protobuf (4.30.2-arm64-darwin)
+      bigdecimal
+      rake (>= 13)
+    google-protobuf (4.30.2-x86_64-linux)
+      bigdecimal
+      rake (>= 13)
+    googleapis-common-protos-types (1.19.0)
+      google-protobuf (>= 3.18, < 5.a)
     googleauth (1.13.1)
       faraday (>= 1.0, < 3.a)
       google-cloud-env (~> 2.2)
@@ -414,6 +422,47 @@ GEM
     omniauth-saml (2.2.1)
       omniauth (~> 2.1)
       ruby-saml (~> 1.17)
+    opentelemetry-api (1.5.0)
+    opentelemetry-common (0.22.0)
+      opentelemetry-api (~> 1.0)
+    opentelemetry-exporter-otlp (0.30.0)
+      google-protobuf (>= 3.18)
+      googleapis-common-protos-types (~> 1.3)
+      opentelemetry-api (~> 1.1)
+      opentelemetry-common (~> 0.20)
+      opentelemetry-sdk (~> 1.2)
+      opentelemetry-semantic_conventions
+    opentelemetry-exporter-otlp-metrics (0.4.0)
+      google-protobuf (>= 3.18, < 5.0)
+      googleapis-common-protos-types (~> 1.3)
+      opentelemetry-api (~> 1.1)
+      opentelemetry-common (~> 0.20)
+      opentelemetry-metrics-api (~> 0.2)
+      opentelemetry-metrics-sdk (~> 0.5)
+      opentelemetry-sdk (~> 1.2)
+      opentelemetry-semantic_conventions
+    opentelemetry-instrumentation-active_job (0.8.0)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-instrumentation-base (~> 0.23.0)
+    opentelemetry-instrumentation-base (0.23.0)
+      opentelemetry-api (~> 1.0)
+      opentelemetry-common (~> 0.21)
+      opentelemetry-registry (~> 0.1)
+    opentelemetry-metrics-api (0.3.0)
+      opentelemetry-api (~> 1.0)
+    opentelemetry-metrics-sdk (0.6.0)
+      opentelemetry-api (~> 1.1)
+      opentelemetry-metrics-api (~> 0.2)
+      opentelemetry-sdk (~> 1.2)
+    opentelemetry-registry (0.4.0)
+      opentelemetry-api (~> 1.1)
+    opentelemetry-sdk (1.8.0)
+      opentelemetry-api (~> 1.1)
+      opentelemetry-common (~> 0.20)
+      opentelemetry-registry (~> 0.2)
+      opentelemetry-semantic_conventions
+    opentelemetry-semantic_conventions (1.11.0)
+      opentelemetry-api (~> 1.0)
     orm_adapter (0.5.0)
     os (1.1.4)
     pagy (9.3.3)
@@ -692,6 +741,14 @@ DEPENDENCIES
   omniauth-entra-id
   omniauth-rails_csrf_protection
   omniauth-saml
+  opentelemetry-api
+  opentelemetry-common
+  opentelemetry-exporter-otlp
+  opentelemetry-exporter-otlp-metrics
+  opentelemetry-instrumentation-active_job
+  opentelemetry-metrics-api
+  opentelemetry-metrics-sdk
+  opentelemetry-sdk
   pagy (~> 9.3.3)
   paranoia
   pathogen_view_components!
diff --git a/config/database.yml b/config/database.yml
index 66a06b525e..5b15206280 100644
--- a/config/database.yml
+++ b/config/database.yml
@@ -19,7 +19,7 @@ default: &default
   # https://guides.rubyonrails.org/configuring.html#database-pooling
   #
   # For GoodJob container, pool size is GOOD_JOB_MAX_THREADS (5) + 3 (1 for LISTEN/NOTIFY, 2 for CRON)
-  pool: <%= $PROGRAM_NAME.include?("good_job") ? ENV.fetch("GOOD_JOB_MAX_THREADS", 5).to_i + 3 : (ENV.fetch("RAILS_MAX_THREADS", 5).to_i) %>
+  pool: <%= $PROGRAM_NAME.include?("good_job") ? ENV.fetch("GOOD_JOB_MAX_THREADS", 5).to_i + 3 : (ENV.fetch("RAILS_MAX_THREADS", 5).to_i + 1) %>
   <% if ENV["DB_HOST"] %>
   host: <%= ENV["DB_HOST"] %>
   username: postgres
diff --git a/config/initializers/opentelemetry.rb b/config/initializers/opentelemetry.rb
new file mode 100644
index 0000000000..2ef1308a39
--- /dev/null
+++ b/config/initializers/opentelemetry.rb
@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+require 'opentelemetry/sdk'
+require 'opentelemetry-exporter-otlp'
+
+require 'opentelemetry-metrics-sdk'
+require 'opentelemetry-exporter-otlp-metrics'
+
+require 'irida/metrics_reporter'
+
+# Do not continue if OTEL EXPORTER endpoints are not set
+return unless ENV['OTEL_EXPORTER_OTLP_METRICS_ENDPOINT'] || ENV['OTEL_EXPORTER_OTLP_TRACES_ENDPOINT']
+
+# Before configuring OTL exporter, disable default metric exporter if we defined the endpoint.
+ENV['OTEL_METRICS_EXPORTER'] = 'none' if ENV['OTEL_EXPORTER_OTLP_METRICS_ENDPOINT']
+
+# Configure the SDK depending on which endpoints are active
+if ENV['OTEL_EXPORTER_OTLP_TRACES_ENDPOINT']
+  OpenTelemetry::SDK.configure do |c|
+    c.use 'OpenTelemetry::Instrumentation::ActiveJob'
+  end
+else
+  OpenTelemetry::SDK.configure
+end
+
+# Configure our custom otel metrics reporter
+if ENV['OTEL_EXPORTER_OTLP_METRICS_ENDPOINT']
+  otlp_metric_exporter = OpenTelemetry::Exporter::OTLP::Metrics::MetricsExporter.new
+  OpenTelemetry.meter_provider.add_metric_reader(otlp_metric_exporter)
+
+  Rails.application.config.after_initialize do
+    # start the metrics reporting thread
+    send_interval = ENV['OTEL_METRICS_SEND_INTERVAL']&.to_i || 10
+    Irida::MetricsReporter.instance.run(send_interval)
+  end
+
+  at_exit do
+    Irida::MetricsReporter.instance.stop
+  end
+end
diff --git a/docs-site/docs/configuration/options.md b/docs-site/docs/configuration/options.md
index 114153d3fb..bac284c318 100644
--- a/docs-site/docs/configuration/options.md
+++ b/docs-site/docs/configuration/options.md
@@ -35,3 +35,6 @@ Primary ENV Variable which sets what type of environment to run
 | `ENABLE_CRON`                | Enables built in cron cleanup jobs for samples, attachments, and data exports.                           | `true`                     |
 | `CRON_CLEANUP_AFTER_DAYS`    | Set the number of days old a deleted sample/attachment must be before it is cleaned.                     | `7`                        |
 | `RAILS_ENABLE_WEB_CONSOLE`    | Development mode only: When set, a rails console will be present on every webpage.                      | `nil`                        |
+| `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`    | When set, custom metrics telemetry will be sent to the metrics endpoint specified. (example: "localhost:4318/v1/metrics").                      | `nil`                        |
+| `OTEL_METRICS_SEND_INTERVAL`    | Number of seconds to sleep between telemetry batch send calls.                      | `10`                        |
+| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`    | When set, configured trace telemetry will be sent to the traces endpoint specified. (example: "localhost:4318/v1/traces").                      | `nil`                        |
diff --git a/lib/irida/job_queue_metrics.rb b/lib/irida/job_queue_metrics.rb
new file mode 100644
index 0000000000..53d81ed01f
--- /dev/null
+++ b/lib/irida/job_queue_metrics.rb
@@ -0,0 +1,126 @@
+# frozen_string_literal: true
+
+require 'singleton'
+
+module Irida
+  # Functions for creating instruments and updating values on metrics meter
+  class JobQueueMetrics
+    include Singleton
+
+    def update_minimum_queue_times
+      now = Time.now.utc
+      time_by_queue = GoodJob::Job
+                      .queued
+                      .group(:queue_name)
+                      .pluck(:queue_name, Arel.sql('min(coalesce(scheduled_at, created_at))'))
+                      .to_h
+
+      latency_hash = {}
+      time_by_queue.each do |queue_name, queue_time|
+        latency = queue_time ? (now - queue_time).ceil : 0 # seconds
+        latency_hash[queue_name] = latency
+      end
+
+      latency_to_send = build_data_to_send(latency_hash, job_queue_latency_previous_value_map)
+
+      latency_to_send.each do |queue_name, latency|
+        metric_update_queue_min_latency(queue_name, latency)
+      end
+    end
+
+    def update_queue_counts
+      queue_counts = GoodJob::Job
+                     .queued
+                     .group(:queue_name)
+                     .count
+
+      queue_counts_to_send = build_data_to_send(queue_counts, job_queue_count_previous_value_map)
+
+      queue_counts_to_send.each do |queue_name, count|
+        metric_update_job_queue_count(queue_name, count)
+      end
+    end
+
+    private
+
+    def meter
+      @meter ||= OpenTelemetry.meter_provider.meter('JOB_QUEUE_METER')
+    end
+
+    def job_queue_count_previous_value_map
+      @job_queue_count_previous_value_map ||= {}
+    end
+
+    def job_queue_latency_previous_value_map
+      @job_queue_latency_previous_value_map ||= {}
+    end
+
+    # This method reduces number of call to the instruments by
+    #  only reporting when the value is different than what was sent previously
+    # It also includes a 0 when a value was previously passed but is not reported by GoodJob
+    #  This prevents the issue of large counts/latency being shown on metrics graphs when the queue's are actually empty
+    def build_data_to_send(data_map, previous_data_map) # rubocop:disable Metrics/AbcSize,Metrics/MethodLength
+      data_to_send = {}
+      keys = (data_map.keys + previous_data_map.keys).uniq
+      keys.each do |queue_name|
+        if previous_data_map.key?(queue_name)
+          if data_map.key?(queue_name)
+            # unless this is a new value, don't update it
+            unless previous_data_map[queue_name] == data_map[queue_name]
+              previous_data_map[queue_name] = data_map[queue_name]
+              data_to_send[queue_name] = data_map[queue_name]
+            end
+          else
+            # queue_name not in GoodJob query, so its value is 0
+            # unless this is a new value, don't update it
+            unless previous_data_map[queue_name].zero?
+              previous_data_map[queue_name] = 0
+              data_to_send[queue_name] = 0
+            end
+          end
+        else
+          previous_data_map[queue_name] = data_map[queue_name]
+          data_to_send[queue_name] = data_map[queue_name]
+        end
+      end
+
+      data_to_send
+    end
+
+    def job_queue_instrument_map
+      @job_queue_instrument_map ||= {}
+    end
+
+    def get_instrument(instrument_name, instrument_type)
+      unless job_queue_instrument_map.key?(instrument_name)
+        job_queue_instrument_map[instrument_name] = init_instrument(instrument_name, instrument_type)
+      end
+
+      job_queue_instrument_map[instrument_name]
+    end
+
+    def init_instrument(instrument_name, instrument_type)
+      if instrument_type == :queue_count
+        meter.create_gauge(
+          instrument_name, description: 'Current number of jobs in queue'
+        )
+      else # queue_latency
+        meter.create_gauge(
+          instrument_name, unit: 's', description: 'Minimum time a queued job is waiting to be performed.'
+        )
+      end
+    end
+
+    def metric_update_job_queue_count(queue_name, value)
+      job_queue_count_instrument_name = "#{queue_name}_queue_count"
+      instrument = get_instrument(job_queue_count_instrument_name, :queue_count)
+      instrument.record(value)
+    end
+
+    def metric_update_queue_min_latency(queue_name, value)
+      job_queue_latency_instrument_name = "#{queue_name}_queue_min_wait_time"
+      instrument = get_instrument(job_queue_latency_instrument_name, :queue_latency)
+      instrument.record(value)
+    end
+  end
+end
diff --git a/lib/irida/metrics_reporter.rb b/lib/irida/metrics_reporter.rb
new file mode 100644
index 0000000000..cca47da7e5
--- /dev/null
+++ b/lib/irida/metrics_reporter.rb
@@ -0,0 +1,92 @@
+# frozen_string_literal: true
+
+require 'irida/job_queue_metrics'
+
+require 'singleton'
+
+module Irida
+  class MetricsReporter # rubocop:disable Style/Documentation
+    include Singleton
+
+    def run(sleep_time)
+      if running?
+        Rails.logger.debug { "TelemetryReporter is already running on process #{@proc_id}" }
+        return
+      end
+      unless caller_can_control_reporter?
+        Rails.logger.debug { 'TelemetryReporter cannot be run by non primary thread.' }
+        return
+      end
+
+      run!(sleep_time)
+    end
+
+    def stop
+      unless running?
+        Rails.logger.debug { 'TelemetryReporter cannot be stopped as it is not running.' }
+        return
+      end
+      unless caller_can_control_reporter?
+        Rails.logger.debug { 'TelemetryReporter cannot be stopped by non primary thread.' }
+        return
+      end
+
+      stop!
+    end
+
+    private
+
+    def run!(sleep_time)
+      @proc_id = Process.pid
+      @sleep_time = sleep_time
+
+      # start the reporting loop
+      proc_loop
+    end
+
+    def stop!
+      Rails.logger.debug { 'Stopping metrics telemetry thread.' }
+
+      @reporter_thread&.terminate
+      @reporter_thread = nil
+      @proc_id = nil
+      OpenTelemetry.meter_provider.shutdown
+    end
+
+    def running?
+      @proc_id == Process.pid
+    end
+
+    def proc_loop
+      Rails.logger.debug do
+        "Starting TelemetryReporter on process #{@proc_id}, with interval time of #{@sleep_time} seconds"
+      end
+
+      @reporter_thread = Thread.new do
+        loop do
+          # run updates for metrics that are collected once per cycle and not per action in code
+          Irida::JobQueueMetrics.instance.update_queue_counts
+          Irida::JobQueueMetrics.instance.update_minimum_queue_times
+
+          report
+          sleep @sleep_time
+        end
+      end
+    end
+
+    # verifies that the current caller is a main process and not a console/runner/good_job instance
+    def caller_can_control_reporter?
+      console = caller.any? { |call| call.include?('console_command.rb') }
+      runner = caller.any? { |call| call.include?('runner_command.rb') }
+      good_job_instance = caller.any? { |call| call.include?('good_job/cli.rb') }
+
+      !(console || runner || good_job_instance)
+    end
+
+    def report
+      Rails.logger.debug { 'Reporting metrics to telemetry' }
+      # batch send telemetry
+      OpenTelemetry.meter_provider.metric_readers.each(&:pull)
+    end
+  end
+end
diff --git a/test/jobs/opentelemetry_job_metrics_test.rb b/test/jobs/opentelemetry_job_metrics_test.rb
new file mode 100644
index 0000000000..98810b8a8b
--- /dev/null
+++ b/test/jobs/opentelemetry_job_metrics_test.rb
@@ -0,0 +1,158 @@
+# frozen_string_literal: true
+
+require 'test_helper'
+require 'minitest/autorun'
+
+class OpenTelemetryJobMetricsTest < ActiveJob::TestCase
+  def setup
+    # Set up a mock OpenTelemetry Meter to handle calls from the job metrics singleton
+    @job_queue_metrics = Irida::JobQueueMetrics.instance
+    @mock_meter = Minitest::Mock.new
+    @job_queue_metrics.instance_variable_set(:@meter, @mock_meter)
+  end
+
+  def teardown
+    @mock_meter = nil
+    @job_queue_metrics.instance_variable_set(:@meter, nil)
+    @job_queue_metrics.instance_variable_set(:@job_queue_count_previous_value_map, nil)
+    @job_queue_metrics.instance_variable_set(:@job_queue_latency_previous_value_map, nil)
+    @job_queue_metrics.instance_variable_set(:@job_queue_instrument_map, nil)
+  end
+
+  class DummyJob < ApplicationJob
+    self.queue_adapter = :good_job
+    queue_as :default
+
+    def perform(time)
+      Rails.logger.info "Performing dummy job with sleep time #{time}"
+      sleep time
+    end
+  end
+
+  test 'metric instrument record' do
+    # mock instrument on meter that records values
+    mock_instrument = Minitest::Mock.new
+    mock_instrument.expect(:record, nil, [234])
+    # using 'default' as our queue name
+    @mock_meter.expect(:create_gauge, mock_instrument, ['default_queue_count'], description: String)
+
+    # call private method to test basic record
+    @job_queue_metrics.instance_eval("metric_update_job_queue_count('default',234)", __FILE__, __LINE__)
+
+    # verify objects called as expected
+    assert mock_instrument.verify
+    assert @mock_meter.verify
+  end
+
+  test 'update queue counts' do
+    # mock instrument on meter that records values
+    mock_queue_count_instrument = Minitest::Mock.new
+    mock_queue_count_instrument.expect(:record, nil, [1])
+    # using 'default' as our queue name
+    @mock_meter.expect(:create_gauge, mock_queue_count_instrument, ['default_queue_count'], description: String)
+
+    # queue a job
+    DummyJob.set(wait: 2.minutes).perform_later(1)
+    travel_to(3.minutes.from_now) do
+      # run metrics while job is queued
+      @job_queue_metrics.update_queue_counts
+      # don't let job procs hang
+      GoodJob.perform_inline
+    end
+
+    # verify objects called as expected
+    assert mock_queue_count_instrument.verify
+    assert @mock_meter.verify
+  end
+
+  test 'update queue latency' do
+    # mock instrument on meter that records values
+    mock_queue_latency_instrument = Minitest::Mock.new
+    mock_queue_latency_instrument.expect(:record, nil) do |latency|
+      latency > 55 && latency < 65 # Expect latency to be about a minute
+    end
+    # using 'default' as our queue name
+    @mock_meter.expect(
+      :create_gauge, mock_queue_latency_instrument, ['default_queue_min_wait_time'], unit: String, description: String
+    )
+
+    # queue a job
+    DummyJob.set(wait: 2.minutes).perform_later(1)
+    travel_to(3.minutes.from_now) do
+      # run metrics while job is queued
+      @job_queue_metrics.update_minimum_queue_times
+      # don't let job procs hang
+      GoodJob.perform_inline
+    end
+
+    # verify objects called as expected
+    assert mock_queue_latency_instrument.verify
+    assert @mock_meter.verify
+  end
+
+  test 'update queue count to 0 when queue is empty' do
+    # mock instrument on meter that records values
+    mock_queue_count_instrument = Minitest::Mock.new
+    mock_queue_count_instrument.expect(:record, nil, [3])
+    mock_queue_count_instrument.expect(:record, nil, [0])
+    # using 'default' as our queue name
+    @mock_meter.expect(:create_gauge, mock_queue_count_instrument, ['default_queue_count'], description: String)
+
+    # queue some jobs
+    DummyJob.set(wait: 2.minutes).perform_later(1)
+    DummyJob.set(wait: 2.minutes).perform_later(1)
+    DummyJob.set(wait: 2.minutes).perform_later(1)
+    travel_to(3.minutes.from_now) do
+      # run metrics while job is queued
+      @job_queue_metrics.update_queue_counts
+      # run metrics while job is queued AGAIN to verify repeat data points don't get resent
+      @job_queue_metrics.update_queue_counts
+      # don't let job procs hang
+      GoodJob.perform_inline
+    end
+
+    # run metrics again after jobs have finished
+    @job_queue_metrics.update_queue_counts
+    # run metrics again after jobs have finished AGAIN to verify repeat data points don't get resent
+    @job_queue_metrics.update_queue_counts
+
+    # verify objects called as expected
+    assert mock_queue_count_instrument.verify
+    assert @mock_meter.verify
+  end
+
+  test 'update queue latency to 0 when queue is empty' do
+    # mock instrument on meter that records values
+    mock_queue_latency_instrument = Minitest::Mock.new
+    mock_queue_latency_instrument.expect(:record, nil) do |latency|
+      latency > 55 && latency < 65 # Expect latency to be about a minute
+    end
+    mock_queue_latency_instrument.expect(:record, nil, [0]) # should be 0 when jobs are cleared
+    # using 'default' as our queue name
+    @mock_meter.expect(
+      :create_gauge, mock_queue_latency_instrument, ['default_queue_min_wait_time'], unit: String, description: String
+    )
+
+    # queue some jobs
+    DummyJob.set(wait: 2.minutes).perform_later(1)
+    DummyJob.set(wait: 2.minutes).perform_later(1)
+    DummyJob.set(wait: 2.minutes).perform_later(1)
+    travel_to(3.minutes.from_now) do
+      # run metrics while job is queued
+      @job_queue_metrics.update_minimum_queue_times
+      # run metrics while job is queued AGAIN to verify repeat data points don't get resent
+      @job_queue_metrics.update_minimum_queue_times
+      # don't let job procs hang
+      GoodJob.perform_inline
+    end
+
+    # run metrics again after jobs have finished
+    @job_queue_metrics.update_minimum_queue_times
+    # run metrics again after jobs have finished AGAIN to verify repeat data points don't get resent
+    @job_queue_metrics.update_minimum_queue_times
+
+    # verify objects called as expected
+    assert mock_queue_latency_instrument.verify
+    assert @mock_meter.verify
+  end
+end


Reviews:
user: login: ericenns
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T12:31:28Z
user: login: JeffreyThiessen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T19:00:03Z
user: login: deepsidhu85
body: Looks good to me!
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T19:34:36Z
user: login: ericenns
body: This is looking good, just one minor comment.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-28T16:21:58Z
user: login: JeffreyThiessen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-28T16:31:37Z
user: login: ericenns
body: Looks good to me!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-29T13:22:15Z

Review comments:
diff_hunk: @@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+require 'opentelemetry/sdk'
+require 'opentelemetry-exporter-otlp'
+
+require 'opentelemetry-metrics-sdk'
+require 'opentelemetry-exporter-otlp-metrics'
+
+require 'irida/metrics_reporter'
+
+if Flipper.enabled?(:telemetry) && ENV['OTEL_EXPORTER_OTLP_METRICS_ENDPOINT']
path: config/initializers/opentelemetry.rb
user: login: ericenns
body: **nitpick (blocking):** Flipper flags cannot be checked at initialization, they are more intended to be used at runtime. Instead can we just base this off of `ENV.include?('OTEL_EXPORTER_OTLP_METRICS_ENDPOINT')`
created_at: 2025-04-16T12:31:28Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+require 'opentelemetry/sdk'
+require 'opentelemetry-exporter-otlp'
+
+require 'opentelemetry-metrics-sdk'
+require 'opentelemetry-exporter-otlp-metrics'
+
+require 'irida/metrics_reporter'
+
+if Flipper.enabled?(:telemetry) && ENV['OTEL_EXPORTER_OTLP_METRICS_ENDPOINT']
path: config/initializers/opentelemetry.rb
user: login: JeffreyThiessen
body: Fixed in https://github.com/phac-nml/irida-next/pull/1026/commits/26620dbe6dd7f4e85c50794c91ec84869ea5ebf3
created_at: 2025-04-16T19:00:02Z
author_association: MEMBER
diff_hunk: @@ -19,7 +19,7 @@ default: &default
   # https://guides.rubyonrails.org/configuring.html#database-pooling
   #
   # For GoodJob container, pool size is GOOD_JOB_MAX_THREADS (5) + 3 (1 for LISTEN/NOTIFY, 2 for CRON)
-  pool: <%= $PROGRAM_NAME.include?("good_job") ? ENV.fetch("GOOD_JOB_MAX_THREADS", 5).to_i + 3 : (ENV.fetch("RAILS_MAX_THREADS", 5).to_i) %>
+  pool: <%= $PROGRAM_NAME.include?("good_job") ? ENV.fetch("GOOD_JOB_MAX_THREADS", 5).to_i + 3 : (ENV.fetch("RAILS_MAX_THREADS", 6).to_i) %>
path: config/database.yml
user: login: ericenns
body: **nitpick (blocking):** Instead of changing the default to 6 can you instead add  ` + 1` to the end otherwise, if an instance specifies `RAILS_MAX_THREADS` it would be 1 short for the telemetry collection.
created_at: 2025-04-28T16:18:32Z
author_association: MEMBER
diff_hunk: @@ -19,7 +19,7 @@ default: &default
   # https://guides.rubyonrails.org/configuring.html#database-pooling
   #
   # For GoodJob container, pool size is GOOD_JOB_MAX_THREADS (5) + 3 (1 for LISTEN/NOTIFY, 2 for CRON)
-  pool: <%= $PROGRAM_NAME.include?("good_job") ? ENV.fetch("GOOD_JOB_MAX_THREADS", 5).to_i + 3 : (ENV.fetch("RAILS_MAX_THREADS", 5).to_i) %>
+  pool: <%= $PROGRAM_NAME.include?("good_job") ? ENV.fetch("GOOD_JOB_MAX_THREADS", 5).to_i + 3 : (ENV.fetch("RAILS_MAX_THREADS", 6).to_i) %>
path: config/database.yml
user: login: JeffreyThiessen
body: Good catch. Fixed in https://github.com/phac-nml/irida-next/pull/1026/commits/add032ba03fa77f05bbdad9654d12883949cb443
created_at: 2025-04-28T16:31:37Z
author_association: MEMBER

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-10T18:51:01Z
body: ## Code Metrics Report
| Coverage | Test Execution Time |
|---------:|--------------------:|
| 94.2%    | 12m11s              |


### Code coverage of files in pull request scope (34.2%)

|                                                                           Files                                                                           | Coverage |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------|---------:|
| [app/jobs/application_job.rb](https://github.com/phac-nml/irida-next/blob/b224c03f74487de3b8abdc94bd8fae73f6442d57/app%2Fjobs%2Fapplication_job.rb)       | 100.0%   |
| [lib/irida/job_queue_metrics.rb](https://github.com/phac-nml/irida-next/blob/b224c03f74487de3b8abdc94bd8fae73f6442d57/lib%2Firida%2Fjob_queue_metrics.rb) | 83.3%    |
| [lib/irida/metrics_reporter.rb](https://github.com/phac-nml/irida-next/blob/b224c03f74487de3b8abdc94bd8fae73f6442d57/lib%2Firida%2Fmetrics_reporter.rb)   | 0.0%     |

---
Reported by [octocov](https://github.com/k1LoW/octocov)
<!-- octocov -->
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-16T18:48:07Z
body: ## Code Metrics Report
| Coverage | Test Execution Time |
|---------:|--------------------:|
| 93.8%    | 12m49s              |


### Code coverage of files in pull request scope (1.5%)

|                                                                           Files                                                                           | Coverage |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------|---------:|
| [app/jobs/application_job.rb](https://github.com/phac-nml/irida-next/blob/26620dbe6dd7f4e85c50794c91ec84869ea5ebf3/app%2Fjobs%2Fapplication_job.rb)       | 33.3%    |
| [lib/irida/job_queue_metrics.rb](https://github.com/phac-nml/irida-next/blob/26620dbe6dd7f4e85c50794c91ec84869ea5ebf3/lib%2Firida%2Fjob_queue_metrics.rb) | 0.0%     |
| [lib/irida/metrics_reporter.rb](https://github.com/phac-nml/irida-next/blob/26620dbe6dd7f4e85c50794c91ec84869ea5ebf3/lib%2Firida%2Fmetrics_reporter.rb)   | 0.0%     |

---
Reported by [octocov](https://github.com/k1LoW/octocov)
<!-- octocov -->
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-17T20:09:01Z
body: ## Code Metrics Report
| Coverage | Test Execution Time |
|---------:|--------------------:|
| 93.7%    | 11m45s              |


### Code coverage of files in pull request scope (1.5%)

|                                                                           Files                                                                           | Coverage |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------|---------:|
| [app/jobs/application_job.rb](https://github.com/phac-nml/irida-next/blob/5d2775ef5ce16318680c0d1e0149a5a43bda9967/app%2Fjobs%2Fapplication_job.rb)       | 33.3%    |
| [lib/irida/job_queue_metrics.rb](https://github.com/phac-nml/irida-next/blob/5d2775ef5ce16318680c0d1e0149a5a43bda9967/lib%2Firida%2Fjob_queue_metrics.rb) | 0.0%     |
| [lib/irida/metrics_reporter.rb](https://github.com/phac-nml/irida-next/blob/5d2775ef5ce16318680c0d1e0149a5a43bda9967/lib%2Firida%2Fmetrics_reporter.rb)   | 0.0%     |

---
Reported by [octocov](https://github.com/k1LoW/octocov)
<!-- octocov -->
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-28T18:52:27Z
body: ## Code Metrics Report
| Coverage | Test Execution Time |
|---------:|--------------------:|
| 93.6%    | 12m38s              |


### Code coverage of files in pull request scope (0.0%)

|                                                                           Files                                                                           | Coverage |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------|---------:|
| [lib/irida/job_queue_metrics.rb](https://github.com/phac-nml/irida-next/blob/d964e449ebe2d31f8aeeeda6312e91f1ce618490/lib%2Firida%2Fjob_queue_metrics.rb) | 0.0%     |
| [lib/irida/metrics_reporter.rb](https://github.com/phac-nml/irida-next/blob/d964e449ebe2d31f8aeeeda6312e91f1ce618490/lib%2Firida%2Fmetrics_reporter.rb)   | 0.0%     |

---
Reported by [octocov](https://github.com/k1LoW/octocov)
<!-- octocov -->
author_association: NONE