Repository: department-of-veterans-affairs/vets-api
Pull request #21529: remove kafka schema dynamic fetching flipper
Author: login: aherzberg
Labels: name: test-passing
name: require-backend-approval
Created: 2025-04-01T18:50:41Z | Merged: 2025-04-02T18:13:41Z

## Summary
Now that we have confirmed that schema's can be looked up dynamically, this removes associated code that switches between using a hard-coded schema and making a request to the schema registry.

## Related issue(s)
https://github.com/orgs/department-of-veterans-affairs/projects/1509/views/5?pane=issue&itemId=104404080&issue=department-of-veterans-affairs%7CVES%7C4661

## Testing done

- specs

## Screenshots
_Note: Optional_

## What areas of the site does it impact?
*(Describe what parts of the site are impacted and*if*code touched other areas)*

## Acceptance criteria

- [ ]  I fixed|updated|added unit tests and integration tests for each feature (if applicable).
- [ ]  No error nor warning in the console.
- [ ]  Events are being sent to the appropriate logging solution
- [ ]  Documentation has been updated (link to documentation)
- [ ]  No sensitive information (i.e. PII/credentials/internal URLs/etc.) is captured in logging, hardcoded, or specs
- [ ]  Feature/bug has a monitor built into Datadog (if applicable)
- [ ]  If app impacted requires authentication, did you login to a local build and verify all authenticated routes work as expected
- [ ]  I added a screenshot of the developed feature

## Requested Feedback

(OPTIONAL)_What should the reviewers know in addition to the above. Is there anything specific you wish the reviewer to assist with. Do you have any concerns with this PR, why?_



Diff:
diff --git a/config/features.yml b/config/features.yml
index fbcbab50981a..53b611528cbe 100644
--- a/config/features.yml
+++ b/config/features.yml
@@ -2090,9 +2090,6 @@ features:
   kafka_producer:
     actor_type: cookie_id
     description: Enables the Kafka producer for the VA.gov platform
-  kafka_producer_fetch_schema_dynamically:
-    actor_type: cookie_id
-    description: Enables the Kafka producer to fetch schema dynamically
   show_about_yellow_ribbon_program:
     actor_type: user
     description: If enabled, show additional info about the yellow ribbon program
diff --git a/lib/kafka/avro_producer.rb b/lib/kafka/avro_producer.rb
index 7189a31f567e..5d2ccc5f8ac6 100644
--- a/lib/kafka/avro_producer.rb
+++ b/lib/kafka/avro_producer.rb
@@ -29,15 +29,9 @@ def produce(topic, payload, schema_version: 'latest')
     private
 
     def get_schema(topic, schema_version)
-      if Flipper.enabled?(:kafka_producer_fetch_schema_dynamically)
-        response = @registry.subject_version(topic, schema_version)
-        schema = response['schema']
-        @schema_id = response['id']
-      else
-        schema_path = Rails.root.join('lib', 'kafka', 'schemas', "#{topic}-value-#{schema_version}.avsc")
-        schema = File.read(schema_path)
-      end
-
+      response = @registry.subject_version(topic, schema_version)
+      schema = response['schema']
+      @schema_id = response['id']
       Avro::Schema.parse(schema)
     end
 
@@ -52,9 +46,6 @@ def encode_payload(schema, payload)
 
       # Add magic byte and schema ID to the payload
       magic_byte = [0].pack('C')
-      # NOTE: Use fetched schema id from schema registry but if not found,
-      # ID = 5 is the Event Bus schema ID for test schema.replace this with the actual schema ID when running locally
-      @schema_id ||= 5
       schema_id_bytes = [@schema_id].pack('N') # should be schema id
       magic_byte + schema_id_bytes + avro_payload
     end
diff --git a/lib/kafka/schema_registry/service.rb b/lib/kafka/schema_registry/service.rb
index 85dca037595f..34958f9367f9 100644
--- a/lib/kafka/schema_registry/service.rb
+++ b/lib/kafka/schema_registry/service.rb
@@ -21,11 +21,13 @@ def subjects
 
       # List all versions for a subject
       def subject_versions(topic)
+        validate_topic(topic)
         request("#{SCHEMA_REGISTRY_PATH_PREFIX}/subjects/#{topic}-value/versions", idempotent: true)
       end
 
       # Get a specific version for a subject
       def subject_version(topic, version = 'latest')
+        validate_topic(topic)
         request("#{SCHEMA_REGISTRY_PATH_PREFIX}/subjects/#{topic}-value/versions/#{version}", idempotent: true)
       end
 
@@ -36,6 +38,7 @@ def schema_subject_versions(schema_id)
 
       # Check if a schema exists. Returns nil if not found.
       def check(topic, schema)
+        validate_topic(topic)
         data = request("#{SCHEMA_REGISTRY_PATH_PREFIX}/subjects/#{topic}-value",
                        method: :post,
                        expects: [200, 404],
@@ -51,6 +54,7 @@ def check(topic, schema)
       # - false if incompatible
       # http://docs.confluent.io/3.1.2/schema-registry/docs/api.html#compatibility
       def compatible?(topic, schema, version = 'latest')
+        validate_topic(topic)
         data = request("#{SCHEMA_REGISTRY_PATH_PREFIX}/compatibility/subjects/#{topic}-value/versions/#{version}",
                        method: :post,
                        expects: [200, 404],
@@ -65,6 +69,7 @@ def compatible?(topic, schema, version = 'latest')
       # - a list of compatibility issues
       # https://docs.confluent.io/platform/current/schema-registry/develop/api.html#sr-api-compatibility
       def compatibility_issues(topic, schema, version = 'latest')
+        validate_topic(topic)
         data = request("#{SCHEMA_REGISTRY_PATH_PREFIX}/compatibility/subjects/#{topic}-value/versions/#{version}",
                        method: :post,
                        expects: [200, 404],
@@ -79,6 +84,7 @@ def global_config
 
       # Get config for subject
       def subject_config(topic)
+        validate_topic(topic)
         request("#{SCHEMA_REGISTRY_PATH_PREFIX}/config/#{topic}-value", idempotent: true)
       end
 
@@ -92,6 +98,10 @@ def request(path, method: :get, **options)
 
         JSON.parse(response.body)
       end
+
+      def validate_topic(topic)
+        raise WaterDrop::Errors::MessageInvalidError.new(topic: 'no topic provided') if topic.blank?
+      end
     end
   end
 end
diff --git a/lib/kafka/schemas/submission_trace_mock_dev-value-1.avsc b/lib/kafka/schemas/submission_trace_mock_dev-value-1.avsc
deleted file mode 100644
index 6ce8f6f26b36..000000000000
--- a/lib/kafka/schemas/submission_trace_mock_dev-value-1.avsc
+++ /dev/null
@@ -1,12 +0,0 @@
-{
-  "type": "record",
-  "name": "TestRecord",
-  "namespace": "gov.va.eventbus.test.data",
-  "fields": [
-    {
-      "name": "data",
-      "type": { "type": "map", "values": "string" },
-      "default": {}
-    }
-  ]
-}
\ No newline at end of file
diff --git a/lib/kafka/schemas/submission_trace_schema_dev-value-1.avsc b/lib/kafka/schemas/submission_trace_schema_dev-value-1.avsc
deleted file mode 100644
index 160f5ff3155f..000000000000
--- a/lib/kafka/schemas/submission_trace_schema_dev-value-1.avsc
+++ /dev/null
@@ -1,93 +0,0 @@
-{
-  "type": "record",
-  "name": "SubmissionTrace",
-  "namespace": "gov.va.submissiontrace.form.data",
-  "fields": [
-    {
-      "name": "priorId",
-      "type": [
-        "null",
-        "string"
-      ],
-      "doc": "ID that represents a unique identifier from a upstream system"
-    },
-    {
-      "name": "currentId",
-      "type": "string",
-      "doc": "ID that represents a unique identifier in the current system"
-    },
-    {
-      "name": "nextId",
-      "type": [
-        "null",
-        "string"
-      ],
-      "doc": "ID that represents a unique identifier in the downstream system"
-    },
-    {
-      "name": "icn",
-      "type": [
-        "null",
-        "string"
-      ],
-      "doc": "Unique Veteran ID to identify the Veteran independently of other IDs provided"
-    },
-    {
-      "name": "vasiId",
-      "type": "string",
-      "doc": "ID that represents a unique identifier in the Veteran Affairs Systems Inventory (VASI)"
-    },
-    {
-      "name": "systemName",
-      "type": [
-        {
-          "type": "enum",
-          "name": "SystemName",
-          "symbols": [
-            "Lighthouse",
-            "CMP",
-            "VBMS",
-            "VA_gov",
-            "VES"
-          ]
-        }
-      ],
-      "doc": "System submitting status update, e.g. va.gov"
-    },
-    {
-      "name": "submissionName",
-      "type": [
-        {
-          "type": "enum",
-          "name": "SubmissionName",
-          "symbols": [
-            "F1010EZ",
-            "F527EZ"
-          ]
-        }
-      ],
-      "doc": "Form or name of the submission; should be the same across systems, e.g. 526ez, 4142, Notice Of Disagreement, etc."
-    },
-    {
-      "name": "state",
-      "type": "string",
-      "doc": "What triggered the event, limited to this set (but not enforced): received, sent, error, completed"
-    },
-    {
-      "name": "timestamp",
-      "type": "string",
-      "doc": "Current datetime in JS standard format ISO 8601"
-    },
-    {
-      "name": "additionalIds",
-      "type": [
-        "null",
-        {
-          "type": "array",
-          "items": "string"
-        }
-      ],
-      "doc": "(Optional) for cases when more than one current ID is appropriate"
-    }
-  ]
-}
\ No newline at end of file
diff --git a/spec/lib/kafka/avro_producer_spec.rb b/spec/lib/kafka/avro_producer_spec.rb
index c56569047293..8a0db05029c7 100644
--- a/spec/lib/kafka/avro_producer_spec.rb
+++ b/spec/lib/kafka/avro_producer_spec.rb
@@ -3,19 +3,24 @@
 require 'rails_helper'
 require 'kafka/avro_producer'
 require 'kafka/oauth_token_refresher'
+require 'kafka/schema_registry/service'
 
 describe Kafka::AvroProducer do
   let(:avro_producer) { described_class.new }
-  let(:schema_path) { Rails.root.join('lib', 'kafka', 'schemas', 'submission_trace_mock_dev-value-1.avsc') }
-  let(:schema_file) { File.read(schema_path) }
-  let(:schema) { Avro::Schema.parse(schema_file) }
   let(:valid_payload) { { 'data' => { 'key' => 'value' } } }
   let(:invalid_payload) { { 'invalid_key' => 'value' } }
+  let(:schema) do
+    VCR.use_cassette('kafka/topics') do
+      response = Kafka::SchemaRegistry::Service.new.subject_version('topic-1', 'latest')
+
+      schema = response['schema']
+      Avro::Schema.parse(schema)
+    end
+  end
 
   before do
     allow(Rails).to receive(:env).and_return(ActiveSupport::StringInquirer.new('test'))
     allow(Flipper).to receive(:enabled?).with(:kafka_producer).and_return(true)
-    allow(Flipper).to receive(:enabled?).with(:kafka_producer_fetch_schema_dynamically).and_return(true)
     allow(Kafka::OauthTokenRefresher).to receive(:new).and_return(double(on_oauthbearer_token_refresh: 'token'))
   end
 
@@ -82,38 +87,31 @@
         end
       end
     end
-
-    context 'with hardcoded schema registry retrieval' do
-      before do
-        allow(Flipper).to receive(:enabled?).with(:kafka_producer_fetch_schema_dynamically).and_return(false)
-        allow(File).to receive(:read).and_return(schema_file)
-      end
-
-      it 'produces a message to the specified topic' do
-        avro_producer.produce('submission_trace_mock_dev', valid_payload)
-        expect(avro_producer.producer.client.messages.length).to eq(1)
-        topic_1_messages = avro_producer.producer.client.messages_for('submission_trace_mock_dev')
-        expect(topic_1_messages.length).to eq(1)
-        expect(topic_1_messages[0][:payload]).to be_a(String)
-        expect(topic_1_messages[0][:payload]).to eq(topic1_payload_value)
-      end
-    end
   end
 
   context 'when an error occurs' do
     before do
       Kafka::ProducerManager.instance.send(:setup_producer)
-      allow(avro_producer).to receive(:get_schema).and_return(schema)
     end
 
-    it 'triggers MessageInvalidError if no valid topic is provided' do
+    it 'triggers MessageInvalidError if empty string topic is provided' do
+      expect(Rails.logger).to receive(:error).with(/Message is invalid/)
+
+      # Send an invalid message to trigger an error (no topic provided)
+      expect do
+        avro_producer.produce('', valid_payload)
+      end.to raise_error(WaterDrop::Errors::MessageInvalidError,
+                         { topic: 'no topic provided' }.to_s)
+    end
+
+    it 'triggers MessageInvalidError if nil topic is provided' do
       expect(Rails.logger).to receive(:error).with(/Message is invalid/)
 
       # Send an invalid message to trigger an error (no topic provided)
       expect do
         avro_producer.produce(nil, valid_payload)
       end.to raise_error(WaterDrop::Errors::MessageInvalidError,
-                         { topic: 'does not match the topic allowed format' }.to_s)
+                         { topic: 'no topic provided' }.to_s)
     end
 
     it 'triggers MessageInvalidError if no valid payload is provided' do
@@ -126,7 +124,9 @@
 
       # Send an invalid message to trigger an error (no payload provided)
       expect do
-        avro_producer.produce('topic-1', large_payload)
+        VCR.use_cassette('kafka/topics') do
+          avro_producer.produce('topic-1', large_payload)
+        end
       end.to raise_error(WaterDrop::Errors::MessageInvalidError,
                          { payload: 'is more than `max_payload_size` config value' }.to_s)
     end
@@ -150,9 +150,11 @@
         .and_raise(WaterDrop::Errors::ProduceError)
 
       # Trigger the error and handle it
-      expect do
-        avro_producer.produce('topic-1', valid_payload)
-      end.to raise_error(WaterDrop::Errors::ProduceError)
+      VCR.use_cassette('kafka/topics') do
+        expect do
+          avro_producer.produce('topic-1', valid_payload)
+        end.to raise_error(WaterDrop::Errors::ProduceError)
+      end
     end
 
     it 'logs a message when a SchemaValidationError occurs' do
@@ -161,14 +163,17 @@
       # allow(Avro::SchemaValidator).to receive(:validate!).and_raise(Avro::SchemaValidator::ValidationError)
 
       # Trigger the error using an invalid schema
-      expect do
-        avro_producer.produce('topic-1', invalid_payload)
-      end.to raise_error(Avro::SchemaValidator::ValidationError)
+      VCR.use_cassette('kafka/topics') do
+        expect do
+          avro_producer.produce('topic-1', invalid_payload)
+        end.to raise_error(Avro::SchemaValidator::ValidationError)
+      end
     end
   end
 
   describe '#encode_payload' do
     it 'encodes the payload using the specified schema' do
+      avro_producer.instance_variable_set(:@schema_id, 5)
       encoder = Avro::IO::BinaryEncoder.new(StringIO.new)
       datum_writer = Avro::IO::DatumWriter.new(schema)
 


Reviews:
user: login: madebydna
body: This looks good. I just added a comment to remove the hard-coded schema id we still have in the `AvroProducer`. 
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T02:39:22Z
user: login: aherzberg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T15:45:51Z
user: login: madebydna
body: Nice work!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T15:57:03Z
user: login: LindseySaari
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T16:54:21Z
user: login: LindseySaari
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T16:55:01Z

Review comments:
path: lib/kafka/avro_producer.rb
user: login: madebydna
body: Note that there is a hard-coded schema ID in the `encode_payload` method that should also be removed.
created_at: 2025-04-01T21:12:40Z
author_association: CONTRIBUTOR
path: lib/kafka/avro_producer.rb
user: login: aherzberg
body: good catch. removed the hard-coded schema id
created_at: 2025-04-02T15:45:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2090,9 +2090,6 @@ features:
   kafka_producer:
     actor_type: cookie_id
     description: Enables the Kafka producer for the VA.gov platform
-  kafka_producer_fetch_schema_dynamically:
path: config/features.yml
user: login: LindseySaari
body: Just a reminder that this will need to be manually removed via the rails console. [Guidance here](https://depo-platform-documentation.scrollhelp.site/developer-docs/feature-toggles-guide#FeatureTogglesGuide-Removingold/unusedfeaturetoggles)
created_at: 2025-04-02T16:54:21Z
author_association: CONTRIBUTOR

PR comments:
user: login: madebydna
created_at: 2025-04-02T13:45:16Z
body: @aherzberg, I just found this [guide on removing feature toggles](https://depo-platform-documentation.scrollhelp.site/developer-docs/feature-toggles-guide#FeatureTogglesGuide-Removingold/unusedfeaturetoggles). I'm asking for clarification on whether removing the toggle from the code and from the `features.yml` file needs to be done in two steps. In any case, there is a non-obvious step of removing feature toggles manually from the DB in all the different environments. 
author_association: CONTRIBUTOR
user: login: madebydna
created_at: 2025-04-02T15:18:24Z
body: Update: I found out that removing the toggle from the code and from `features.yml` in one PR is okay. 
author_association: CONTRIBUTOR