Repository: Enterprise-CMCS/macpro-mdct-mcr
Pull request #12077: update to kafka topics to include form templates
Author: login: britt-mo
Created: 2025-03-18T14:54:24Z | Merged: 2025-03-21T14:35:56Z

### Description
Some fields were missing from Kafka topics, this PR updates the topics to also include the formTemplates as well as formData from s3.

Also had to remove the `lodash` import due to ClowdWatch errors.


### Related ticket(s)
<!-- Link to related ticket(s) or issue(s) -->
<!-- Hint: Type CMDCT-<ticket-number> for autolinking -->
CMDCT-4419

---
### How to test
Go into the AWS console for mcr-dev
go into Lambda, search by `4419` or `createTopics`, go into the logs and see that the expected topics (from `condensedTopicList` have been created.
Go into Dynamo Db -> explore items -> cmdct-4419-form-template-versions
select and delete the template version item that you'll be testing (MCPAR, MLR, etc)
Go to the deployed endpoint (https://d2962y1kys7hj8.cloudfront.net/), log in, and create a program.
Go to Cloudwatch logs and select the log that relates to the program you created (postMLRBucketData, postMCPARBucketData, etc) 
Verify in the logs that both `fieldData/` and `formTemplates/` topics have been created


### Notes
<!-- Changed dependencies, .env files, configs, etc. -->
<!-- Instructions for local dev, e.g. requires new installs in directories -->


---
### Pre-review checklist
<!-- Complete the following steps before opening for review -->
- [ ] I have added [thorough](https://shorturl.at/aejkF) tests, if necessary
- [ ] I have updated relevant documentation, if necessary
- [ ] I have performed a self-review of my code
- [ ] I have manually tested this PR in the deployed cloud environment

---
### Pre-merge checklist
<!-- Complete the following steps before merging -->

#### Review
- [ ] Design: This work has been reviewed and approved by design, if necessary
- [ ] Product: This work has been reviewed and approved by product owner, if necessary

#### Security
_If either of the following are true, notify the team's ISSO (Information System Security Officer)._

- [ ] These changes are significant enough to require an update to the SIA.
- [ ] These changes are significant enough to require a penetration test.
---

<!-- If deploying to val or prod, click 'Preview' and select template -->
_convert to a different template: [test → val](?expand=1&template=test-to-val-deployment.md)_ | _[val → prod](?expand=1&template=val-to-prod-deployment.md)_



Diff:
diff --git a/services/app-api/handlers/kafka/post/postKafkaData.ts b/services/app-api/handlers/kafka/post/postKafkaData.ts
index 22b93f2bd..c302362a5 100644
--- a/services/app-api/handlers/kafka/post/postKafkaData.ts
+++ b/services/app-api/handlers/kafka/post/postKafkaData.ts
@@ -14,9 +14,36 @@ const tables = [
   { sourceName: reportTables.NAAAR, topicName: tableTopics.NAAAR },
 ];
 const buckets = [
-  { sourceName: reportBuckets.MCPAR, topicName: bucketTopics.MCPAR },
-  { sourceName: reportBuckets.MLR, topicName: bucketTopics.MLR },
-  { sourceName: reportBuckets.NAAAR, topicName: bucketTopics.NAAAR },
+  {
+    sourceName: reportBuckets.MCPAR,
+    topicName: bucketTopics.MCPAR,
+    s3Prefix: "fieldData/",
+  },
+  {
+    sourceName: reportBuckets.MCPAR,
+    topicName: bucketTopics.MCPAR_TEMPLATE,
+    s3Prefix: "formTemplates/",
+  },
+  {
+    sourceName: reportBuckets.MLR,
+    topicName: bucketTopics.MLR,
+    s3Prefix: "fieldData/",
+  },
+  {
+    sourceName: reportBuckets.MLR,
+    topicName: bucketTopics.MLR_TEMPLATE,
+    s3Prefix: "formTemplates/",
+  },
+  {
+    sourceName: reportBuckets.NAAAR,
+    topicName: bucketTopics.NAAAR,
+    s3Prefix: "fieldData/",
+  },
+  {
+    sourceName: reportBuckets.NAAAR,
+    topicName: bucketTopics.NAAAR_TEMPLATE,
+    s3Prefix: "formTemplates/",
+  },
 ];
 
 const postKafkaData = new KafkaSourceLib(topicPrefix, version, tables, buckets);
diff --git a/services/app-api/serverless.yml b/services/app-api/serverless.yml
index d095bf4d2..226c802d8 100644
--- a/services/app-api/serverless.yml
+++ b/services/app-api/serverless.yml
@@ -344,7 +344,6 @@ functions:
           forceDeploy: true
           event: s3:ObjectCreated:*
           rules:
-            - prefix: fieldData/
             - suffix: .json
       - s3:
           bucket: ${self:custom.naaarFormBucket}
@@ -352,7 +351,6 @@ functions:
           forceDeploy: true
           event: s3:ObjectTagging:Put
           rules:
-            - prefix: fieldData/
             - suffix: .json
   postMlrBucketData:
     handler: handlers/kafka/post/postKafkaData.handler
@@ -374,7 +372,6 @@ functions:
           forceDeploy: true
           event: s3:ObjectCreated:*
           rules:
-            - prefix: fieldData/
             - suffix: .json
       - s3:
           bucket: ${self:custom.mlrFormBucket}
@@ -382,7 +379,6 @@ functions:
           forceDeploy: true
           event: s3:ObjectTagging:Put
           rules:
-            - prefix: fieldData/
             - suffix: .json
   postMcparBucketData:
     handler: handlers/kafka/post/postKafkaData.handler
@@ -404,7 +400,6 @@ functions:
           forceDeploy: true
           event: s3:ObjectCreated:*
           rules:
-            - prefix: fieldData/
             - suffix: .json
       - s3:
           bucket: ${self:custom.mcparFormBucket}
@@ -412,7 +407,6 @@ functions:
           forceDeploy: true
           event: s3:ObjectTagging:Put
           rules:
-            - prefix: fieldData/
             - suffix: .json
 resources:
   Resources:
diff --git a/services/app-api/utils/constants/constants.ts b/services/app-api/utils/constants/constants.ts
index b6140fe0c..8c29ee25d 100644
--- a/services/app-api/utils/constants/constants.ts
+++ b/services/app-api/utils/constants/constants.ts
@@ -53,6 +53,9 @@ export const bucketTopics = {
   MCPAR: "mcpar-form",
   MLR: "mlr-form",
   NAAAR: "naaar-form",
+  MCPAR_TEMPLATE: "mcpar-form-template",
+  MLR_TEMPLATE: "mlr-form-template",
+  NAAAR_TEMPLATE: "naaar-form-template",
 };
 
 export const formTemplateTableName = process.env.FORM_TEMPLATE_TABLE_NAME!;
diff --git a/services/app-api/utils/kafka/kafka-source-lib.test.ts b/services/app-api/utils/kafka/kafka-source-lib.test.ts
index 67dff0016..e5f02dadb 100644
--- a/services/app-api/utils/kafka/kafka-source-lib.test.ts
+++ b/services/app-api/utils/kafka/kafka-source-lib.test.ts
@@ -27,6 +27,7 @@ const table = { sourceName: `${stage}-aTable`, topicName: "aTable-reports" };
 const bucket = {
   sourceName: `database-${stage}-aBucket`,
   topicName: "aBucket-forms",
+  s3Prefix: "fieldData/",
 };
 const brokerString = "brokerA,brokerB,brokerC";
 const dynamoEvent = {
@@ -188,7 +189,9 @@ describe("Test Kafka Lib", () => {
   });
 
   test("Does not pass through events from unrelated tables or buckets", async () => {
-    const badMaps = [{ sourceName: "bad", topicName: "bad" }];
+    const badMaps = [
+      { sourceName: "bad", topicName: "bad", s3Prefix: "fieldData/" },
+    ];
     const sourceLib = new KafkaSourceLib(
       "mcr",
       "v0",
diff --git a/services/app-api/utils/kafka/kafka-source-lib.ts b/services/app-api/utils/kafka/kafka-source-lib.ts
index 6e0d2bfef..122af14f0 100644
--- a/services/app-api/utils/kafka/kafka-source-lib.ts
+++ b/services/app-api/utils/kafka/kafka-source-lib.ts
@@ -19,6 +19,12 @@ type SourceTopicMapping = {
   topicName: string;
 };
 
+type SourceBucketTopicMapping = {
+  sourceName: string;
+  topicName: string;
+  s3Prefix: string;
+};
+
 let kafka: Kafka;
 let producer: Producer;
 
@@ -39,7 +45,7 @@ class KafkaSourceLib {
   topicPrefix: string;
   version: string | null;
   tables: SourceTopicMapping[];
-  buckets: SourceTopicMapping[];
+  buckets: SourceBucketTopicMapping[];
   connected: boolean;
   topicNamespace: string;
   stage: string;
@@ -47,7 +53,7 @@ class KafkaSourceLib {
     topicPrefix: string,
     version: string | null,
     tables: SourceTopicMapping[],
-    buckets: SourceTopicMapping[]
+    buckets: SourceBucketTopicMapping[]
   ) {
     if (!process.env.BOOTSTRAP_BROKER_STRING_TLS) {
       throw new Error("Missing Broker Config. ");
@@ -110,9 +116,12 @@ class KafkaSourceLib {
    * @param bucketArn - ARN formatted like 'arn:aws:s3:::{stack}-{stage}-{bucket}' e.g. arn:aws:s3:::database-main-mcpar
    * @returns A formatted topic name with "-form" specified
    */
-  determineS3TopicName(bucketArn: string) {
+  determineS3TopicName(bucketArn: string, key: string) {
     for (const bucket of this.buckets) {
-      if (bucketArn.includes(bucket.sourceName)) {
+      if (
+        bucketArn.includes(bucket.sourceName) &&
+        key.startsWith(bucket.s3Prefix)
+      ) {
         return this.topic(bucket.topicName);
       }
     }
@@ -174,14 +183,10 @@ class KafkaSourceLib {
         // Handle any S3 events
         const s3Record = record as S3EventRecord;
         const key: string = s3Record.s3.object.key;
-        topicName = this.determineS3TopicName(s3Record.s3.bucket.arn);
+        topicName = this.determineS3TopicName(s3Record.s3.bucket.arn, key);
 
         // Filter for only the response info
-        if (
-          !topicName ||
-          !key.startsWith("fieldData/") ||
-          !key.includes(".json")
-        ) {
+        if (!topicName || !key.includes(".json")) {
           continue;
         }
         payload = await this.createS3Payload(record);
diff --git a/services/topics/handlers/createTopics.js b/services/topics/handlers/createTopics.js
index 9a6234b94..03dfb1138 100644
--- a/services/topics/handlers/createTopics.js
+++ b/services/topics/handlers/createTopics.js
@@ -1,5 +1,4 @@
 const topics = require("../libs/topics-lib.js");
-import _ from "lodash";
 
 const condensedTopicList = [
   {
@@ -11,10 +10,13 @@ const condensedTopicList = [
     topics: [
       ".mcpar-reports",
       ".mcpar-form",
+      ".mcpar-form-template",
       ".mlr-reports",
       ".mlr-form",
+      ".mlr-form-template",
       ".naaar-reports",
       ".naaar-form",
+      ".naaar-form-template",
     ],
   },
 ];
@@ -32,13 +34,11 @@ exports.handler = async function (event, _context, _callback) {
   // Generate the complete topic list from the condensed version above.
   for (var element of condensedTopicList) {
     topicList.push(
-      ..._.map(element.topics, (topic) => {
-        return {
-          topic: `${element.topicPrefix}${topic}${element.version}`,
-          numPartitions: element.numPartitions,
-          replicationFactor: element.replicationFactor,
-        };
-      })
+      ...element.topics.map((topic) => ({
+        topic: `${element.topicPrefix}${topic}${element.version}`,
+        numPartitions: element.numPartitions,
+        replicationFactor: element.replicationFactor,
+      }))
     );
   }
 


Reviews:
user: login: karla-vm
body: LGTM! Admittedly, Kafka is not my forte, so I'd love some extra eyes from @BearHanded if possible 🙏🏽 
state: DISMISSED
author_association: COLLABORATOR
submitted_at: 2025-03-18T19:15:17Z
user: login: BearHanded
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T21:04:05Z
user: login: britt-mo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T13:59:59Z
user: login: BearHanded
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-19T14:44:30Z
user: login: jessabean
body: **boop:**
<img width="920" alt="Screenshot 2025-03-20 at 5 11 52 PM" src="https://github.com/user-attachments/assets/c3286616-6e8d-4ab1-8091-71db2a6b7f05" />

**and boop:**
<img width="907" alt="Screenshot 2025-03-20 at 5 12 55 PM" src="https://github.com/user-attachments/assets/dd9e0976-da2b-4bff-9ada-941162d0742d" />

Nice job!
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T21:13:50Z
user: login: BearHanded
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T13:21:58Z
user: login: gmrabian
body: witnessed this working in pair session 👏 
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T13:51:52Z
user: login: gmrabian
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T14:22:58Z
user: login: BearHanded
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T14:35:44Z

Review comments:
diff_hunk: @@ -14,9 +14,36 @@ const tables = [
   { sourceName: reportTables.NAAAR, topicName: tableTopics.NAAAR },
 ];
 const buckets = [
-  { sourceName: reportBuckets.MCPAR, topicName: bucketTopics.MCPAR },
-  { sourceName: reportBuckets.MLR, topicName: bucketTopics.MLR },
-  { sourceName: reportBuckets.NAAAR, topicName: bucketTopics.NAAAR },
+  {
+    sourceName: reportBuckets.MCPAR,
+    topicName: bucketTopics.MCPAR,
+    s3Prefix: "fieldData/",
path: services/app-api/handlers/kafka/post/postKafkaData.ts
user: login: BearHanded
body: This preps for the handling, but KafkaSourceLib has to then handle the filtering you provide it. You'll need to update that library.
created_at: 2025-03-18T21:03:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -17,6 +17,7 @@ type KafkaPayload = {
 type SourceTopicMapping = {
   sourceName: string;
   topicName: string;
+  s3Prefix?: string;
 };
path: services/app-api/utils/kafka/kafka-source-lib.ts
user: login: britt-mo
body: @BearHanded like so?
created_at: 2025-03-19T13:59:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -17,6 +17,7 @@ type KafkaPayload = {
 type SourceTopicMapping = {
   sourceName: string;
   topicName: string;
+  s3Prefix?: string;
 };
path: services/app-api/utils/kafka/kafka-source-lib.ts
user: login: BearHanded
body: The chunk down around line 113, determineS3TopicName will need to make use of that information as well. It will need to determine the topic name based on both teh source & prefix, if a prefix exists.
created_at: 2025-03-19T14:44:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -220,6 +225,7 @@ class KafkaSourceLib {
     }
 
     // if dynamo
+    console.log("event.Record:", event.Record);
path: services/app-api/utils/kafka/kafka-source-lib.ts
user: login: BearHanded
body: might want to remove the log now that you've proven it
created_at: 2025-03-21T13:21:58Z
author_association: CONTRIBUTOR

PR comments:
user: login: jessabean
created_at: 2025-03-18T21:01:16Z
body: - Can you elaborate on where to see the topics created in the logs?
- I think one of the acceptance criteria is to verify that the message content is being sent. Have you verified that the form template data JSON is being sent via the `postMlrBucketData` handler, etc? I created a report on the deployed URL for this branch and checked the logs for `postMlrBucketData` and I can verify that I'm seeing the data for the `.mlr-form` topic, but I can't seem to find `.mlr-form-template` data. Not sure if I am looking in the right place!

<img width="1512" alt="Screenshot 2025-03-18 at 4 55 13 PM" src="https://github.com/user-attachments/assets/2aa6b518-ffd9-419d-8614-c496d66c4aa5" />


author_association: CONTRIBUTOR
user: login: jessabean
created_at: 2025-03-18T21:24:50Z
body: I think when this is correct, in the logs we should see two events (is that the right word? I do not know):

**the form data**

```
{
    "topic": "--mcr--cmdct-4419--aws.mdct.mcr.mlr-form.v0",
    "messages": [
        {
            "key": "fieldData/MN/2uVPBnRAldAnBPXXQpyzW4z05MP.json",
            "value": "{\"versionControl\":[{\"key\":\"versionControl-KFCd3rfEu3eT4UFskUhDtx\",\"value\":\"No, this is an initial submission\"}],\"stateName\":\"Minnesota\"}",
            ...all the other stuff
        }
    ]
}
```

**the form template data (we do not see this yet)**

```

{
    "topic": "--mcr--cmdct-4419--aws.mdct.mcr.mlr-form-template.v0",
    "messages": [
        {
            "key": "formTemplates/2uUmEq4vP0guI6Z2b8Lu1nghu3Z.json",
            "value": "{ \"type\": \"MLR\", \"name\": \"MLR Report Submission Form\",\"entities\": { blah blah blah},\"routes\": [blah blah blah],\"validationJson\": {blah blah}}",
            ...all the other stuff
        }
    ]
}
```
author_association: CONTRIBUTOR
user: login: BearHanded
created_at: 2025-03-21T13:25:27Z
body: Looks good! The test log can probably be cleaned up afterwards.

This will get new form templates flowing through the system when an insert or update triggers. I think you'll need a script to make updates to any old versions you want to send though, causing an update trigger and sending it across. Messages for the past events have already been consumed by the filter.
author_association: CONTRIBUTOR
user: login: qlty-cloud-legacy[bot]
created_at: 2025-03-21T13:58:22Z
body: Code Climate has analyzed commit c1059939 and detected **0 issues** on this pull request.

The test coverage on the diff in this pull request is **100.0%** (**90%** is the threshold).

This pull request will bring the total coverage in the repository to **96.9%** (**0.0%** change).

View more on [Code Climate](https://codeclimate.com/repos/644971a20bd7db00dfeaf21a/pull/12077).

author_association: NONE