Repository: CDCgov/prime-reportstream
Pull request #17596: Add CSV header check for code condition mapping endpoint
Author: login: the-andrew
Created: 2025-03-18T20:19:52Z | Merged: 2025-03-21T19:36:23Z

This PR adds a check on the backend for the CSV received in the request body for the code condition mapping endpoint.
The CSV header is checked to make sure the 3 required columns are present.
Expected headers are:
- `test code`
- `test description`
- `coding system`

Test Steps:
1. Send HTTP request to `/api/sender/conditionCode/comparison` with CSV in request body with all 3 required headers
2. Send same request but with one or more headers missing

## Changes
- Slight change on backend to check request body CSV's headers against the list of 3 required headers

## Checklist

### Testing
- [ ] Tested locally?
- [ ] Ran `./prime test` or `./gradlew testSmoke` against local Docker ReportStream container?
- [ ] (For Changes to /frontend-react/...) Ran `npm run lint:write`?
- [ ] Added tests?

### Process
- [ ] Are there licensing issues with any new dependencies introduced?
- [ ] Includes a summary of what a code reviewer should test/verify?
- [ ] Updated the release notes?
- [ ] Database changes are submitted as a separate PR?
- [ ] DevOps team has been notified if PR requires ops support?

## Linked Issues
- Fixes #issue

## To Be Done
*Create GitHub issues to track the work remaining, if any*
- #issue

## Specific Security-related subjects a reviewer should pay specific attention to
- Does this PR introduce new endpoints?
    - new endpoint A
    - new endpoint B
- Does this PR include changes in authentication and/or authorization of existing endpoints?
- Does this change introduce new dependencies that need vetting?
- Does this change require changes to our infrastructure?
- Does logging contain sensitive data?
- Does this PR include or remove any sensitive information itself?

If you answered '_yes_' to any of the questions above, conduct a detailed Review that addresses at least:

- What are the potential security threats and mitigations? Please list the _STRIDE_ threats and how they are mitigated
    - **S**poofing (faking authenticity)
        - Threat _T_, which could be achieved by _A_, is mitigated by _M_
    - **T**ampering (influence or sabotage the integrity of information, data, or system)
    - **R**epudiation (the ability to dispute the origin or originator of an action)
    - **I**nformation disclosure (data made available to entities who should not have it)
    - **D**enial of service (make a resource unavailable)
    - **E**levation of Privilege (reduce restrictions that apply or gain privileges one should not have)
- Have you ensured logging does not contain sensitive data?
- Have you received any additional approvals needed for this change?



Diff:
diff --git a/prime-router/src/main/kotlin/azure/SenderFunction.kt b/prime-router/src/main/kotlin/azure/SenderFunction.kt
index 56bc29408a4..a90c31c4961 100644
--- a/prime-router/src/main/kotlin/azure/SenderFunction.kt
+++ b/prime-router/src/main/kotlin/azure/SenderFunction.kt
@@ -2,6 +2,9 @@ package gov.cdc.prime.router.azure
 
 import com.fasterxml.jackson.databind.ObjectMapper
 import com.github.doyaaaaaken.kotlincsv.dsl.csvReader
+import com.github.doyaaaaaken.kotlincsv.util.CSVFieldNumDifferentException
+import com.github.doyaaaaaken.kotlincsv.util.CSVParseFormatException
+import com.github.doyaaaaaken.kotlincsv.util.MalformedCSVException
 import com.microsoft.azure.functions.HttpMethod
 import com.microsoft.azure.functions.HttpRequestMessage
 import com.microsoft.azure.functions.HttpResponseMessage
@@ -56,7 +59,39 @@ class SenderFunction(
 
             // Read request body CSV
             val bodyCsvText = request.body ?: ""
+
+            // Check CSV headers in request body
+            val csvHeader = bodyCsvText.lineSequence().firstOrNull()
+            // 400 if CSV is empty
+            if (csvHeader.isNullOrBlank()) {
+                return HttpUtilities.bad(request, "CSV file is empty")
+            }
+            val requiredHeaders = setOf("test code", "test description", "coding system")
+            val headersToCheck = csvHeader.split(",").toSet()
+            val missingHeaders = requiredHeaders.minus(headersToCheck)
+
+            // 400 with list of missing headers if any
+            if (missingHeaders.isNotEmpty()) {
+                return HttpUtilities.bad(
+                    request,
+                    "CSV file is missing the following header column(s): '$missingHeaders'"
+                )
+            }
+
             val bodyCsv = csvReader().readAllWithHeader(bodyCsvText)
+            // 400 if CSV contains no data rows
+            if (bodyCsv.isEmpty()) {
+                return HttpUtilities.bad(request, "CSV file contains no rows of data")
+            } else {
+                // Find rows that have columns with empty values
+                val rowsWithEmptyValues = bodyCsv.filter { row ->
+                    row.filter { (_, v) -> v.isEmpty() }.isNotEmpty()
+                }
+                // 400 if there are rows with columns with empty values
+                if (rowsWithEmptyValues.isNotEmpty()) {
+                    return HttpUtilities.bad(request, "CSV file contains rows with empty values")
+                }
+            }
 
             // Get observation mapping table
             val tableMapper = LookupTableConditionMapper(workflowEngine.metadata)
@@ -70,9 +105,8 @@ class SenderFunction(
 
             // Create output JSON with mapping comparison result
             // For this output specifically, rename the keys for the JSON output
-            val comparisonsRemappedKeys = mutableListOf<Map<String, String>>()
-            for (entry in conditionCodeComparison) {
-                val replacedMap = entry.map { (key, value) ->
+            val comparisonsRemappedKeys = conditionCodeComparison.map { entry ->
+                entry.map { (key, value) ->
                     when (key) {
                         "test code" -> "testCode" to value
                         "test description" -> "testDescription" to value
@@ -81,18 +115,25 @@ class SenderFunction(
                         else -> key to value
                     }
                 }.toMap()
-                comparisonsRemappedKeys.add(replacedMap)
             }
             val conditionCodeComparisonJson = ObjectMapper().writeValueAsString(comparisonsRemappedKeys)
 
             return HttpUtilities.okResponse(request, conditionCodeComparisonJson)
         } catch (ex: Exception) {
-            if (ex.message != null) {
-                logger.error(ex.message!!, ex)
-            } else {
-                logger.error(ex)
+            return when (ex) {
+                is CSVFieldNumDifferentException -> {
+                    logger.error(ex)
+                    HttpUtilities.bad(request, "CSV file contains rows with missing data")
+                }
+                is CSVParseFormatException, is MalformedCSVException -> {
+                    logger.error(ex)
+                    HttpUtilities.bad(request, "Error parsing CSV")
+                }
+                else -> {
+                    logger.error(ex)
+                    HttpUtilities.internalErrorResponse(request)
+                }
             }
-            return HttpUtilities.internalErrorResponse(request)
         }
     }
 }
\ No newline at end of file
diff --git a/prime-router/src/test/kotlin/azure/SenderFunctionTest.kt b/prime-router/src/test/kotlin/azure/SenderFunctionTest.kt
index 4e6ec22b815..789891485ad 100644
--- a/prime-router/src/test/kotlin/azure/SenderFunctionTest.kt
+++ b/prime-router/src/test/kotlin/azure/SenderFunctionTest.kt
@@ -47,6 +47,22 @@ class SenderFunctionTest {
         "80382-5,Influenza virus A Ag [Presence] in Upper respiratory specimen by Rapid immunoassay,LOINC\n" +
         "12345,Flu B,LOCAL"
 
+    val REQ_BODY_TEST_CSV_EMPTY = ""
+
+    val REQ_BODY_TEST_CSV_HEADERS_ONLY = "test code,test description,coding system"
+
+    val REQ_BODY_TEST_CSV_MISSING_HEADER = "test code,test description,coding syste"
+
+    val REQ_BODY_TEST_CSV_MISSING_VALUE = "test code,test description,coding system\n" +
+        "97097-0,SARS-CoV-2 (COVID-19) Ag [Presence] in Upper respiratory specimen by Rapid  immunoassay,LOINC\n" +
+        "80382-5,Influenza virus A Ag [Presence] in Upper respiratory specimen by Rapid immunoassay,LOINC\n" +
+        "12345,Flu B"
+
+    val REQ_BODY_TEST_CSV_WITH_EMPTY_VALUE = "test code,test description,coding system\n" +
+        "97097-0,SARS-CoV-2 (COVID-19) Ag [Presence] in Upper respiratory specimen by Rapid  immunoassay,LOINC\n" +
+        "80382-5,Influenza virus A Ag [Presence] in Upper respiratory specimen by Rapid immunoassay,LOINC\n" +
+        "12345,,LOCAL"
+
     val testOrganization = DeepOrganization(
         "phd",
         "test",
@@ -152,6 +168,176 @@ class SenderFunctionTest {
         assertEquals(HttpStatus.OK, response.status)
     }
 
+    @Test
+    fun `test SenderFunction conditionCodeComparisonPostRequest empty csv`() {
+        val metadata = UnitTestUtils.simpleMetadata
+        val settings = FileSettings().loadOrganizations(testOrganization)
+        val sender = UniversalPipelineSender(
+            name = "Test Sender",
+            organizationName = "testOrganization",
+            format = MimeFormat.HL7,
+            topic = Topic.FULL_ELR
+        )
+
+        val workflowEngine = makeEngine(metadata, settings)
+        val actionHistory = spyk(ActionHistory(TaskAction.receive))
+        val senderFunction = spyk(SenderFunction(workflowEngine, actionHistory))
+
+        val testRequest = MockHttpRequestMessage(REQ_BODY_TEST_CSV_EMPTY)
+
+        every { workflowEngine.settings.findSender("Test Sender") } returns sender
+
+        mockkObject(AuthenticatedClaims)
+        val mockClaims = mockk<AuthenticatedClaims>()
+        every { AuthenticatedClaims.authenticate(any()) } returns mockClaims
+        every { mockClaims.authorizedForSendOrReceive(any(), any()) } returns true
+
+        testRequest.httpHeaders += mapOf(
+            "client" to "Test Sender",
+            "content-length" to "4"
+        )
+
+        val response = senderFunction.conditionCodeComparisonPostRequest(testRequest)
+
+        assertEquals(HttpStatus.BAD_REQUEST, response.status)
+    }
+
+    @Test
+    fun `test SenderFunction conditionCodeComparisonPostRequest csv headers only`() {
+        val metadata = UnitTestUtils.simpleMetadata
+        val settings = FileSettings().loadOrganizations(testOrganization)
+        val sender = UniversalPipelineSender(
+            name = "Test Sender",
+            organizationName = "testOrganization",
+            format = MimeFormat.HL7,
+            topic = Topic.FULL_ELR
+        )
+
+        val workflowEngine = makeEngine(metadata, settings)
+        val actionHistory = spyk(ActionHistory(TaskAction.receive))
+        val senderFunction = spyk(SenderFunction(workflowEngine, actionHistory))
+
+        val testRequest = MockHttpRequestMessage(REQ_BODY_TEST_CSV_HEADERS_ONLY)
+
+        every { workflowEngine.settings.findSender("Test Sender") } returns sender
+
+        mockkObject(AuthenticatedClaims)
+        val mockClaims = mockk<AuthenticatedClaims>()
+        every { AuthenticatedClaims.authenticate(any()) } returns mockClaims
+        every { mockClaims.authorizedForSendOrReceive(any(), any()) } returns true
+
+        testRequest.httpHeaders += mapOf(
+            "client" to "Test Sender",
+            "content-length" to "4"
+        )
+
+        val response = senderFunction.conditionCodeComparisonPostRequest(testRequest)
+
+        assertEquals(HttpStatus.BAD_REQUEST, response.status)
+    }
+
+    @Test
+    fun `test SenderFunction conditionCodeComparisonPostRequest csv with missing header`() {
+        val metadata = UnitTestUtils.simpleMetadata
+        val settings = FileSettings().loadOrganizations(testOrganization)
+        val sender = UniversalPipelineSender(
+            name = "Test Sender",
+            organizationName = "testOrganization",
+            format = MimeFormat.HL7,
+            topic = Topic.FULL_ELR
+        )
+
+        val workflowEngine = makeEngine(metadata, settings)
+        val actionHistory = spyk(ActionHistory(TaskAction.receive))
+        val senderFunction = spyk(SenderFunction(workflowEngine, actionHistory))
+
+        val testRequest = MockHttpRequestMessage(REQ_BODY_TEST_CSV_MISSING_HEADER)
+
+        every { workflowEngine.settings.findSender("Test Sender") } returns sender
+
+        mockkObject(AuthenticatedClaims)
+        val mockClaims = mockk<AuthenticatedClaims>()
+        every { AuthenticatedClaims.authenticate(any()) } returns mockClaims
+        every { mockClaims.authorizedForSendOrReceive(any(), any()) } returns true
+
+        testRequest.httpHeaders += mapOf(
+            "client" to "Test Sender",
+            "content-length" to "4"
+        )
+
+        val response = senderFunction.conditionCodeComparisonPostRequest(testRequest)
+
+        assertEquals(HttpStatus.BAD_REQUEST, response.status)
+    }
+
+    @Test
+    fun `test SenderFunction conditionCodeComparisonPostRequest csv with missing values`() {
+        val metadata = UnitTestUtils.simpleMetadata
+        val settings = FileSettings().loadOrganizations(testOrganization)
+        val sender = UniversalPipelineSender(
+            name = "Test Sender",
+            organizationName = "testOrganization",
+            format = MimeFormat.HL7,
+            topic = Topic.FULL_ELR
+        )
+
+        val workflowEngine = makeEngine(metadata, settings)
+        val actionHistory = spyk(ActionHistory(TaskAction.receive))
+        val senderFunction = spyk(SenderFunction(workflowEngine, actionHistory))
+
+        val testRequest = MockHttpRequestMessage(REQ_BODY_TEST_CSV_MISSING_VALUE)
+
+        every { workflowEngine.settings.findSender("Test Sender") } returns sender
+
+        mockkObject(AuthenticatedClaims)
+        val mockClaims = mockk<AuthenticatedClaims>()
+        every { AuthenticatedClaims.authenticate(any()) } returns mockClaims
+        every { mockClaims.authorizedForSendOrReceive(any(), any()) } returns true
+
+        testRequest.httpHeaders += mapOf(
+            "client" to "Test Sender",
+            "content-length" to "4"
+        )
+
+        val response = senderFunction.conditionCodeComparisonPostRequest(testRequest)
+
+        assertEquals(HttpStatus.BAD_REQUEST, response.status)
+    }
+
+    @Test
+    fun `test SenderFunction conditionCodeComparisonPostRequest csv with empty values`() {
+        val metadata = UnitTestUtils.simpleMetadata
+        val settings = FileSettings().loadOrganizations(testOrganization)
+        val sender = UniversalPipelineSender(
+            name = "Test Sender",
+            organizationName = "testOrganization",
+            format = MimeFormat.HL7,
+            topic = Topic.FULL_ELR
+        )
+
+        val workflowEngine = makeEngine(metadata, settings)
+        val actionHistory = spyk(ActionHistory(TaskAction.receive))
+        val senderFunction = spyk(SenderFunction(workflowEngine, actionHistory))
+
+        val testRequest = MockHttpRequestMessage(REQ_BODY_TEST_CSV_WITH_EMPTY_VALUE)
+
+        every { workflowEngine.settings.findSender("Test Sender") } returns sender
+
+        mockkObject(AuthenticatedClaims)
+        val mockClaims = mockk<AuthenticatedClaims>()
+        every { AuthenticatedClaims.authenticate(any()) } returns mockClaims
+        every { mockClaims.authorizedForSendOrReceive(any(), any()) } returns true
+
+        testRequest.httpHeaders += mapOf(
+            "client" to "Test Sender",
+            "content-length" to "4"
+        )
+
+        val response = senderFunction.conditionCodeComparisonPostRequest(testRequest)
+
+        assertEquals(HttpStatus.BAD_REQUEST, response.status)
+    }
+
     @Test
     fun `test SenderFunction conditionCodeComparisonPostRequest with no sender`() {
         val metadata = UnitTestUtils.simpleMetadata


Reviews:
user: login: etanb
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-18T22:55:13Z
user: login: victor-chaparro
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-19T15:05:20Z
user: login: victor-chaparro
body: Nice!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-21T14:16:59Z

Review comments:
diff_hunk: @@ -56,6 +56,25 @@ class SenderFunction(
 
             // Read request body CSV
             val bodyCsvText = request.body ?: ""
+
+            // Check CSV headers in request body
+            val csvHeader = bodyCsvText.lineSequence().firstOrNull()
+            if (csvHeader != null) {
+                val headersToCheck = csvHeader.split(",")
+
+                val missingHeaders = mutableListOf<String>()
+                val requiredHeaders = listOf("test code", "test description", "coding system")
+                for (header in requiredHeaders) {
path: prime-router/src/main/kotlin/azure/SenderFunction.kt
user: login: etanb
body: @victor-chaparro is this all the validation we need for this CSV? do we want have that list of hard coded header strings to leave here? we can also do this validation on the F/E as well, but either we could mimic this directly `"test code", "test description", "coding system"` or if we expose the required headers from the B/E than we don't have to maintain two sources of truth.
created_at: 2025-03-18T22:55:13Z
author_association: COLLABORATOR
diff_hunk: @@ -56,6 +56,25 @@ class SenderFunction(
 
             // Read request body CSV
             val bodyCsvText = request.body ?: ""
+
+            // Check CSV headers in request body
+            val csvHeader = bodyCsvText.lineSequence().firstOrNull()
+            if (csvHeader != null) {
+                val headersToCheck = csvHeader.split(",")
+
+                val missingHeaders = mutableListOf<String>()
+                val requiredHeaders = listOf("test code", "test description", "coding system")
+                for (header in requiredHeaders) {
path: prime-router/src/main/kotlin/azure/SenderFunction.kt
user: login: victor-chaparro
body: There are some other pieces  we should check, but I'm not sure if we're doing that already
Here are some things we should check:

- File does not have any empty rows
- Every row has a code, description and coding system
- Every row has the correct number of columns

I’d rather have all these validations handled on the backend. Since the backend needs to process the file anyway, it makes sense to avoid having the frontend parse through all its contents unnecessarily.

created_at: 2025-03-19T15:05:20Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-18T20:20:07Z
body: <h1>Dependency Review</h1>
✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.<h2>OpenSSF Scorecard</h2>
<table><tr><th>Package</th><th>Version</th><th>Score</th><th>Details</th></tr>
</table><h2>Scanned Manifest Files</h2>


<!-- dependency-review-pr-comment-marker -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-18T20:30:50Z
body: ## Test Results
1 292 tests  +5   1 288 :white_check_mark: +5   8m 4s :stopwatch: +30s
  168 suites ±0       4 :zzz: ±0 
  168 files   ±0       0 :x: ±0 

Results for commit 14828931. ± Comparison against base commit a8d11db5.

[test-results]:data:application/gzip;base64,H4sIAMG13WcC/13MTQ7CIBCG4as0rF0wgAhexiBlEmJbDD+rxrtLUVvi8n0m860E/eQSuQ4g1Wkgqfjc5ViiyT4sFYQSFeoxtzPT7Je3VKxtptRhD//cvnZA46cKdAcXY4hfiWXpVrf6G/3Qsdm6m2zdL9owzz7XICAUU5qDFhpHQ7mQgA6l45y58x0kdRSRwYW83kZI/QcNAQAA

:recycle: This comment has been updated with latest results.
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-18T20:34:54Z
body: ## Integration Test Results
 60 files  ±0   60 suites  ±0   43m 54s :stopwatch: + 1m 27s
428 tests ±0  418 :white_check_mark: ±0  10 :zzz: ±0  0 :x: ±0 
431 runs  ±0  421 :white_check_mark: ±0  10 :zzz: ±0  0 :x: ±0 

Results for commit 14828931. ± Comparison against base commit a8d11db5.

[test-results]:data:application/gzip;base64,H4sIAC693WcC/03MSw7CIBSF4a00jB1wARHcjEHKTYhtMTxGxr1LSS0M/+8k50PQLy6R+yTpZSKp+NxrLtFkH7baTHJRpY55nwVT/3qkYu1OMNDLvysBPQWNX6p0cDGGeEgsWzvlcMT5yboMlw2Gx9bjoQ3r6nMNAkIxpTlooXE2lAsJ6FA6zpm7PkFSRxEZ3Mj3B11qh/MKAQAA

:recycle: This comment has been updated with latest results.
author_association: CONTRIBUTOR
user: login: sonarqubecloud[bot]
created_at: 2025-03-21T19:03:49Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=CDCgov_prime-data-hub&pullRequest=17596) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [8 New issues](https://sonarcloud.io/project/issues?id=CDCgov_prime-data-hub&pullRequest=17596&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=CDCgov_prime-data-hub&pullRequest=17596&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=CDCgov_prime-data-hub&pullRequest=17596&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [90.4% Coverage on New Code](https://sonarcloud.io/component_measures?id=CDCgov_prime-data-hub&pullRequest=17596&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=CDCgov_prime-data-hub&pullRequest=17596&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=CDCgov_prime-data-hub&pullRequest=17596)


author_association: NONE