Repository: pharmaverse/aNCA
Pull request #238: Enhancement: move filters section
Author: login: js3110
Created: 2025-02-28T08:17:05Z | Merged: 2025-03-13T14:58:41Z

## Issue

Closes #230 

## Description

Move filtering section to before column mapping, so the user can filter with raw data instead.

## Definition of Done

The requirements for the feature to be complete:
- [x] Filters in data upload section
- [x] User can filter for values that are then applied to the rest of the analysis

## How to test

Add filters at the start and check if the rest of the app works.

## Contributor checklist
- [x] Code passes lintr checks
- [x] Code passes all unit tests
- [x] New logic covered by unit tests
- [x] New logic is documented
- [x] Package version is incremented

## Notes to reviewer

Anything that the reviewer should know before tacking the pull request?


Linked issue #230: Enhancement: GUI: move filters section
### Description
Move filtering section to before column mapping, so the user can filter with raw data instead.

### Definition of Done
The requirements for the feature to be complete:
- [ ] Filters in data upload section
- [x] User can filter for values that are then applied to the rest of the analysis


Diff:
diff --git a/DESCRIPTION b/DESCRIPTION
index 122c7d286..beb13ebd4 100644
--- a/DESCRIPTION
+++ b/DESCRIPTION
@@ -1,6 +1,6 @@
 Package: aNCA
 Title: (Pre-)Clinical NCA in a Dynamic Shiny App
-Version: 0.0.0.9025
+Version: 0.0.0.9026
 Authors@R: c(
     person("Ercan", "Suekuer", , "ercan.suekuer@roche.com", role = c("aut", "cre"),
            comment = c(ORCID = "0009-0001-1626-1526")),
diff --git a/inst/shiny/modules/slope_selector.R b/inst/shiny/modules/slope_selector.R
index ae2ba5577..80663950d 100644
--- a/inst/shiny/modules/slope_selector.R
+++ b/inst/shiny/modules/slope_selector.R
@@ -192,7 +192,7 @@ slope_selector_server <- function(
                         unname(unlist(mydata()$conc$columns$groups)),
                         "DOSNO"))) %>%
         filter(USUBJID %in% search_patient) %>%
-        select(slopes_groups()) %>%
+        select(slopes_groups(), USUBJID) %>%
         unique() %>%
         arrange(USUBJID)
 
diff --git a/inst/shiny/modules/tab_data.R b/inst/shiny/modules/tab_data.R
index a3ebf5401..592fe5d98 100644
--- a/inst/shiny/modules/tab_data.R
+++ b/inst/shiny/modules/tab_data.R
@@ -22,14 +22,6 @@ tab_data_ui <- function(id) {
           placeholder = ".csv",
           buttonLabel = list(icon("folder"), "Upload File..."),
           accept = c(".csv", ".rds")
-        )
-      ),
-      reactableOutput(ns("filecontents"))
-    ),
-    nav_panel("Mapping and Filters",
-      layout_columns(
-        card(
-          column_mapping_ui(ns("column_mapping"))
         ),
         card(
           div(
@@ -46,6 +38,14 @@ tab_data_ui <- function(id) {
             )
           )
         )
+      ),
+      reactableOutput(ns("filecontents"))
+    ),
+    nav_panel("Column Mapping",
+      layout_columns(
+        card(
+          column_mapping_ui(ns("column_mapping"))
+        )
       )
     ),
     nav_panel("Review Data",
@@ -87,10 +87,34 @@ tab_data_server <- function(id) {
       ADNCA(new_adnca)
     })
 
+    # Handle user-provided filters
+    filters <- reactiveValues()
+
+    observeEvent(input$add_filter, {
+      # Create a unique ID for each filter
+      filter_id <- paste0("filter_", input$add_filter)
+
+      # Insert a new filter UI
+      insertUI(
+        selector = paste0("#", session$ns("filters")),
+        ui = input_filter_ui(session$ns(filter_id), colnames(ADNCA()))
+      )
+
+      filters[[filter_id]] <- input_filter_server(filter_id)
+    })
+
+    adnca_filtered <- reactive({
+      # Extract filters from reactive values
+      applied_filters <- lapply(reactiveValuesToList(filters), \(x) x())
+
+      # Filter and return data
+      apply_filters(ADNCA(), applied_filters)
+    }) |> bindEvent(input$submit_filters, processed_data())
+
     output$filecontents <- renderReactable({
-      req(ADNCA())
+      req(adnca_filtered())
       reactable(
-        ADNCA(),
+        adnca_filtered(),
         searchable = TRUE,
         sortable = TRUE,
         highlight = TRUE,
@@ -126,7 +150,7 @@ tab_data_server <- function(id) {
     # Call the column mapping module
     column_mapping <- column_mapping_server(
       id = "column_mapping",
-      data = ADNCA,
+      data = adnca_filtered,
       manual_units = manual_units,
       on_submit = change_to_review_tab
     )
@@ -135,12 +159,12 @@ tab_data_server <- function(id) {
       if (!is.null(processed_data()) && nrow(processed_data()) > 0) {
         tagList(
           "This is the data set that will be used for the analysis.
-          If you would like to make any changes please return to the 'Mapping and Filters' tab.",
+          If you would like to make any changes please return to the previous tabs.",
           reactableOutput(ns("data_processed"))
         )
       } else {
         div(
-          "Please map your data in the 'Mapping and Filters' section before reviewing it."
+          "Please map your data in the 'Column Mapping' section before reviewing it."
         )
       }
     })
@@ -151,44 +175,15 @@ tab_data_server <- function(id) {
     # Global variable to store grouping variables
     grouping_variables <- column_mapping$grouping_variables
 
-    # Handle user-provided filters
-    filters <- reactiveValues()
-
-    observeEvent(input$add_filter, {
-      # Create a unique ID for each filter
-      filter_id <- paste0("filter_", input$add_filter)
-
-      # Insert a new filter UI
-      insertUI(
-        selector = paste0("#", session$ns("filters")),
-        ui = input_filter_ui(session$ns(filter_id), colnames(processed_data()))
-      )
-
-      filters[[filter_id]] <- input_filter_server(filter_id)
-    })
-
-    # Create reactive value with applied filters
-    data <- reactiveVal(NULL)
-    observeEvent(list(input$submit_filters, processed_data()), {
-      # Extract filters from reactive values
-      applied_filters <- lapply(reactiveValuesToList(filters), \(x) x())
-
-      # Filter and overwrite data
-      filtered_data <- apply_filters(
-        processed_data(), applied_filters
-      )
-      data(filtered_data)
-    }, ignoreInit = FALSE)
-
     # Update the data table object with the filtered data
     output$data_processed <- renderReactable({
-      req(data())
+      req(processed_data())
 
       # Generate column definitions
-      col_defs <- generate_col_defs(data())
+      col_defs <- generate_col_defs(processed_data())
 
       reactable(
-        data(),
+        processed_data(),
         columns = col_defs,
         searchable = TRUE,
         sortable = TRUE,
@@ -204,7 +199,7 @@ tab_data_server <- function(id) {
     })
 
     list(
-      data = data,
+      data = processed_data,
       grouping_variables = grouping_variables
     )
   })
diff --git a/inst/shiny/modules/tab_nca.R b/inst/shiny/modules/tab_nca.R
index a6c25f13e..8f339f777 100644
--- a/inst/shiny/modules/tab_nca.R
+++ b/inst/shiny/modules/tab_nca.R
@@ -113,9 +113,21 @@ tab_nca_server <- function(id, data, grouping_vars) {
         duration = "ADOSEDUR"
       )
 
+      #create basic intervals so that PKNCAdata can be created
+      #TODO: investigate if this is required
+      intervals <-
+        data.frame(
+          start = 0, end = Inf,
+          cmax = TRUE,
+          tmax = TRUE,
+          auclast = FALSE,
+          aucinf.obs = FALSE
+        )
+
       mydata <- PKNCA::PKNCAdata(
         data.conc = myconc,
         data.dose = mydose,
+        intervals = intervals,
         units = PKNCA::pknca_units_table(
           concu = myconc$data$AVALU[1],
           doseu = myconc$data$DOSEU[1],


Reviews:
user: login: m-kolomanski
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-03-03T06:48:08Z
user: login: js3110
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-11T15:00:38Z
user: login: js3110
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-11T15:02:57Z
user: login: m-kolomanski
body: Looks good, thanks for the adjustments!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-11T15:45:09Z
user: login: Gero1999
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-03-12T11:48:29Z
user: login: js3110
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-13T09:27:36Z
user: login: Gero1999
body: Hey @js3110, I see the error now. Instead of a temporary fix, I suggest we find and address the root cause and open a new issue for it.

I will approve the pr, but please mark the relevant code lines with a #ToDo. The error most likely originates from the dataset or the `format_pkncadose_data()` function when creating the dose dataset for one subject. It's important to identify when and why this happens and open an issue for it.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-13T13:05:23Z

Review comments:
path: inst/shiny/modules/tab_nca.R
user: login: m-kolomanski
body: question: This change seems unrelated, should it be here?
created_at: 2025-03-03T06:17:57Z
author_association: COLLABORATOR
diff_hunk: @@ -87,10 +87,39 @@ tab_data_server <- function(id) {
       ADNCA(new_adnca)
     })
 
+    adnca_filter <- reactiveVal(NULL)
path: inst/shiny/modules/tab_data.R
user: login: m-kolomanski
body: issue: I think the name of this variable should be something like `adnca_filtered`. The word `filter` here is a bit confusing.
created_at: 2025-03-03T06:20:12Z
author_association: COLLABORATOR
diff_hunk: @@ -87,10 +87,39 @@ tab_data_server <- function(id) {
       ADNCA(new_adnca)
     })
 
+    adnca_filter <- reactiveVal(NULL)
+    # Handle user-provided filters
+    filters <- reactiveValues()
+
+    observeEvent(input$add_filter, {
+      # Create a unique ID for each filter
+      filter_id <- paste0("filter_", input$add_filter)
+
+      # Insert a new filter UI
+      insertUI(
+        selector = paste0("#", session$ns("filters")),
+        ui = input_filter_ui(session$ns(filter_id), colnames(ADNCA()))
+      )
+
+      filters[[filter_id]] <- input_filter_server(filter_id)
+    })
+
+    # Create reactive value with applied filters
+    observeEvent(list(input$submit_filters, processed_data()), {
+      # Extract filters from reactive values
+      applied_filters <- lapply(reactiveValuesToList(filters), \(x) x())
+
+      # Filter and overwrite data
+      filtered_data <- apply_filters(
+        ADNCA(), applied_filters
+      )
+      adnca_filter(filtered_data)
+    }, ignoreInit = FALSE)
path: inst/shiny/modules/tab_data.R
user: login: m-kolomanski
body: 1. issue: Seems like there is no need for `adnca_filter` to be a `reactiveVal`, could be a `reactiveEvent` or `reactive(...) |> bindEvent()`
2. question: With this setup, when column mapping is applied, filters will also be auto-applied, without explicit user confirmation. Is that a desired behavior?
created_at: 2025-03-03T06:43:43Z
author_association: COLLABORATOR
path: inst/shiny/modules/tab_nca.R
user: login: js3110
body: i think that i was getting a crash because there were no intervals.. not sure how it was related to changing the filtering but it seemed to happen only after I moved the filters section
created_at: 2025-03-11T15:00:38Z
author_association: COLLABORATOR
diff_hunk: @@ -87,10 +87,39 @@ tab_data_server <- function(id) {
       ADNCA(new_adnca)
     })
 
+    adnca_filter <- reactiveVal(NULL)
+    # Handle user-provided filters
+    filters <- reactiveValues()
+
+    observeEvent(input$add_filter, {
+      # Create a unique ID for each filter
+      filter_id <- paste0("filter_", input$add_filter)
+
+      # Insert a new filter UI
+      insertUI(
+        selector = paste0("#", session$ns("filters")),
+        ui = input_filter_ui(session$ns(filter_id), colnames(ADNCA()))
+      )
+
+      filters[[filter_id]] <- input_filter_server(filter_id)
+    })
+
+    # Create reactive value with applied filters
+    observeEvent(list(input$submit_filters, processed_data()), {
+      # Extract filters from reactive values
+      applied_filters <- lapply(reactiveValuesToList(filters), \(x) x())
+
+      # Filter and overwrite data
+      filtered_data <- apply_filters(
+        ADNCA(), applied_filters
+      )
+      adnca_filter(filtered_data)
+    }, ignoreInit = FALSE)
path: inst/shiny/modules/tab_data.R
user: login: js3110
body: yes, the idea is that filtering is done beforehand, and then applied to the rest of the analysis
created_at: 2025-03-11T15:02:57Z
author_association: COLLABORATOR
diff_hunk: @@ -113,9 +113,20 @@ tab_nca_server <- function(id, data, grouping_vars) {
         duration = "ADOSEDUR"
       )
 
+      #create basic intervals so that PKNCAdata can be created
+      intervals <-
+        data.frame(
+          start = 0, end = Inf,
+          cmax = TRUE,
+          tmax = TRUE,
+          auclast = FALSE,
+          aucinf.obs = FALSE
+        )
+
       mydata <- PKNCA::PKNCAdata(
         data.conc = myconc,
         data.dose = mydose,
+        intervals = intervals,
path: inst/shiny/modules/tab_nca.R
user: login: Gero1999
body: issue: Hello Jana! Thanks for the changes I am just reluctant to this. I tested the branch taking it off and everything seems to keep working well. If it does not for you, let me know under what specific filter/actions the crush occurs
created_at: 2025-03-12T11:47:16Z
author_association: COLLABORATOR
diff_hunk: @@ -113,9 +113,20 @@ tab_nca_server <- function(id, data, grouping_vars) {
         duration = "ADOSEDUR"
       )
 
+      #create basic intervals so that PKNCAdata can be created
+      intervals <-
+        data.frame(
+          start = 0, end = Inf,
+          cmax = TRUE,
+          tmax = TRUE,
+          auclast = FALSE,
+          aucinf.obs = FALSE
+        )
+
       mydata <- PKNCA::PKNCAdata(
         data.conc = myconc,
         data.dose = mydose,
+        intervals = intervals,
path: inst/shiny/modules/tab_nca.R
user: login: js3110
body: @Gero1999 I went back to check, and its if you filter for only 1 subject, the app crashes, with the error: 
`Warning: Error in check.interval.specification: interval specification has no rows`

Also, with this code, we no longer get all the warning messages that say `Warning in PKNCAdata.default(data.conc = data.conc, data.dose = data.dose,  :
  STUDYID=XX01; PCSPEC=SERUM; DRUG=Analyte01; USUBJID=11201; ANALYTE=Analyte01; data_intervals=NULL: No intervals generated likely due to limited concentration data` etc

Let me know what you think! if theres another way to avoid this error that you prefer to use then we can try that
created_at: 2025-03-13T09:27:36Z
author_association: COLLABORATOR

PR comments:
user: login: js3110
created_at: 2025-03-13T13:09:42Z
body: > Hey @js3110, I see the error now. Instead of a temporary fix, I suggest we find and address the root cause and open a new issue for it.
> 
> I will approve the pr, but please mark the relevant code lines with a #ToDo. The error most likely originates from the dataset or the `format_pkncadose_data()` function when creating the dose dataset for one subject. It's important to identify when and why this happens and open an issue for it.

Ok thanks! will do, although are you sure this is a temporary fix? the issue is due to there being no intervals set, so setting intervals is the fix, no?
author_association: COLLABORATOR
user: login: Gero1999
created_at: 2025-03-13T13:19:18Z
body: I am concerned that we could be just hiding the issue. The `intervals` is not a mandatory arg and `PKNCAdata` by default should be generating intervals based on the dose data (even if we don't currently use them in the app). I think is worthy we investigate it more at some point
author_association: COLLABORATOR