Repository: AlexsLemonade/scpca-paper-figures
Pull request #261: Update metadata parsing
Author: login: sjspielman
Created: 2025-04-11T19:11:07Z | Merged: 2025-04-14T18:01:03Z

Closes #259 (if pursued!)

This PR updates our approach for preparing metadata files for reproducibility, according to the scheme I laid out in my opening comment of #259.

To get the full library information we need, I also save and parse bulk metadata TSVs and metadata TSVs for projects with CITE-seq, and handle cellhash samples a bit more carefully to ensure we get their technology information. To get the full sample information we need, I manually add in a table with the information we need.

The result seems to sufficiently match our metadata files! I used these new files, for example, to re-create Table S1 and Table S2. No diff for Table S2 🎉, but for Table S1 these versions of the metadata led to one diff from the table made with our internal metadata - the diagnosis for SCPCP000009 are in a different order, but are all the same. This is presumably just a row order thing and I'm not worried about it (unless we want to sort somehow in general) since the table has the same content.

At some point while I was here too, I made a few other changes (which can be cherry-picked out if we prefer):
 a bit of rearranging of directory definitions (eg the merged sce directory) to place them all higher in the script to create them all at once. I also fixed an exciting bug where I had called the library metadata file `"library-sample-metadata.tsv"`, which is...not correct.

Linked issue #259: Update metadata parsing
While testing code towards #211 & #245, I encountered some issues/discrepancies with the metadata generation. In this issue, I'm laying out what I'm pretty sure is what we need to do to get the full data we need.

Currently, we rely on just the portal-wide metadata download, which isn't going to be comprehensive enough for a few reasons:
- It does not include an indicator for libraries with CITE-seq
- It does not include libraries that _only_ have bulk sequencing 

Therefore, we additionally need to obtain additional information from these files to complete the metadata:
- project-specific metadata files for CITE projects 
- bulk-specific metadata files

Moreover, there are four samples which are currently bulk-only that we need to include in the metadata files. Notably, for the sample metadata file that we need to recreate, we need information about `diagnosis`, `disease_timing`, and `is_cell_line`. However, none of this information appears to be obtainable from the portal:
- these samples are not in the portal-wide metadata download since they do not have single-cell libraries
- these samples are therefore not in the project-specific single-cell metadata either
- these samples _are_ in the bulk metadata tsvs, _but not with_ `diagnosis`, `disease_timing`, and `is_cell_line` columns to parse out.

Therefore, we may need to manually include these samples when creating the sample metadata tsv. 

(Note further, the BULK libraries from these samples are on the portal, but NOT any single-cell equivalent if it exists(?). So, it's not immediately clear to me if the fact that we have these annotated internally as `on_portal = TRUE` is correct?)

Bearing all this in mind, how should we update the parsing?

## Library metadata

- Combine information from bulk tsvs, citeseq project tsvs, and the portal-wide tsv
- Note that we cannot skip the portal-wide tsv here, since the project tsvs don't give all the information we need about about spatial libraries, but the portal-wide one does. 
  - For example, the sample `SCPCS000170` from `SCPCP000006` has _three_ rows in the portal-wide metadata, but only 1 row in the project-specific single cell representing the single-cell. So, the portal-wide seems necessary


## Sample data
- Use information from the portal-wide metadata, but _manually add in_ rows for the 4 samples which we can't get metadata for from the portal directly
Linked issue comments: user: login: allyhawkins
created_at: 2025-04-11T17:41:04Z
body: > (Note further, the BULK libraries from these samples are on the portal, but NOT any single-cell equivalent if it exists(?). So, it's not immediately clear to me if the fact that we have these annotated internally as on_portal = TRUE is correct?)

The annotation of `on_portal = TRUE` is correct. Those samples ONLY have a bulk library and do not have a single-cell library. They are available as part of the bulk download on the Portal and should be included in the `single_cell_metadata.tsv` for the entire project if you download with bulk. If that is not the case then that is a portal change that should be addressed. 

Here's how I might approach this issue: 

- Use our internal library/sample metadata files to create a whitelist of bulk and cite-seq library IDs and save that list to the repo
- create a table with the metadata for samples that are bulk only and therefore not included in the portal metadata (but are on the portal) and save that to the repo 
- use both of those things along with the portal wide metadata as input to a script that can be used by others to create the library and sample metadata that is required as input to create the figures

Another thought is that we should just create a simplified version of our internal sample and library metadata files and add it to this repo. To do that, we could delete any columns that we don't use and make sure that only libraries/samples that have `on_portal = TRUE` are included. I'm not totally sure how that complies with our T&C, but if we just include sample ID, library ID, diagnosis, disease timing, seq unit, and technology for all samples and libraries that are on the portal then I think its okay since that information is already publicly available. That is a question for @jaclyn-taroni though. 
author_association: MEMBER
user: login: sjspielman
created_at: 2025-04-11T17:49:04Z
body: I do want to note that I have literally just gotten code up and running with library and sample metadata files that should work. I will plan file that PR as a Draft but will wait to request review, pending additional discussion.

> They are available as part of the bulk download on the Portal and should be included in the single_cell_metadata.tsv for the entire project if you download with bulk. If that is not the case then that is a portal change that should be addressed.

This is not the case; these samples are not included in the `single_cell_metadata.tsv `when you download the project with bulk, which I had assumed is because their single-cell data is still pending portal upload and that they would be included in this file once associated single-cell samples are available for download since the portal indicates they are forthcoming.
author_association: MEMBER
user: login: allyhawkins
created_at: 2025-04-11T18:21:29Z
body: > > They are available as part of the bulk download on the Portal and should be included in the single_cell_metadata.tsv for the entire project if you download with bulk. If that is not the case then that is a portal change that should be addressed.
> 
> This is not the case; these samples are not included in the `single_cell_metadata.tsv `when you download the project with bulk, which I had assumed is because their single-cell data is still pending portal upload and that they would be included in this file once associated single-cell samples are available for download since the portal indicates they are forthcoming.

I filed https://github.com/AlexsLemonade/scpca-portal/issues/1191 to address this at some point in the future. 
author_association: MEMBER
user: login: jaclyn-taroni
created_at: 2025-04-14T12:36:38Z
body: Since a draft pull request is open, do you still need me to weigh in on something? It's hard for me to tell!
author_association: MEMBER
user: login: sjspielman
created_at: 2025-04-14T12:50:38Z
body: > Since a draft pull request is open, do you still need me to weigh in on something? It's hard for me to tell!

I think this is the move now - @allyhawkins, have a glance over #261, and if you strongly disagree with the strategy we'll bring @jaclyn-taroni back around?

> Another thought is that we should just create a simplified version of our internal sample and library metadata files and add it to this repo. 

I do want to respond to this comment in particular from @allyhawkins with some fresh thoughts:
- I would have agreed this would be the thing to do in this circumstance, but since we have to parse the whole portal _anyways_ I figured we're parsing anyways might as well parse
- That said, perhaps this suggestion is more reliably longer-term for reproducing the paper figures given that the portal will continue growing well past the samples/projects included in the paper! _But_ since we have a sample & project whitelist in the repo to limit what is used in the metadata, I think it should actually be ok moving forward to parse the portal metadata. 
author_association: MEMBER
user: login: allyhawkins
created_at: 2025-04-14T14:29:22Z
body: > I do want to respond to this comment in particular from [@allyhawkins](https://github.com/allyhawkins) with some fresh thoughts:
> 
> * I would have agreed this would be the thing to do in this circumstance, but since we have to parse the whole portal _anyways_ I figured we're parsing anyways might as well parse
> * That said, perhaps this suggestion is more reliably longer-term for reproducing the paper figures given that the portal will continue growing well past the samples/projects included in the paper! _But_ since we have a sample & project whitelist in the repo to limit what is used in the metadata, I think it should actually be ok moving forward to parse the portal metadata.

I think if we were starting from the beginning I would favor making the simplified version and storing in the repo, but given that most of the work is already done in #261 and we have a sample/project whitelist then I'm fine with the strategy you are using. 
author_association: MEMBER

Diff:
diff --git a/reproduce-figures/prepare-scpca-portal-data.R b/reproduce-figures/prepare-scpca-portal-data.R
index 43afbc18..29aeb0da 100644
--- a/reproduce-figures/prepare-scpca-portal-data.R
+++ b/reproduce-figures/prepare-scpca-portal-data.R
@@ -9,6 +9,8 @@
 #
 #####################################################################################
 ### CAUTION! YOU WILL NEED AT LEAST 170 GB OF AVAILABLE SPACE TO RUN THIS SCRIPT. ###
+###                                                                               ###
+### In addition, it will take roughly 90 minutes to run this script.              ###
 #####################################################################################
 #
 #
@@ -136,15 +138,26 @@ if ((dir.exists(opts$s3_files_dir) | dir.exists(opts$bulk_data_dir))) {
 consensus_files_dir   <- file.path(opts$s3_files_dir, "cell-type-consensus-results")
 celltype_results_dir  <- file.path(opts$s3_files_dir, "celltype_results")
 s3_files_reference_dir <- file.path(opts$s3_files_dir, "reference_files")
+merged_sce_dir <- file.path(opts$s3_files_dir, "SCPCP000003")
 bulk_reference_dir    <- file.path(opts$bulk_data_dir, "references")
 
+# these directories are for temporary file stored in scratch
+merged_scratch_dir <- file.path(opts$scratch_dir, "SCPCP000003_merged")
+bulk_metadata_scratch_dir <- file.path(opts$scratch_dir, "bulk-metadata")
+citeseq_metadata_scratch_dir <- file.path(opts$scratch_dir, "citeseq-project-metadata")
+
+
 fs::dir_create(c(
   opts$scratch_dir,
   opts$bulk_data_dir,
   consensus_files_dir,
   celltype_results_dir,
   s3_files_reference_dir,
-  bulk_reference_dir
+  merged_sce_dir,
+  bulk_reference_dir,
+  merged_scratch_dir,
+  bulk_metadata_scratch_dir,
+  citeseq_metadata_scratch_dir
 ))
 
 # Define marker genes for recreating expression matrices used for consensus cell type dot plots
@@ -185,6 +198,9 @@ bulk_projects <- c(
   "SCPCP000017"
 )
 
+# Projects with CITE-seq technology which we'll assign technology to when preparing metadata
+citeseq_projects <- c("SCPCP000003", "SCPCP000007", "SCPCP000008")
+
 
 # Define input files ------------------------
 
@@ -211,18 +227,13 @@ stopifnot(
 )
 
 # First, unzip and copy over the merged SCPCP000003 file -----------------------
-merged_sce_dir <- file.path(opts$s3_files_dir, "SCPCP000003")
-merge_scratch_dir <- file.path(opts$scratch_dir, "SCPCP000003_merged")
-fs::dir_create(c(merged_sce_dir, merge_scratch_dir))
-
-unzip(opts$merged_sce_path, exdir = merge_scratch_dir)
+unzip(opts$merged_sce_path, exdir = merged_scratch_dir)
 fs::file_move(
-  file.path(merge_scratch_dir, "SCPCP000003_merged.rds"),
+  file.path(merged_scratch_dir, "SCPCP000003_merged.rds"),
   merged_sce_dir
 )
-
-# clean up
-fs::dir_delete(merge_scratch_dir)
+# clean up for space
+fs::dir_delete(merged_scratch_dir)
 
 # Map over project zip files to organize files for reproducibility -------------
 input_zips |>
@@ -266,6 +277,20 @@ input_zips |>
             }
 
       })
+      
+      # Save metadata files to use when preparing the sample and library metadata files later
+      project_bulk_tsv <- file.path(project_scratch_dir, glue::glue("{project_id}_bulk_metadata.tsv"))
+      if (file.exists(project_bulk_tsv)) {
+        fs::file_copy(project_bulk_tsv, bulk_metadata_scratch_dir, overwrite = TRUE)
+      }
+      
+      if (project_id %in% citeseq_projects) {
+        fs::file_copy(
+          file.path(project_scratch_dir, glue::glue("single_cell_metadata.tsv")), 
+          file.path(citeseq_metadata_scratch_dir, glue::glue("{project_id}_single_cell_metadata.tsv")), 
+          overwrite = TRUE
+        )
+      }
 
       # If this project is used in the bulk analysis, copy to bulk_project_dir
       # We do this after copying samples above since we don't want to remove their (un)filtered files
@@ -274,7 +299,7 @@ input_zips |>
         system(glue::glue("rm {project_scratch_dir}/*/*filtered.rds"))
         system(glue::glue("rm {project_scratch_dir}/*/*html"))
 
-        fs::dir_copy(project_scratch_dir, opts$bulk_data_dir)
+        fs::dir_copy(project_scratch_dir, opts$bulk_data_dir, overwrite = TRUE)
       }
 
       ####### Step 2: Prepare consensus cell type files #######
@@ -325,20 +350,22 @@ input_zips |>
 # Read input metadata file
 portal_metadata <- readr::read_tsv(opts$portal_metadata_path)
 sample_metadata_file <- file.path(opts$s3_files_dir, "scpca-sample-metadata.tsv")
-library_metadata_file <- file.path(opts$s3_files_dir, "library-sample-metadata.tsv")
+library_metadata_file <- file.path(opts$s3_files_dir, "scpca-library-metadata.tsv")
 
 # Create and export sample metadata table
- portal_metadata |>
-  dplyr::select(scpca_project_id, scpca_sample_id, diagnosis, disease_timing, is_cell_line) |>
-  # remove duplicate rows, which occur when there are multiple libraries per sample
-  dplyr::distinct() |>
-  readr::write_tsv(sample_metadata_file)
+prepare_sample_metadata(
+  portal_metadata,
+  sample_metadata_file
+)
 
-# Create and export library metadata table
-library_metadata <- portal_metadata |>
-  dplyr::select(scpca_project_id, scpca_sample_id, scpca_library_id, seq_unit, technology) |>
-  readr::write_tsv(library_metadata_file)
 
+# Create and export library metadata table
+prepare_library_metadata(
+  portal_metadata, 
+  bulk_metadata_scratch_dir,
+  citeseq_metadata_scratch_dir,
+  library_metadata_file
+)
 
 # Sync reference files from S3 ------------------------------------
 
diff --git a/reproduce-figures/utils.R b/reproduce-figures/utils.R
index a311ca35..66e41b7b 100644
--- a/reproduce-figures/utils.R
+++ b/reproduce-figures/utils.R
@@ -1,8 +1,5 @@
-# This file contains functions used in prepare-scpca-portal-data.R to process consensus cell types.
-
-
-
-
+# This file contains functions used in prepare-scpca-portal-data.R to process consensus cell types
+#  and to create metadata files.
 
 #' Prepare and export TSV of consensus cell types
 #'
@@ -75,3 +72,123 @@ prepare_gene_expression_tsv <- function(sce, marker_genes, output_tsv) {
   # export the marker gene expression tsv
   readr::write_tsv(gene_exp_df, output_tsv)
 }
+
+
+
+
+#' Prepare and export sample metadata file
+#'
+#' @param portal_metadata Data frame of portal-wide metadata
+#' @param output_tsv Path to output TSV file
+#'
+prepare_sample_metadata <- function(
+    portal_metadata, 
+    output_tsv) {
+  
+  sample_metadata <- portal_metadata |>
+    dplyr::select(scpca_project_id, scpca_sample_id, diagnosis, disease_timing, is_cell_line) |>
+    # remove duplicate rows, which occur when there are multiple libraries per sample
+    dplyr::distinct()
+  
+  # Prepare data frame with samples whose full metadata is not yet available on the portal
+  bulk_only_sample_metadata <-  tibble::tribble(
+    ~scpca_project_id, ~scpca_sample_id, ~diagnosis,        ~disease_timing,     ~is_cell_line, 
+    ###########################################################################################
+    "SCPCP000006",     "SCPCS000210",    "Wilms tumor",     "Initial diagnosis", FALSE,
+    "SCPCP000006",     "SCPCS000211",    "Wilms tumor",     "Initial diagnosis", FALSE,
+    "SCPCP000009",     "SCPCS000129",    "Medulloblastoma", "Initial diagnosis", FALSE,
+    "SCPCP000017",     "SCPCS000606",    "Osteosarcoma",    "Recurrence",        FALSE
+  )
+  
+  sample_metadata |>
+    dplyr::bind_rows(bulk_only_sample_metadata) |>
+    readr::write_tsv(output_tsv)
+}
+  
+  
+  
+
+
+#' Prepare and export library metadata file
+#'
+#' @param portal_metadata Data frame of portal-wide metadata
+#' @param bulk_metadata_dir Directory with bulk metadata TSV files
+#' @param project_metadata_dir Directory with project-specific metadata TSV files
+#' @param output_tsv Path to output TSV file
+#'
+prepare_library_metadata <- function(
+    portal_metadata, 
+    bulk_metadata_dir, 
+    project_metadata_dir, 
+    output_tsv) {
+  
+  # final columns to include in the exported library metadata tsv
+  final_columns <- c(
+    "scpca_project_id",
+    "scpca_sample_id",
+    "scpca_library_id",
+    "seq_unit",
+    "technology"
+  )
+  
+  library_metadata <- portal_metadata |>
+    # Group multiplexed sample ids back together; keep has_cellhash for later manipulation
+    dplyr::group_by(scpca_project_id, scpca_library_id, seq_unit, technology, has_cellhash) |>
+    dplyr::summarize(scpca_sample_id = paste(scpca_sample_id, collapse = ";")) |>
+    dplyr::ungroup()
+  
+  # duplicate multiplexed rows so we have a row for cellhash technology
+  cellhash_rows <- library_metadata |>
+    dplyr::filter(has_cellhash) |>
+    # in our code, we detect this technology with "cellhash" only, so the version isn't needed
+    dplyr::mutate(technology = "cellhash")
+    
+  library_metadata <- library_metadata |>
+    dplyr::bind_rows(cellhash_rows) |>
+    dplyr::select(-has_cellhash)
+  
+  # Define bulk and project metadata files
+  bulk_files <- list.files(
+    path = bulk_metadata_dir, 
+    pattern = "*_bulk_metadata\\.tsv", 
+    full.names = TRUE
+  )
+  citeseq_files <- list.files(
+    path = project_metadata_dir, 
+    pattern = "*_single_cell_metadata\\.tsv", 
+    full.names = TRUE
+  ) 
+  
+  # Parse bulk metadata
+  bulk_metadata <- bulk_files |>
+    purrr::map(readr::read_tsv) |>
+    purrr::list_rbind() |>
+    dplyr::rename(
+      scpca_project_id = project_id, 
+      scpca_sample_id = sample_id, 
+      scpca_library_id = library_id
+    ) |>
+    dplyr::select(all_of(final_columns))
+  
+  # Parse project metadata to get rows for CITE-seq
+  cite_metadata <- citeseq_files |>
+    purrr::map(
+      \(file) {
+        metadata_df <- readr::read_tsv(file) |>
+          dplyr::filter(!is.na(adt_filtering_method)) |> 
+          # in our code, we detect this technology with "CITE" only, so the version isn't needed
+          dplyr::mutate(technology = "CITEseq") |>
+          dplyr::select(all_of(final_columns))
+        
+        return(metadata_df)
+      }
+    ) |>
+    purrr::list_rbind()
+  
+  # Combine data frames and export
+  library_metadata |>
+    dplyr::bind_rows(bulk_metadata) |>
+    dplyr::bind_rows(cite_metadata) |>
+    readr::write_tsv(output_tsv)
+}
+


Reviews:
user: login: allyhawkins
body: This looks mostly good, I just have a comment about saving some copying of metadata files. 
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-14T15:28:21Z
user: login: allyhawkins
body: LGTM
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-14T17:59:09Z

Review comments:
diff_hunk: @@ -75,3 +72,123 @@ prepare_gene_expression_tsv <- function(sce, marker_genes, output_tsv) {
   # export the marker gene expression tsv
   readr::write_tsv(gene_exp_df, output_tsv)
 }
+
+
+
+
+#' Prepare and export sample metadata file
+#'
+#' @param portal_metadata Data frame of portal-wide metadata
+#' @param output_tsv Path to output TSV file
+#'
+prepare_sample_metadata <- function(
+    portal_metadata, 
+    output_tsv) {
+  
+  sample_metadata <- portal_metadata |>
+    dplyr::select(scpca_project_id, scpca_sample_id, diagnosis, disease_timing, is_cell_line) |>
+    # remove duplicate rows, which occur when there are multiple libraries per sample
+    dplyr::distinct()
+  
+  # Prepare data frame with samples whose full metadata is not yet available on the portal
+  bulk_only_sample_metadata <-  tibble::tribble(
+    ~scpca_project_id, ~scpca_sample_id, ~diagnosis,        ~disease_timing,     ~is_cell_line, 
+    ###########################################################################################
+    "SCPCP000006",     "SCPCS000210",    "Wilms tumor",     "Initial diagnosis", FALSE,
+    "SCPCP000006",     "SCPCS000211",    "Wilms tumor",     "Initial diagnosis", FALSE,
+    "SCPCP000009",     "SCPCS000129",    "Medulloblastoma", "Initial diagnosis", FALSE,
+    "SCPCP000017",     "SCPCS000606",    "Osteosarcoma",    "Recurrence",        FALSE
+  )
+  
+  sample_metadata |>
+    dplyr::bind_rows(bulk_only_sample_metadata) |>
+    readr::write_tsv(output_tsv)
+}
+  
+  
+  
+
+
+#' Prepare and export library metadata file
+#'
+#' @param portal_metadata Data frame of portal-wide metadata
+#' @param bulk_metadata_dir Directory with bulk metadata TSV files
+#' @param project_metadata_dir Directory with project-specific metadata TSV files
+#' @param output_tsv Path to output TSV file
+#'
+prepare_library_metadata <- function(
+    portal_metadata, 
+    bulk_metadata_dir, 
+    project_metadata_dir, 
+    output_tsv) {
+  
+  # final columns to include in the exported library metadata tsv
+  final_columns <- c(
+    "scpca_project_id",
+    "scpca_sample_id",
+    "scpca_library_id",
+    "seq_unit",
+    "technology"
+  )
+  
+  library_metadata <- portal_metadata |>
+    # Group multiplexed sample ids back together; keep has_cellhash for later manipulation
+    dplyr::group_by(scpca_project_id, scpca_library_id, seq_unit, technology, has_cellhash) |>
+    dplyr::summarize(scpca_sample_id = paste(scpca_sample_id, collapse = ";")) |>
+    dplyr::ungroup()
+  
+  # duplicate multiplexed rows so we have a row for cellhash technology
+  cellhash_rows <- library_metadata |>
+    dplyr::filter(has_cellhash) |>
+    # in our code, we detect this technology with "cellhash" only, so the version isn't needed
+    dplyr::mutate(technology = "cellhash")
+    
+  library_metadata <- library_metadata |>
+    dplyr::bind_rows(cellhash_rows) |>
+    dplyr::select(-has_cellhash)
+  
+  # Define bulk and project metadata files
+  bulk_files <- list.files(
+    path = bulk_metadata_dir, 
+    pattern = "*_bulk_metadata\\.tsv", 
+    full.names = TRUE
+  )
path: reproduce-figures/utils.R
user: login: allyhawkins
body: I think you should be able to search for this within the `project_scratch_dir` rather than copy files over to a bulk scratch directory first. 
created_at: 2025-04-14T15:17:03Z
author_association: MEMBER
diff_hunk: @@ -75,3 +72,123 @@ prepare_gene_expression_tsv <- function(sce, marker_genes, output_tsv) {
   # export the marker gene expression tsv
   readr::write_tsv(gene_exp_df, output_tsv)
 }
+
+
+
+
+#' Prepare and export sample metadata file
+#'
+#' @param portal_metadata Data frame of portal-wide metadata
+#' @param output_tsv Path to output TSV file
+#'
+prepare_sample_metadata <- function(
+    portal_metadata, 
+    output_tsv) {
+  
+  sample_metadata <- portal_metadata |>
+    dplyr::select(scpca_project_id, scpca_sample_id, diagnosis, disease_timing, is_cell_line) |>
+    # remove duplicate rows, which occur when there are multiple libraries per sample
+    dplyr::distinct()
+  
+  # Prepare data frame with samples whose full metadata is not yet available on the portal
+  bulk_only_sample_metadata <-  tibble::tribble(
+    ~scpca_project_id, ~scpca_sample_id, ~diagnosis,        ~disease_timing,     ~is_cell_line, 
+    ###########################################################################################
+    "SCPCP000006",     "SCPCS000210",    "Wilms tumor",     "Initial diagnosis", FALSE,
+    "SCPCP000006",     "SCPCS000211",    "Wilms tumor",     "Initial diagnosis", FALSE,
+    "SCPCP000009",     "SCPCS000129",    "Medulloblastoma", "Initial diagnosis", FALSE,
+    "SCPCP000017",     "SCPCS000606",    "Osteosarcoma",    "Recurrence",        FALSE
+  )
+  
+  sample_metadata |>
+    dplyr::bind_rows(bulk_only_sample_metadata) |>
+    readr::write_tsv(output_tsv)
+}
+  
+  
+  
+
+
+#' Prepare and export library metadata file
+#'
+#' @param portal_metadata Data frame of portal-wide metadata
+#' @param bulk_metadata_dir Directory with bulk metadata TSV files
+#' @param project_metadata_dir Directory with project-specific metadata TSV files
+#' @param output_tsv Path to output TSV file
+#'
+prepare_library_metadata <- function(
+    portal_metadata, 
+    bulk_metadata_dir, 
+    project_metadata_dir, 
+    output_tsv) {
+  
+  # final columns to include in the exported library metadata tsv
+  final_columns <- c(
+    "scpca_project_id",
+    "scpca_sample_id",
+    "scpca_library_id",
+    "seq_unit",
+    "technology"
+  )
+  
+  library_metadata <- portal_metadata |>
+    # Group multiplexed sample ids back together; keep has_cellhash for later manipulation
+    dplyr::group_by(scpca_project_id, scpca_library_id, seq_unit, technology, has_cellhash) |>
+    dplyr::summarize(scpca_sample_id = paste(scpca_sample_id, collapse = ";")) |>
+    dplyr::ungroup()
+  
+  # duplicate multiplexed rows so we have a row for cellhash technology
+  cellhash_rows <- library_metadata |>
+    dplyr::filter(has_cellhash) |>
+    # in our code, we detect this technology with "cellhash" only, so the version isn't needed
+    dplyr::mutate(technology = "cellhash")
+    
+  library_metadata <- library_metadata |>
+    dplyr::bind_rows(cellhash_rows) |>
+    dplyr::select(-has_cellhash)
+  
+  # Define bulk and project metadata files
+  bulk_files <- list.files(
+    path = bulk_metadata_dir, 
+    pattern = "*_bulk_metadata\\.tsv", 
+    full.names = TRUE
+  )
+  citeseq_files <- list.files(
+    path = project_metadata_dir, 
+    pattern = "*_single_cell_metadata\\.tsv", 
+    full.names = TRUE
+  ) 
path: reproduce-figures/utils.R
user: login: allyhawkins
body: I think same comment here. You can just provide one argument `project_scratch_dir` and grab the metadata files from there by recursively searching and then filtering based on the project ID being a CITE seq project. 
created_at: 2025-04-14T15:26:16Z
author_association: MEMBER
diff_hunk: @@ -266,6 +277,20 @@ input_zips |>
             }
 
       })
+      
+      # Save metadata files to use when preparing the sample and library metadata files later
+      project_bulk_tsv <- file.path(project_scratch_dir, glue::glue("{project_id}_bulk_metadata.tsv"))
+      if (file.exists(project_bulk_tsv)) {
+        fs::file_copy(project_bulk_tsv, bulk_metadata_scratch_dir, overwrite = TRUE)
+      }
+      
+      if (project_id %in% citeseq_projects) {
+        fs::file_copy(
+          file.path(project_scratch_dir, glue::glue("single_cell_metadata.tsv")), 
+          file.path(citeseq_metadata_scratch_dir, glue::glue("{project_id}_single_cell_metadata.tsv")), 
+          overwrite = TRUE
+        )
+      }
path: reproduce-figures/prepare-scpca-portal-data.R
user: login: allyhawkins
body: Note some of my comments below, but I think you could remove this entirely and you don't need to copy metadata files from one scratch to the next scratch. 
created_at: 2025-04-14T15:26:48Z
author_association: MEMBER

PR comments:
user: login: sjspielman
created_at: 2025-04-14T16:50:41Z
body: So actually, the reason I am copying is to avoid hammering disk space. At the very end of the main purrr "loop", I have a line to remove the project scratch dir. This means that, after a given project is parsed, it's unzipped directory in scratch is removed:
https://github.com/AlexsLemonade/scpca-paper-figures/blob/a052a79f631865fe515fc87fcee2dc24ce6d316a/reproduce-figures/prepare-scpca-portal-data.R#L318-L320

If we don't so this, then the scratch directory will end up with all unzipped projects in it (which we'd remove in one line at the very end). This means users would need like 300-400 GB free space to run the code which is a heck of a lot and I wanted to avoid that burden. 

In any case, I'm not seeing a different strategy where we can avoid all the storage _and_ be able to not copy the files, so please cue me in if you see it?
author_association: MEMBER
user: login: allyhawkins
created_at: 2025-04-14T17:58:58Z
body: > If we don't so this, then the scratch directory will end up with all unzipped projects in it (which we'd remove in one line at the very end). This means users would need like 300-400 GB free space to run the code which is a heck of a lot and I wanted to avoid that burden.

Ah that makes sense as to why you were copying then! 
author_association: MEMBER