teddy/data_processing/preprocessing/README.md

PreprocessReadMe.md

The preprocess.py script is designed to preprocess gene expression data for use in our models. It takes in data.raw.X or data.X data, applies various preprocessing techniques, and prepares it for training or inference.

General Workflow

The script follows these main steps: 0. Load Data and Metadata: The script starts by loading the gene expression data from an AnnData file and metadata from a JSON file.

1. Set Raw Layer: It checks if the data.raw.X layer is set in the AnnData object. If not, it sets it based on the integer counts in the data.X.
2. Initialize Processed Layer: It initializes the data.layer['processed'] in the AnnData object, which is the layer that will be affected by preprocessing.
3. Filter Genes by Reference ID: It filters genes based on reference IDs if specified in the hyperparameters.
4. Remove Assays: It removes specified assays from the data.
5. Filter Cells by Gene Counts: It filters out cells with gene counts below a specified threshold.
6. Filter Cells by Mitochondrial Fraction: It removes cells with a high mitochondrial gene fraction.
7. Filter Highly Variable Genes: It filters genes to retain only highly variable ones using specified methods.
8. Normalize Data: It normalizes the data by applying row (gene level) normalization and scaling.
9. Scale Columns by Median: It scales columns based on median values from a specified dictionary.
10. Log Transform: It applies a log+1 transformation to the data.
11. Compute Medians: It computes and saves medians of the processed data if specified.
12. Update Metadata: It updates the metadata with cell counts and processing arguments.
13. Save and Cleanup: It saves the processed data and metadata to disk and performs garbage collection.

Preprocessing Arguments

The script uses several preprocessing arguments to control its behavior. Here is an explanation of each argument and the steps they influence:

• reference_id_only
  • Description: Specifies whether to filter genes by reference ID.
  • Impact: If enabled, the script filters genes based on reference IDs.
• remove_assays
  • Description: List of assays to remove from the data.
  • Impact: The script removes specified assays from the data.
• min_gene_counts
  • Description: Minimum gene counts required for cells to be retained.
  • Impact: The script filters out cells with gene counts below this threshold.
• max_mitochondrial_prop
  • Description: Maximum mitochondrial gene fraction allowed for cells.
  • Impact: The script removes cells with a mitochondrial gene fraction above this threshold.
• hvg_method
  • Description: Method to use for filtering highly variable genes.
  • Impact: The script filters genes to retain only highly variable ones using the specified method.
• normalized_total
  • Description: Value to normalize the total gene expression to.
  • Impact: The script normalizes the data by applying row (gene level) normalization and scaling.
• median_dict
  • Description: Path to a JSON file containing median values for scaling columns.
  • Impact: The script scales columns based on median values from the specified dictionary.
• median_column
  • Description: Column name to use for looking up median values.
  • Impact: The script uses this column to look up median values for scaling.
• log1p
  • Description: Indicates whether to apply a log transformation to the data.
  • Impact: If enabled, the script applies a log transformation to the data.
• compute_medians
  • Description: Indicates whether to compute and save medians of the processed data.
  • Impact: If enabled, the script computes and saves medians of the processed data.