Skill: renhaosu2024/clinical-data-cleaner (version 0.1.1)

--- name: clinical-data-cleaner description: Use when cleaning clinical trial data, preparing data for FDA/EMA submission, standardizing SDTM datasets, handling missing values in clinical studies, detecting outliers in lab results, or converting raw CRF data to CDISC format. Cleans and standardizes clinical trial data for regulatory compliance with audit trails. allowed-tools: "Read Write Bash Edit" license: MIT metadata: skill-author: AIPOCH version: "2.0" --- # Clinical Data Cleaner Clean, validate, and standardize clinical trial data to meet CDISC SDTM standards for regulatory submissions to FDA or EMA. ## Quick Start ```python from scripts.main import ClinicalDataCleaner # Initialize for Demographics domain cleaner = ClinicalDataCleaner(domain='DM') # Clean data with default settings cleaned = cleaner.clean(raw_data) # Save with audit trail cleaner.save_report('output.csv') ``` ## Core Capabilities ### 1. SDTM Domain Validation ```python cleaner = ClinicalDataCleaner(domain='DM') # or 'LB', 'VS' is_valid, missing = cleaner.validate_domain(data) ``` **Required Fields:** - **DM**: STUDYID, USUBJID, SUBJID, RFSTDTC, RFENDTC, SITEID, AGE, SEX, RACE - **LB**: STUDYID, USUBJID, LBTESTCD, LBCAT, LBORRES, LBORRESU, LBSTRESC, LBDTC - **VS**: STUDYID, USUBJID, VSTESTCD, VSORRES, VSORRESU, VSSTRESC, VSDTC ### 2. Missing Value Handling ```python cleaner = ClinicalDataCleaner( domain='DM', missing_strategy='median' # mean, median, mode, forward, drop ) cleaned = cleaner.handle_missing_values(data) ``` ### 3. Outlier Detection ```python cleaner = ClinicalDataCleaner( domain='LB', outlier_method='domain', # iqr, zscore, domain outlier_action='flag' # flag, remove, cap ) flagged = cleaner.detect_outliers(data) ``` **Clinical Thresholds:** | Parameter | Range | Unit | |-----------|-------|------| | Glucose | 50-500 | mg/dL | | Hemoglobin | 5-20 | g/dL | | Systolic BP | 70-220 | mmHg | ### 4. Date Standardization ```python standardized = cleaner.standardize_dates(data) # Converts to ISO 8601: 2023-01-15T09:30:00 ``` ### 5. Complete Pipeline ```python cleaner = ClinicalDataCleaner( domain='DM', missing_strategy='median', outlier_method='iqr', outlier_action='flag' ) cleaned_data = cleaner.clean(data) cleaner.save_report('output.csv') ``` **Output Files:** - `output.csv` - Cleaned SDTM data - `output.report.json` - Audit trail for regulatory submission ## CLI Usage ```bash # Clean demographics python scripts/main.py \ --input dm_raw.csv \ --domain DM \ --output dm_clean.csv \ --missing-strategy median \ --outlier-method iqr \ --outlier-action flag # Clean lab data with clinical thresholds python scripts/main.py \ --input lb_raw.csv \ --domain LB \ --output lb_clean.csv \ --outlier-method domain ``` ## Common Patterns See [references/common-patterns.md](references/common-patterns.md) for detailed examples: - Regulatory Submission Preparation - Interim Analysis Data Preparation - Database Migration Cleanup - External Lab Data Integration ## Troubleshooting See [references/troubleshooting.md](references/troubleshooting.md) for solutions to: - Validation failures - Date parsing errors - Memory errors with large datasets - Outlier detection issues ## Quality Checklist **Pre-Cleaning:** - [ ] IACUC approval obtained (animal studies) - [ ] Sample size adequately powered - [ ] Randomization method documented **Post-Cleaning:** - [ ] Validate against CDISC SDTM IG - [ ] Review all cleaning actions in audit trail - [ ] Test import to analysis software ## References - `references/sdtm_ig_guide.md` - CDISC SDTM Implementation Guide - `references/domain_specs.json` - Domain-specific field requirements - `references/outlier_thresholds.json` - Clinical outlier thresholds - `references/common-patterns.md` - Detailed usage patterns - `references/troubleshooting.md` - Problem-solving guide --- **Skill ID**: 189 | **Version**: 2.0 | **License**: MIT

path: references/common-patterns.md
content: ## Common Patterns

### Pattern 1: Regulatory Submission Preparation

**Scenario**: Preparing SDTM datasets for FDA submission.

```json
{
  "submission_type": "FDA NDA",
  "domains": ["DM", "LB", "VS", "AE", "MH"],
  "cleaning_approach": "Conservative - flag rather than remove",
  "validation": "CDISC SDTM IG v3.2",
  "audit_requirements": "Complete traceability of all changes",
  "deliverables": [
    "Cleaned datasets",
    "Cleaning reports",
    "Programming specifications"
  ]
}
```

**Workflow:**
1. Load raw data from EDC system
2. Validate against SDTM domain specifications
3. Clean using conservative settings (flag outliers)
4. Generate comprehensive audit trails
5. Validate final datasets with Pinnacle 21
6. Document all cleaning decisions
7. Package for regulatory submission

**Output Example:**
```
FDA Submission Package:
  
Datasets:
  ✓ dm.xpt (1,247 subjects)
  ✓ lb.xpt (45,678 records)
  ✓ vs.xpt (12,470 records)
  
Cleaning Statistics:
  - Missing values imputed: 234
  - Outliers flagged: 89
  - Date formats standardized: 1,247
  
Audit Trail:
  - Cleaning reports: 3 files
  - All actions documented
  - 21 CFR Part 11 compliant
  
Validation:
  ✓ Pinnacle 21: 0 errors, 3 warnings
  ✓ CDISC SDTM IG v3.2 compliant
```

### Pattern 2: Interim Analysis Data Preparation

**Scenario**: Cleaning data for interim analysis during ongoing trial.

```json
{
  "analysis_type": "Interim efficacy",
  "data_cutoff": "2023-12-31",
  "cleaning_priority": "Speed with quality",
  "domains_needed": ["DM", "LB", "VS"],
  "outlier_handling": "Flag for statistician review",
  "timeline": "3 days"
}
```

**Workflow:**
1. Extract data with cutoff date
2. Quick validation of key fields
3. Impute missing values with median
4. Flag outliers (don't remove)
5. Standardize dates
6. Deliver to statistics team
7. Document known issues

**Output Example:**
```
Interim Analysis Dataset:
  Data cutoff: 2023-12-31
  Subjects: 456/500 enrolled
  
Cleaning Summary:
  - Processing time: 2 hours
  - Missing values: 45 (imputed)
  - Outliers flagged: 12
  - Ready for analysis: Yes
  
Notes for Statistician:
  - 3 subjects with incomplete follow-up
  - 1 site with delayed data entry
  - All outliers reviewed and plausible
```

### Pattern 3: Database Migration Cleanup

**Scenario**: Cleaning data when migrating between data management systems.

```json
{
  "migration_type": "EDC system upgrade",
  "source_system": "Legacy EDC",
  "target_system": "New Veeva",
  "challenges": [
    "Different date formats",
    "Field name changes",
    "Encoding issues"
  ],
  "validation": "Compare before/after counts"
}
```

**Workflow:**
1. Export all data from legacy system
2. Map old field names to SDTM
3. Standardize formats (dates, categories)
4. Clean missing/outlier values
5. Validate record counts match
6. Test import to new system
7. Document all transformations

**Output Example:**
```
Database Migration Results:
  Legacy records: 15,678
  Migrated records: 15,678
  Match: ✓ 100%
  
Transformations Applied:
  - Date format: 23,456 fields
  - Field names: 156 mappings
  - Missing values: 234 imputed
  - Outliers: 45 flagged
  
Validation:
  ✓ Subject counts match
  ✓ Record counts match
  ✓ Critical values preserved
  ✓ Import to new system successful
```

### Pattern 4: External Data Integration

**Scenario**: Integrating external lab data with clinical database.

```json
{
  "data_source": "Central lab",
  "integration_type": "LB domain augmentation",
  "challenges": [
    "Different units",
    "Varying reference ranges",
    "Date/time zone issues"
  ],
  "cleaning_focus": "Standardization and validation"
}
```

**Workflow:**
1. Load central lab data export
2. Map local lab codes to LBTESTCD
3. Standardize units (convert if needed)
4. Validate against reference ranges
5. Handle date/time zones
6. Merge with existing LB data
7. Validate no duplicates

**Output Example:**
```
Central Lab Integration:
  External records: 8,234
  Successfully integrated: 8,234 (100%)
  
Standardization:
  - Unit conversions: 234 records
  - Date adjustments: 8,234 records
  - Code mappings: 45 tests
  
Validation:
  ✓ No duplicate records
  ✓ All USUBJID matched
  ✓ Units standardized
  ✓ Reference ranges aligned
  
Data Quality:
  - Outliers flagged: 23
  - Missing values: 0
  - Ready for analysis: Yes
```

---

## Quality Checklist

**Pre-Cleaning:**
- [ ] **CRITICAL**: Verify input data is from validated source (EDC, not draft)
- [ ] Confirm data export date and cutoff
- [ ] Check file format (CSV/Excel) and encoding
- [ ] Verify domain specification (DM, LB, VS)
- [ ] Review study protocol for expected data structure
- [ ] Check for data lock status
- [ ] Confirm access permissions and data security
- [ ] Document raw data file location (for audit)

**Cleaning Configuration:**
- [ ] **CRITICAL**: Select appropriate missing value strategy for data type
- [ ] Choose outlier method appropriate for domain (use 'domain' for LB/VS)
- [ ] Set outlier action based on regulatory requirements (prefer 'flag')
- [ ] Review custom configuration if provided
- [ ] Confirm cleaning parameters with statistician
- [ ] Document rationale for all parameter choices
- [ ] Check if stratification needed (by site, treatment arm)
- [ ] Validate cleaning approach in Statistical Analysis Plan

**During Cleaning:**
- [ ] **CRITICAL**: Review validation warnings (missing fields)
- [ ] Check missing value imputation counts
- [ ] Review outlier detection results
- [ ] Verify flagged outliers are appropriate
- [ ] Check date standardization success rate
- [ ] Monitor for unexpected data loss
- [ ] Review cleaning log for anomalies
- [ ] Compare row counts before/after

**Post-Cleaning:**
- [ ] **CRITICAL**: Validate cleaned data against CDISC SDTM IG
- [ ] Check Pinnacle 21 (or similar) validation results
- [ ] Review all cleaning actions in audit trail
- [ ] Verify SDTM domain structure correct
- [ ] Test import to analysis software (SAS, R)
- [ ] Generate summary statistics for key variables
- [ ] Compare with expected ranges
- [ ] Document any deviations from protocol

**Regulatory Compliance:**
- [ ] **CRITICAL**: All cleaning actions documented with rationale
- [ ] Audit trail complete and archived
- [ ] Cleaning programs under version control
- [ ] Validation documentation complete
- [ ] Reviewed by independent QC (if required)
- [ ] Approved by statistician/medical monitor
- [ ] Aligned with Statistical Analysis Plan
- [ ] Ready for regulatory submission

---

## Common Pitfalls

**Data Quality Issues:**
- ❌ **Cleaning raw/draft data** → Final data different from cleaned
  - ✅ Only clean locked/validated data
  
- ❌ **Removing outliers without investigation** → Lose legitimate extreme values
  - ✅ Flag outliers; review before removing
  
- ❌ **Inappropriate imputation** → Biases statistical analysis
  - ✅ Choose strategy based on missingness mechanism
  
- ❌ **Ignoring missing patterns** → MNAR data treated as MCAR
  - ✅ Analyze missingness patterns; consult statistician

**Regulatory Issues:**
- ❌ **Incomplete audit trail** → Regulatory rejection
  - ✅ Log every single change with rationale
  
- ❌ **Changing data without documentation** → Compliance violation
  - ✅ Never modify raw data; create new cleaned dataset
  
- ❌ **Not validating against SDTM IG** → Submission issues
  - ✅ Always run CDISC validation tools
  
- ❌ **Cleaning after database lock** → Protocol violation
  - ✅ Clean before lock; any post-lock changes need documented approval

**Technical Issues:**
- ❌ **Wrong date parsing** → Incorrect temporal relationships
  - ✅ Validate date formats; check against CRFs
  
- ❌ **Unit conversion errors** → Invalid clinical values
  - ✅ Double-check all unit conversions
  
- ❌ **Subject ID mismatches** → Data linkage failures
  - ✅ Verify USUBJID consistency across domains
  
- ❌ **Overwriting original files** → Data loss
  - ✅ Always save to new file; preserve raw data

---



path: references/domain_specs.json
content: {
  "domains": {
    "DM": {
      "description": "Demographics",
      "required_fields": [
        "STUDYID",
        "USUBJID",
        "SUBJID",
        "RFSTDTC",
        "RFENDTC",
        "SITEID",
        "AGE",
        "SEX",
        "RACE"
      ],
      "field_types": {
        "STUDYID": "text",
        "USUBJID": "text",
        "SUBJID": "text",
        "RFSTDTC": "datetime",
        "RFENDTC": "datetime",
        "SITEID": "text",
        "AGE": "numeric",
        "SEX": "coded",
        "RACE": "coded"
      },
      "valid_values": {
        "SEX": ["M", "F", "U", "UNDIFFERENTIATED"],
        "RACE": [
          "AMERICAN INDIAN OR ALASKA NATIVE",
          "ASIAN",
          "BLACK OR AFRICAN AMERICAN",
          "NATIVE HAWAIIAN OR OTHER PACIFIC ISLANDER",
          "WHITE",
          "MULTIPLE",
          "NOT REPORTED",
          "UNKNOWN"
        ]
      }
    },
    "LB": {
      "description": "Laboratory Test Results",
      "required_fields": [
        "STUDYID",
        "USUBJID",
        "LBTESTCD",
        "LBCAT",
        "LBORRES",
        "LBORRESU",
        "LBSTRESC",
        "LBDTC"
      ],
      "field_types": {
        "STUDYID": "text",
        "USUBJID": "text",
        "LBTESTCD": "coded",
        "LBCAT": "coded",
        "LBORRES": "text",
        "LBORRESU": "text",
        "LBSTRESC": "text",
        "LBDTC": "datetime"
      },
      "common_tests": {
        "GLUC": "Glucose",
        "HGB": "Hemoglobin",
        "WBC": "White Blood Cell Count",
        "PLAT": "Platelet Count",
        "CREAT": "Creatinine",
        "ALT": "Alanine Aminotransferase",
        "AST": "Aspartate Aminotransferase",
        "BILI": "Bilirubin",
        "SODIUM": "Sodium",
        "POTAS": "Potassium",
        "CHOL": "Cholesterol"
      }
    },
    "VS": {
      "description": "Vital Signs",
      "required_fields": [
        "STUDYID",
        "USUBJID",
        "VSTESTCD",
        "VSORRES",
        "VSORRESU",
        "VSSTRESC",
        "VSDTC"
      ],
      "field_types": {
        "STUDYID": "text",
        "USUBJID": "text",
        "VSTESTCD": "coded",
        "VSORRES": "text",
        "VSORRESU": "text",
        "VSSTRESC": "text",
        "VSDTC": "datetime"
      },
      "common_tests": {
        "SYSBP": "Systolic Blood Pressure",
        "DIABP": "Diastolic Blood Pressure",
        "PULSE": "Pulse Rate",
        "RESP": "Respiratory Rate",
        "TEMP": "Temperature",
        "WEIGHT": "Weight",
        "HEIGHT": "Height",
        "BMI": "Body Mass Index"
      }
    }
  },
  "global_rules": {
    "date_format": "%Y-%m-%dT%H:%M:%S",
    "missing_value_representations": ["NA", "N/A", "", "null", "NULL", "."],
    "text_case": "upper",
    "decimal_precision": 2
  }
}


path: references/outlier_thresholds.json
content: {
  "LB": {
    "GLUC": {
      "description": "Glucose",
      "normal_range": {"min": 70, "max": 100},
      "critical_low": 40,
      "critical_high": 500,
      "unit": "mg/dL",
      "unit_alternatives": ["mg/dL", "mg/dl", "mgdL"]
    },
    "HGB": {
      "description": "Hemoglobin",
      "normal_range": {"min": 12, "max": 17},
      "critical_low": 5,
      "critical_high": 20,
      "unit": "g/dL",
      "unit_alternatives": ["g/dL", "g/dl", "gdL"]
    },
    "WBC": {
      "description": "White Blood Cell Count",
      "normal_range": {"min": 4500, "max": 11000},
      "critical_low": 1000,
      "critical_high": 50000,
      "unit": "cells/uL",
      "unit_alternatives": ["cells/uL", "K/uL", "10^3/uL"]
    },
    "PLAT": {
      "description": "Platelet Count",
      "normal_range": {"min": 150000, "max": 450000},
      "critical_low": 20000,
      "critical_high": 1000000,
      "unit": "cells/uL",
      "unit_alternatives": ["cells/uL", "K/uL", "10^3/uL"]
    },
    "CREAT": {
      "description": "Creatinine",
      "normal_range": {"min": 0.7, "max": 1.3},
      "critical_low": 0.3,
      "critical_high": 15,
      "unit": "mg/dL",
      "unit_alternatives": ["mg/dL", "mg/dl", "mgdL"]
    },
    "ALT": {
      "description": "Alanine Aminotransferase",
      "normal_range": {"min": 7, "max": 56},
      "critical_low": 5,
      "critical_high": 500,
      "unit": "U/L",
      "unit_alternatives": ["U/L", "U/l", "IU/L"]
    },
    "AST": {
      "description": "Aspartate Aminotransferase",
      "normal_range": {"min": 10, "max": 40},
      "critical_low": 5,
      "critical_high": 500,
      "unit": "U/L",
      "unit_alternatives": ["U/L", "U/l", "IU/L"]
    },
    "BILI": {
      "description": "Total Bilirubin",
      "normal_range": {"min": 0.1, "max": 1.2},
      "critical_low": 0.1,
      "critical_high": 15,
      "unit": "mg/dL",
      "unit_alternatives": ["mg/dL", "mg/dl", "mgdL"]
    }
  },
  "VS": {
    "SYSBP": {
      "description": "Systolic Blood Pressure",
      "normal_range": {"min": 90, "max": 120},
      "critical_low": 70,
      "critical_high": 220,
      "unit": "mmHg",
      "unit_alternatives": ["mmHg", "mm Hg", "mmhg"]
    },
    "DIABP": {
      "description": "Diastolic Blood Pressure",
      "normal_range": {"min": 60, "max": 80},
      "critical_low": 40,
      "critical_high": 140,
      "unit": "mmHg",
      "unit_alternatives": ["mmHg", "mm Hg", "mmhg"]
    },
    "PULSE": {
      "description": "Pulse Rate",
      "normal_range": {"min": 60, "max": 100},
      "critical_low": 40,
      "critical_high": 180,
      "unit": "beats/min",
      "unit_alternatives": ["beats/min", "bpm", "/min"]
    },
    "RESP": {
      "description": "Respiratory Rate",
      "normal_range": {"min": 12, "max": 20},
      "critical_low": 8,
      "critical_high": 40,
      "unit": "breaths/min",
      "unit_alternatives": ["breaths/min", "/min", "rpm"]
    },
    "TEMP": {
      "description": "Temperature",
      "normal_range": {"min": 97, "max": 99},
      "critical_low": 94,
      "critical_high": 108,
      "unit": "F",
      "unit_alternatives": ["F", "°F", "degF", "C", "°C"],
      "conversion": {"F_to_C": "(x - 32) * 5/9", "C_to_F": "x * 9/5 + 32"}
    },
    "WEIGHT": {
      "description": "Weight",
      "normal_range": {"min": 50, "max": 150},
      "critical_low": 50,
      "critical_high": 500,
      "unit": "kg",
      "unit_alternatives": ["kg", "KG", "kgs", "lb", "lbs"],
      "conversion": {"lb_to_kg": "x * 0.453592", "kg_to_lb": "x / 0.453592"}
    },
    "HEIGHT": {
      "description": "Height",
      "normal_range": {"min": 150, "max": 200},
      "critical_low": 100,
      "critical_high": 250,
      "unit": "cm",
      "unit_alternatives": ["cm", "CM", "m", "ft", "in"],
      "conversion": {"in_to_cm": "x * 2.54", "cm_to_in": "x / 2.54", "m_to_cm": "x * 100"}
    }
  }
}


path: references/sample_dm.csv
content: STUDYID,USUBJID,SUBJID,SITEID,AGE,SEX,RACE,RFSTDTC,RFENDTC
ST-001,ST-001-001,001,S01,45,M,WHITE,2023-01-15,2023-06-15
ST-001,ST-001-002,002,S01,,F,ASIAN,2023-01-16,2023-06-16
ST-001,ST-001-003,003,S02,67,M,,2023-01-17,2023-06-17
ST-001,ST-001-004,004,S02,34,F,BLACK,2023-01-18,2023-06-18
ST-001,ST-001-005,005,S03,52,M,WHITE,,
ST-001,ST-001-006,006,S03,28,F,ASIAN,2023-01-20,2023-06-20


path: references/sdtm_ig_guide.md
content: # CDISC SDTM Implementation Guide Reference

## Overview

The Study Data Tabulation Model (SDTM) is a standard for organizing and formatting data to streamline processes in collection, management, analysis and reporting.

## Key Domains

### DM - Demographics
Contains demographic information about study subjects.

**Required Fields:**
- STUDYID: Study Identifier
- USUBJID: Unique Subject Identifier
- SUBJID: Subject Identifier for the Study
- RFSTDTC: Subject Reference Start Date/Time
- RFENDTC: Subject Reference End Date/Time
- SITEID: Study Site Identifier
- AGE: Age
- SEX: Sex
- RACE: Race

### LB - Laboratory Test Results
Contains laboratory test results including chemistry, hematology, urinalysis.

**Required Fields:**
- STUDYID: Study Identifier
- USUBJID: Unique Subject Identifier
- LBTESTCD: Lab Test Short Name
- LBCAT: Category for Lab Test
- LBORRES: Result or Finding in Original Units
- LBORRESU: Original Units
- LBSTRESC: Character Result/Finding in Std Format
- LBDTC: Date/Time of Specimen Collection

### VS - Vital Signs
Contains vital signs measurements like blood pressure, heart rate, temperature.

**Required Fields:**
- STUDYID: Study Identifier
- USUBJID: Unique Subject Identifier
- VSTESTCD: Vital Signs Test Short Name
- VSORRES: Result or Finding in Original Units
- VSORRESU: Original Units
- VSSTRESC: Character Result/Finding in Std Format
- VSDTC: Date/Time of Vital Signs

## Data Quality Requirements

1. **Missing Values**: Must be coded as empty, not as text like "NA" or "N/A"
2. **Dates**: ISO 8601 format (YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS)
3. **Coded Values**: Use controlled terminology
4. **Units**: Standardize to conventional units

## References

- CDISC SDTM Implementation Guide v3.4
- CDISC Controlled Terminology
- FDA Study Data Technical Conformance Guide


path: references/troubleshooting.md
content: ## Troubleshooting

**Problem: Validation fails with missing required fields**
- Symptoms: Error listing missing SDTM fields
- Causes:
  - Data export missing columns
  - Field names different from SDTM
  - Partial data extract
- Solutions:
  - Check data export settings
  - Map field names to SDTM standards
  - Export complete dataset

**Problem: Too many outliers detected**
- Symptoms: >20% of data flagged as outliers
- Causes:
  - Wrong detection method for data distribution
  - Data quality issues
  - Incorrect units
- Solutions:
  - Use domain-specific thresholds for clinical data
  - Review raw data for systematic errors
  - Verify unit consistency

**Problem: Date parsing errors**
- Symptoms: Many dates converted to NaT (Not a Time)
- Causes:
  - Mixed date formats in source
  - Non-standard date strings
  - International date confusion (DD/MM vs MM/DD)
- Solutions:
  - Standardize date format in source system
  - Specify date format explicitly
  - Manually review unparseable dates

**Problem: Imputation creates unrealistic values**
- Symptoms: Imputed values outside plausible range
- Causes:
  - Wrong imputation strategy
  - Highly skewed data
  - Different subgroups mixed
- Solutions:
  - Use median instead of mean for skewed data
  - Impute within relevant subgroups
  - Consider multiple imputation

**Problem: Cleaned data larger than original**
- Symptoms: Output file bigger than input
- Causes:
  - Outlier flag columns added
  - Date format expanded (added time)
  - String columns widened
- Solutions:
  - Normal behavior - audit columns add size
  - Compress if needed for storage
  - Archive raw and cleaned separately

**Problem: Memory errors with large datasets**
- Symptoms: "MemoryError" or system freeze
- Causes:
  - Dataset too large for available RAM
  - Inefficient data types
  - Memory leaks in processing
- Solutions:
  - Process data in chunks
  - Optimize data types (categorical, smaller floats)
  - Use cloud-based processing for very large datasets
  - Consider database-based cleaning

---



path: requirements.txt
content: numpy
pandas
scipy


path: scripts/main.py
content: #!/usr/bin/env python3
"""
Clinical Data Cleaner
Clean and standardize raw clinical trial data for SDTM compliance.

Usage:
    python main.py --input <file> --domain <DM|LB|VS> --output <file>
"""

import argparse
import json
import sys
from pathlib import Path
from datetime import datetime
from typing import Dict, List, Optional, Tuple, Any
import warnings

import pandas as pd
import numpy as np

# Try to import scipy, fall back to numpy if not available
try:
    from scipy import stats
    HAS_SCIPY = True
except ImportError:
    HAS_SCIPY = False
    stats = None

warnings.filterwarnings('ignore')


class ClinicalDataCleaner:
    """Main class for cleaning clinical trial data."""
    
    # SDTM domain required fields
    DOMAIN_FIELDS = {
        'DM': ['STUDYID', 'USUBJID', 'SUBJID', 'RFSTDTC', 'RFENDTC', 'SITEID', 'AGE', 'SEX', 'RACE'],
        'LB': ['STUDYID', 'USUBJID', 'LBTESTCD', 'LBCAT', 'LBORRES', 'LBORRESU', 'LBSTRESC', 'LBDTC'],
        'VS': ['STUDYID', 'USUBJID', 'VSTESTCD', 'VSORRES', 'VSORRESU', 'VSSTRESC', 'VSDTC']
    }
    
    # Standard outlier thresholds for clinical parameters
    OUTLIER_THRESHOLDS = {
        'LB': {
            'GLUC': {'min': 50, 'max': 500, 'unit': 'mg/dL'},
            'HGB': {'min': 5, 'max': 20, 'unit': 'g/dL'},
            'WBC': {'min': 1000, 'max': 50000, 'unit': 'cells/uL'},
            'PLAT': {'min': 20000, 'max': 1000000, 'unit': 'cells/uL'},
            'CREAT': {'min': 0.3, 'max': 15, 'unit': 'mg/dL'},
            'ALT': {'min': 5, 'max': 500, 'unit': 'U/L'},
            'AST': {'min': 5, 'max': 500, 'unit': 'U/L'},
        },
        'VS': {
            'SYSBP': {'min': 70, 'max': 220, 'unit': 'mmHg'},
            'DIABP': {'min': 40, 'max': 140, 'unit': 'mmHg'},
            'PULSE': {'min': 40, 'max': 180, 'unit': 'beats/min'},
            'TEMP': {'min': 94, 'max': 108, 'unit': 'F'},
            'WEIGHT': {'min': 50, 'max': 500, 'unit': 'kg'},
            'HEIGHT': {'min': 100, 'max': 250, 'unit': 'cm'},
        }
    }
    
    def __init__(self, domain: str, missing_strategy: str = 'median',
                 outlier_method: str = 'iqr', outlier_action: str = 'flag',
                 config_path: Optional[str] = None):
        """
        Initialize the cleaner.
        
        Args:
            domain: SDTM domain (DM, LB, VS)
            missing_strategy: How to handle missing values
            outlier_method: Method for outlier detection
            outlier_action: How to handle outliers
            config_path: Path to custom configuration file
        """
        self.domain = domain.upper()
        self.missing_strategy = missing_strategy
        self.outlier_method = outlier_method
        self.outlier_action = outlier_action
        self.config = self._load_config(config_path)
        self.cleaning_log = []
        
    def _load_config(self, config_path: Optional[str]) -> Dict:
        """Load custom configuration if provided."""
        if config_path and Path(config_path).exists():
            with open(config_path, 'r') as f:
                return json.load(f)
        return {}
    
    def load_data(self, input_path: str) -> pd.DataFrame:
        """Load data from CSV or Excel file."""
        path = Path(input_path)
        if not path.exists():
            raise FileNotFoundError(f"Input file not found: {input_path}")
        
        if path.suffix.lower() == '.csv':
            return pd.read_csv(input_path)
        elif path.suffix.lower() in ['.xlsx', '.xls']:
            return pd.read_excel(input_path)
        else:
            raise ValueError(f"Unsupported file format: {path.suffix}")
    
    def validate_domain(self, df: pd.DataFrame) -> Tuple[bool, List[str]]:
        """
        Validate that required SDTM fields are present.
        
        Returns:
            Tuple of (is_valid, missing_fields)
        """
        required = set(self.DOMAIN_FIELDS.get(self.domain, []))
        available = set(df.columns)
        missing = list(required - available)
        
        return len(missing) == 0, missing
    
    def handle_missing_values(self, df: pd.DataFrame) -> pd.DataFrame:
        """
        Handle missing values based on strategy.
        
        Strategies:
        - drop: Remove rows with any missing values
        - mean: Impute with mean (numeric only)
        - median: Impute with median (numeric only)
        - mode: Impute with mode
        - forward: Forward fill
        """
        df_clean = df.copy()
        numeric_cols = df_clean.select_dtypes(include=[np.number]).columns
        
        missing_before = df_clean.isnull().sum().sum()
        
        if self.missing_strategy == 'drop':
            df_clean = df_clean.dropna()
        elif self.missing_strategy == 'mean':
            for col in numeric_cols:
                if df_clean[col].isnull().any():
                    mean_val = df_clean[col].mean()
                    df_clean[col] = df_clean[col].fillna(mean_val)
                    self.cleaning_log.append({
                        'action': 'impute_mean',
                        'column': col,
                        'value': mean_val,
                        'timestamp': datetime.now().isoformat()
                    })
        elif self.missing_strategy == 'median':
            for col in numeric_cols:
                if df_clean[col].isnull().any():
                    median_val = df_clean[col].median()
                    df_clean[col] = df_clean[col].fillna(median_val)
                    self.cleaning_log.append({
                        'action': 'impute_median',
                        'column': col,
                        'value': median_val,
                        'timestamp': datetime.now().isoformat()
                    })
        elif self.missing_strategy == 'mode':
            for col in df_clean.columns:
                if df_clean[col].isnull().any():
                    mode_val = df_clean[col].mode()
                    if len(mode_val) > 0:
                        df_clean[col] = df_clean[col].fillna(mode_val[0])
                        self.cleaning_log.append({
                            'action': 'impute_mode',
                            'column': col,
                            'value': mode_val[0],
                            'timestamp': datetime.now().isoformat()
                        })
        elif self.missing_strategy == 'forward':
            df_clean = df_clean.fillna(method='ffill')
            self.cleaning_log.append({
                'action': 'forward_fill',
                'columns': list(df_clean.columns),
                'timestamp': datetime.now().isoformat()
            })
        
        missing_after = df_clean.isnull().sum().sum()
        print(f"Missing values: {missing_before} → {missing_after}")
        
        return df_clean
    
    def detect_outliers(self, df: pd.DataFrame) -> pd.DataFrame:
        """
        Detect outliers using specified method.
        
        Methods:
        - iqr: Interquartile Range method
        - zscore: Z-score method (|z| > 3)
        - domain: Domain-specific clinical thresholds
        """
        df_flagged = df.copy()
        outlier_col = f"{self.domain}_OUTLIER_FLAG"
        df_flagged[outlier_col] = 0
        
        numeric_cols = df.select_dtypes(include=[np.number]).columns
        
        outlier_count = 0
        
        for col in numeric_cols:
            if col in ['STUDYID', 'USUBJID', 'SUBJID']:
                continue
                
            outliers = pd.Series([False] * len(df))
            
            if self.outlier_method == 'iqr':
                Q1 = df[col].quantile(0.25)
                Q3 = df[col].quantile(0.75)
                IQR = Q3 - Q1
                lower = Q1 - 1.5 * IQR
                upper = Q3 + 1.5 * IQR
                outliers = (df[col] < lower) | (df[col] > upper)
                
            elif self.outlier_method == 'zscore':
                # Calculate z-score manually or using scipy
                col_data = df[col].dropna()
                if len(col_data) > 1:
                    mean_val = col_data.mean()
                    std_val = col_data.std()
                    if std_val > 0:
                        z_scores = np.abs((df[col] - mean_val) / std_val)
                        outliers = z_scores > 3
                
            elif self.outlier_method == 'domain':
                # Check domain-specific thresholds
                test_code_col = f"{self.domain}TESTCD"
                if test_code_col in df.columns and self.domain in self.OUTLIER_THRESHOLDS:
                    thresholds = self.OUTLIER_THRESHOLDS[self.domain]
                    for test_code, limits in thresholds.items():
                        mask = df[test_code_col] == test_code
                        if col.endswith('ORRES') or col.endswith('STRESC'):
                            val_outliers = (df[col] < limits['min']) | (df[col] > limits['max'])
                            outliers = outliers | (mask & val_outliers)
            
            outlier_count += outliers.sum()
            df_flagged.loc[outliers, outlier_col] = 1
            
            if outliers.any():
                self.cleaning_log.append({
                    'action': 'outlier_detected',
                    'column': col,
                    'method': self.outlier_method,
                    'count': int(outliers.sum()),
                    'timestamp': datetime.now().isoformat()
                })
        
        print(f"Outliers detected: {outlier_count}")
        return df_flagged
    
    def handle_outliers(self, df: pd.DataFrame) -> pd.DataFrame:
        """
        Handle outliers based on action strategy.
        
        Actions:
        - flag: Just flag them (already done in detect)
        - remove: Remove outlier rows
        - cap: Cap values at thresholds
        """
        outlier_col = f"{self.domain}_OUTLIER_FLAG"
        
        if self.outlier_action == 'flag':
            return df
        elif self.outlier_action == 'remove':
            if outlier_col in df.columns:
                before = len(df)
                df_clean = df[df[outlier_col] == 0].copy()
                after = len(df_clean)
                removed = before - after
                print(f"Removed {removed} outlier rows")
                self.cleaning_log.append({
                    'action': 'remove_outliers',
                    'rows_removed': removed,
                    'timestamp': datetime.now().isoformat()
                })
                return df_clean
        elif self.outlier_action == 'cap':
            # Cap at 1st and 99th percentiles
            numeric_cols = df.select_dtypes(include=[np.number]).columns
            for col in numeric_cols:
                if col not in ['STUDYID', 'USUBJID', outlier_col]:
                    lower = df[col].quantile(0.01)
                    upper = df[col].quantile(0.99)
                    df[col] = df[col].clip(lower, upper)
            self.cleaning_log.append({
                'action': 'cap_outliers',
                'timestamp': datetime.now().isoformat()
            })
            return df
        
        return df
    
    def standardize_dates(self, df: pd.DataFrame) -> pd.DataFrame:
        """Standardize date formats to ISO 8601."""
        df_clean = df.copy()
        date_cols = [col for col in df.columns if 'DTC' in col or 'DT' in col]
        
        for col in date_cols:
            if col in df_clean.columns:
                try:
                    df_clean[col] = pd.to_datetime(df_clean[col], errors='coerce')
                    df_clean[col] = df_clean[col].dt.strftime('%Y-%m-%dT%H:%M:%S')
                    self.cleaning_log.append({
                        'action': 'standardize_date',
                        'column': col,
                        'timestamp': datetime.now().isoformat()
                    })
                except Exception as e:
                    print(f"Warning: Could not standardize {col}: {e}")
        
        return df_clean
    
    def clean(self, df: pd.DataFrame) -> pd.DataFrame:
        """Execute full cleaning pipeline."""
        print(f"\n{'='*60}")
        print(f"Clinical Data Cleaner - {self.domain} Domain")
        print(f"{'='*60}")
        print(f"Input rows: {len(df)}")
        print(f"Input columns: {len(df.columns)}")
        
        # Validate domain
        is_valid, missing_fields = self.validate_domain(df)
        if not is_valid:
            print(f"\nWarning: Missing required fields: {missing_fields}")
            print("Continuing with available fields...")
        
        # Step 1: Handle missing values
        print(f"\n[Step 1] Handling missing values (strategy: {self.missing_strategy})")
        df = self.handle_missing_values(df)
        
        # Step 2: Detect outliers
        print(f"\n[Step 2] Detecting outliers (method: {self.outlier_method})")
        df = self.detect_outliers(df)
        
        # Step 3: Handle outliers
        print(f"\n[Step 3] Handling outliers (action: {self.outlier_action})")
        df = self.handle_outliers(df)
        
        # Step 4: Standardize dates
        print(f"\n[Step 4] Standardizing dates")
        df = self.standardize_dates(df)
        
        print(f"\n{'='*60}")
        print(f"Output rows: {len(df)}")
        print(f"Cleaning actions logged: {len(self.cleaning_log)}")
        print(f"{'='*60}\n")
        
        return df
    
    def save_report(self, output_path: str):
        """Save cleaning report to JSON."""
        report = {
            'domain': self.domain,
            'timestamp': datetime.now().isoformat(),
            'parameters': {
                'missing_strategy': self.missing_strategy,
                'outlier_method': self.outlier_method,
                'outlier_action': self.outlier_action
            },
            'actions': self.cleaning_log
        }
        
        report_path = Path(output_path).with_suffix('.report.json')
        with open(report_path, 'w') as f:
            json.dump(report, f, indent=2)
        print(f"Cleaning report saved: {report_path}")


def main():
    parser = argparse.ArgumentParser(
        description='Clean and standardize clinical trial data for SDTM compliance'
    )
    parser.add_argument('--input', '-i', required=True, help='Input data file (CSV/Excel)')
    parser.add_argument('--domain', '-d', required=True, choices=['DM', 'LB', 'VS'],
                        help='SDTM domain')
    parser.add_argument('--output', '-o', required=True, help='Output file path')
    parser.add_argument('--missing-strategy', default='median',
                        choices=['drop', 'mean', 'median', 'mode', 'forward'],
                        help='Missing value handling strategy')
    parser.add_argument('--outlier-method', default='iqr',
                        choices=['iqr', 'zscore', 'domain'],
                        help='Outlier detection method')
    parser.add_argument('--outlier-action', default='flag',
                        choices=['flag', 'remove', 'cap'],
                        help='Outlier handling action')
    parser.add_argument('--config', '-c', help='Custom configuration JSON file')
    
    args = parser.parse_args()
    
    # Initialize cleaner
    cleaner = ClinicalDataCleaner(
        domain=args.domain,
        missing_strategy=args.missing_strategy,
        outlier_method=args.outlier_method,
        outlier_action=args.outlier_action,
        config_path=args.config
    )
    
    # Load data
    df = cleaner.load_data(args.input)
    
    # Clean data
    df_cleaned = cleaner.clean(df)
    
    # Save output
    output_path = Path(args.output)
    if output_path.suffix.lower() == '.csv':
        df_cleaned.to_csv(args.output, index=False)
    elif output_path.suffix.lower() in ['.xlsx', '.xls']:
        df_cleaned.to_excel(args.output, index=False)
    else:
        # Default to CSV
        df_cleaned.to_csv(args.output, index=False)
    
    print(f"Cleaned data saved: {args.output}")
    
    # Save report
    cleaner.save_report(args.output)


if __name__ == '__main__':
    main()


path: scripts/requirements.txt
content: numpy
pandas
scipy


path: tile.json
content: {
  "name": "aipoch/clinical-data-cleaner",
  "version": "0.1.0",
  "private": true,
  "summary": "Use when cleaning clinical trial data, preparing data for FDA/EMA submission, standardizing SDTM datasets, handling missing values in clinical studies, detecting outliers in lab results, or converting raw CRF data to CDISC format.",
  "skills": {
    "clinical-data-cleaner": {
      "path": "SKILL.md"
    }
  },
  "entrypoints": {
    "skill": "SKILL.md",
    "references": [
      "references/sdtm_ig_guide.md",
      "references/common-patterns.md",
      "references/troubleshooting.md"
    ]
  }
}


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)