Skill: felix-antonio-sl/kv-migration-architect (version 1.0.0)

--- name: migration-architect description: Plan, execute and validate zero-downtime system migrations. Use when migrating databases (schema evolution, data transfer, CDC), services (strangler-fig, canary, blue-green, parallel run), or infrastructure (cloud-to-cloud, on-prem to cloud). Generates phased migration plans, compatibility analysis, rollback procedures, runbooks, stakeholder communication templates, and success metrics. compatibility: Requires Python 3.8+ for scripts in scripts/. --- # Migration Architect Convert a migration request into a structured, zero-downtime plan with tested rollback procedures. ## Activation Use this skill when the user asks to: - migrate a database schema, data set, or storage layer - move a service or API to a new implementation or platform - migrate infrastructure between clouds or from on-prem - assess migration risk, compatibility, or rollback options - generate runbooks, checklists, or stakeholder comms for a migration ## Workflow 1. **Classify** the migration type: `database` | `service` | `infrastructure` | `hybrid`. 2. **Load patterns** for that type from `{baseDir}/references/migration_patterns_catalog.md`. 3. **Run compatibility check** if schemas or APIs are provided: ```bash python {baseDir}/scripts/compatibility_checker.py --before=<old> --after=<new> ``` 4. **Identify dominant zero-downtime technique** (expand-contract, dual-write, CDC, strangler-fig, blue-green, canary). Load `{baseDir}/references/zero_downtime_techniques.md` if the best technique is unclear. 5. **Generate migration plan** with phased execution: ```bash python {baseDir}/scripts/migration_planner.py --config=<config.json> ``` 6. **Generate rollback procedures** for each phase: ```bash python {baseDir}/scripts/rollback_generator.py --plan=<plan.json> ``` 7. **Define validation checkpoints**: row counts, checksums, business-logic queries. Load `{baseDir}/references/data_reconciliation_strategies.md` for reconciliation patterns. 8. **Emit output**: phased plan + rollback procedures + runbook checklists + success metrics. ## Output Contract - Open with migration classification and dominant risk. - Emit one phased plan with explicit rollback trigger per phase. - Include pre/during/post checklists as Markdown task lists. - Declare `Functor Information Loss` (or equivalent: `Irreversible Data Loss Risk`) when a phase cannot be rolled back. - Close with success metrics (technical + business) and monitoring window recommendation. ## Risk Tiers | Tier | Criteria | Required before execution | |------|----------|--------------------------| | LOW | additive schema changes, no data transformation | staging validation | | MEDIUM | data transformation, dual-write window, service cutover | staging + load test + rollback drill | | HIGH | destructive changes, cross-cloud, compliance scope | all above + stakeholder sign-off | ## Guardrails - Never recommend irreversible steps without an explicit rollback procedure. - Always separate migration phases so each can be independently rolled back. - Flag when the target system cannot express all constraints of the source (`Information Loss`). - Do not generate feature-flag code or circuit-breaker implementations unless explicitly asked — reference the patterns in `references/` instead. ## Self Check Before emitting the plan, verify: - each phase has a rollback procedure; - validation checkpoints are defined between phases; - risk tier is declared and prerequisites are listed; - no phase assumes success of a prior phase without a checkpoint.

path: scripts/migration_planner.py
content: #!/usr/bin/env python3
"""
Migration Planner - Generate comprehensive migration plans with risk assessment

This tool analyzes migration specifications and generates detailed, phased migration plans
including pre-migration checklists, validation gates, rollback triggers, timeline estimates,
and risk matrices.

Author: Migration Architect Skill
Version: 1.0.0
License: MIT
"""

import json
import argparse
import sys
import datetime
import hashlib
import math
from typing import Dict, List, Any, Optional, Tuple
from dataclasses import dataclass, asdict
from enum import Enum


class MigrationType(Enum):
    """Migration type enumeration"""
    DATABASE = "database"
    SERVICE = "service"
    INFRASTRUCTURE = "infrastructure"
    DATA = "data"
    API = "api"


class MigrationComplexity(Enum):
    """Migration complexity levels"""
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"
    CRITICAL = "critical"


class RiskLevel(Enum):
    """Risk assessment levels"""
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"
    CRITICAL = "critical"


@dataclass
class MigrationConstraint:
    """Migration constraint definition"""
    type: str
    description: str
    impact: str
    mitigation: str


@dataclass
class MigrationPhase:
    """Individual migration phase"""
    name: str
    description: str
    duration_hours: int
    dependencies: List[str]
    validation_criteria: List[str]
    rollback_triggers: List[str]
    tasks: List[str]
    risk_level: str
    resources_required: List[str]


@dataclass
class RiskItem:
    """Individual risk assessment item"""
    category: str
    description: str
    probability: str  # low, medium, high
    impact: str  # low, medium, high
    severity: str  # low, medium, high, critical
    mitigation: str
    owner: str


@dataclass
class MigrationPlan:
    """Complete migration plan structure"""
    migration_id: str
    source_system: str
    target_system: str
    migration_type: str
    complexity: str
    estimated_duration_hours: int
    phases: List[MigrationPhase]
    risks: List[RiskItem]
    success_criteria: List[str]
    rollback_plan: Dict[str, Any]
    stakeholders: List[str]
    created_at: str


class MigrationPlanner:
    """Main migration planner class"""
    
    def __init__(self):
        self.migration_patterns = self._load_migration_patterns()
        self.risk_templates = self._load_risk_templates()
        
    def _load_migration_patterns(self) -> Dict[str, Any]:
        """Load predefined migration patterns"""
        return {
            "database": {
                "schema_change": {
                    "phases": ["preparation", "expand", "migrate", "contract", "cleanup"],
                    "base_duration": 24,
                    "complexity_multiplier": {"low": 1.0, "medium": 1.5, "high": 2.5, "critical": 4.0}
                },
                "data_migration": {
                    "phases": ["assessment", "setup", "bulk_copy", "delta_sync", "validation", "cutover"],
                    "base_duration": 48,
                    "complexity_multiplier": {"low": 1.2, "medium": 2.0, "high": 3.0, "critical": 5.0}
                }
            },
            "service": {
                "strangler_fig": {
                    "phases": ["intercept", "implement", "redirect", "validate", "retire"],
                    "base_duration": 168,  # 1 week
                    "complexity_multiplier": {"low": 0.8, "medium": 1.0, "high": 1.8, "critical": 3.0}
                },
                "parallel_run": {
                    "phases": ["setup", "deploy", "shadow", "compare", "cutover", "cleanup"],
                    "base_duration": 72,
                    "complexity_multiplier": {"low": 1.0, "medium": 1.3, "high": 2.0, "critical": 3.5}
                }
            },
            "infrastructure": {
                "cloud_migration": {
                    "phases": ["assessment", "design", "pilot", "migration", "optimization", "decommission"],
                    "base_duration": 720,  # 30 days
                    "complexity_multiplier": {"low": 0.6, "medium": 1.0, "high": 1.5, "critical": 2.5}
                },
                "on_prem_to_cloud": {
                    "phases": ["discovery", "planning", "pilot", "migration", "validation", "cutover"],
                    "base_duration": 480,  # 20 days
                    "complexity_multiplier": {"low": 0.8, "medium": 1.2, "high": 2.0, "critical": 3.0}
                }
            }
        }
    
    def _load_risk_templates(self) -> Dict[str, List[RiskItem]]:
        """Load risk templates for different migration types"""
        return {
            "database": [
                RiskItem("technical", "Data corruption during migration", "low", "critical", "high",
                        "Implement comprehensive backup and validation procedures", "DBA Team"),
                RiskItem("technical", "Extended downtime due to migration complexity", "medium", "high", "high",
                        "Use blue-green deployment and phased migration approach", "DevOps Team"),
                RiskItem("business", "Business process disruption", "medium", "high", "high",
                        "Communicate timeline and provide alternate workflows", "Business Owner"),
                RiskItem("operational", "Insufficient rollback testing", "high", "critical", "critical",
                        "Execute full rollback procedures in staging environment", "QA Team")
            ],
            "service": [
                RiskItem("technical", "Service compatibility issues", "medium", "high", "high",
                        "Implement comprehensive integration testing", "Development Team"),
                RiskItem("technical", "Performance degradation", "medium", "medium", "medium",
                        "Conduct load testing and performance benchmarking", "DevOps Team"),
                RiskItem("business", "Feature parity gaps", "high", "high", "high",
                        "Document feature mapping and acceptance criteria", "Product Owner"),
                RiskItem("operational", "Monitoring gap during transition", "medium", "medium", "medium",
                        "Set up dual monitoring and alerting systems", "SRE Team")
            ],
            "infrastructure": [
                RiskItem("technical", "Network connectivity issues", "medium", "critical", "high",
                        "Implement redundant network paths and monitoring", "Network Team"),
                RiskItem("technical", "Security configuration drift", "high", "high", "high",
                        "Automated security scanning and compliance checks", "Security Team"),
                RiskItem("business", "Cost overrun during transition", "high", "medium", "medium",
                        "Implement cost monitoring and budget alerts", "Finance Team"),
                RiskItem("operational", "Team knowledge gaps", "high", "medium", "medium",
                        "Provide training and create detailed documentation", "Platform Team")
            ]
        }
    
    def _calculate_complexity(self, spec: Dict[str, Any]) -> str:
        """Calculate migration complexity based on specification"""
        complexity_score = 0
        
        # Data volume complexity
        data_volume = spec.get("constraints", {}).get("data_volume_gb", 0)
        if data_volume > 10000:
            complexity_score += 3
        elif data_volume > 1000:
            complexity_score += 2
        elif data_volume > 100:
            complexity_score += 1
        
        # System dependencies
        dependencies = len(spec.get("constraints", {}).get("dependencies", []))
        if dependencies > 10:
            complexity_score += 3
        elif dependencies > 5:
            complexity_score += 2
        elif dependencies > 2:
            complexity_score += 1
        
        # Downtime constraints
        max_downtime = spec.get("constraints", {}).get("max_downtime_minutes", 480)
        if max_downtime < 60:
            complexity_score += 3
        elif max_downtime < 240:
            complexity_score += 2
        elif max_downtime < 480:
            complexity_score += 1
        
        # Special requirements
        special_reqs = spec.get("constraints", {}).get("special_requirements", [])
        complexity_score += len(special_reqs)
        
        if complexity_score >= 8:
            return "critical"
        elif complexity_score >= 5:
            return "high"
        elif complexity_score >= 3:
            return "medium"
        else:
            return "low"
    
    def _estimate_duration(self, migration_type: str, migration_pattern: str, complexity: str) -> int:
        """Estimate migration duration based on type, pattern, and complexity"""
        pattern_info = self.migration_patterns.get(migration_type, {}).get(migration_pattern, {})
        base_duration = pattern_info.get("base_duration", 48)
        multiplier = pattern_info.get("complexity_multiplier", {}).get(complexity, 1.5)
        
        return int(base_duration * multiplier)
    
    def _generate_phases(self, spec: Dict[str, Any]) -> List[MigrationPhase]:
        """Generate migration phases based on specification"""
        migration_type = spec.get("type")
        migration_pattern = spec.get("pattern", "")
        complexity = self._calculate_complexity(spec)
        
        pattern_info = self.migration_patterns.get(migration_type, {})
        if migration_pattern in pattern_info:
            phase_names = pattern_info[migration_pattern]["phases"]
        else:
            # Default phases based on migration type
            phase_names = {
                "database": ["preparation", "migration", "validation", "cutover"],
                "service": ["preparation", "deployment", "testing", "cutover"],
                "infrastructure": ["assessment", "preparation", "migration", "validation"]
            }.get(migration_type, ["preparation", "execution", "validation", "cleanup"])
        
        phases = []
        total_duration = self._estimate_duration(migration_type, migration_pattern, complexity)
        phase_duration = total_duration // len(phase_names)
        
        for i, phase_name in enumerate(phase_names):
            phase = self._create_phase(phase_name, phase_duration, complexity, i, phase_names)
            phases.append(phase)
        
        return phases
    
    def _create_phase(self, phase_name: str, duration: int, complexity: str, 
                     phase_index: int, all_phases: List[str]) -> MigrationPhase:
        """Create a detailed migration phase"""
        phase_templates = {
            "preparation": {
                "description": "Prepare systems and teams for migration",
                "tasks": [
                    "Backup source system",
                    "Set up monitoring and alerting",
                    "Prepare rollback procedures",
                    "Communicate migration timeline",
                    "Validate prerequisites"
                ],
                "validation_criteria": [
                    "All backups completed successfully",
                    "Monitoring systems operational",
                    "Team members briefed and ready",
                    "Rollback procedures tested"
                ],
                "risk_level": "medium"
            },
            "assessment": {
                "description": "Assess current state and migration requirements",
                "tasks": [
                    "Inventory existing systems and dependencies",
                    "Analyze data volumes and complexity",
                    "Identify integration points",
                    "Document current architecture",
                    "Create migration mapping"
                ],
                "validation_criteria": [
                    "Complete system inventory documented",
                    "Dependencies mapped and validated",
                    "Migration scope clearly defined",
                    "Resource requirements identified"
                ],
                "risk_level": "low"
            },
            "migration": {
                "description": "Execute core migration processes",
                "tasks": [
                    "Begin data/service migration",
                    "Monitor migration progress",
                    "Validate data consistency",
                    "Handle migration errors",
                    "Update configuration"
                ],
                "validation_criteria": [
                    "Migration progress within expected parameters",
                    "Data consistency checks passing",
                    "Error rates within acceptable limits",
                    "Performance metrics stable"
                ],
                "risk_level": "high"
            },
            "validation": {
                "description": "Validate migration success and system health",
                "tasks": [
                    "Execute comprehensive testing",
                    "Validate business processes",
                    "Check system performance",
                    "Verify data integrity",
                    "Confirm security controls"
                ],
                "validation_criteria": [
                    "All critical tests passing",
                    "Performance within acceptable range",
                    "Security controls functioning",
                    "Business processes operational"
                ],
                "risk_level": "medium"
            },
            "cutover": {
                "description": "Switch production traffic to new system",
                "tasks": [
                    "Update DNS/load balancer configuration",
                    "Redirect production traffic",
                    "Monitor system performance",
                    "Validate end-user experience",
                    "Confirm business operations"
                ],
                "validation_criteria": [
                    "Traffic successfully redirected",
                    "System performance stable",
                    "User experience satisfactory",
                    "Business operations normal"
                ],
                "risk_level": "critical"
            }
        }
        
        template = phase_templates.get(phase_name, {
            "description": f"Execute {phase_name} phase",
            "tasks": [f"Complete {phase_name} activities"],
            "validation_criteria": [f"{phase_name.title()} phase completed successfully"],
            "risk_level": "medium"
        })
        
        dependencies = []
        if phase_index > 0:
            dependencies.append(all_phases[phase_index - 1])
        
        rollback_triggers = [
            "Critical system failure",
            "Data corruption detected",
            "Performance degradation > 50%",
            "Business process failure"
        ]
        
        resources_required = [
            "Technical team availability",
            "System access and permissions",
            "Monitoring and alerting systems",
            "Communication channels"
        ]
        
        return MigrationPhase(
            name=phase_name,
            description=template["description"],
            duration_hours=duration,
            dependencies=dependencies,
            validation_criteria=template["validation_criteria"],
            rollback_triggers=rollback_triggers,
            tasks=template["tasks"],
            risk_level=template["risk_level"],
            resources_required=resources_required
        )
    
    def _assess_risks(self, spec: Dict[str, Any]) -> List[RiskItem]:
        """Generate risk assessment for migration"""
        migration_type = spec.get("type")
        base_risks = self.risk_templates.get(migration_type, [])
        
        # Add specification-specific risks
        additional_risks = []
        constraints = spec.get("constraints", {})
        
        if constraints.get("max_downtime_minutes", 480) < 60:
            additional_risks.append(
                RiskItem("business", "Zero-downtime requirement increases complexity", "high", "medium", "high",
                        "Implement blue-green deployment or rolling update strategy", "DevOps Team")
            )
        
        if constraints.get("data_volume_gb", 0) > 5000:
            additional_risks.append(
                RiskItem("technical", "Large data volumes may cause extended migration time", "high", "medium", "medium",
                        "Implement parallel processing and progress monitoring", "Data Team")
            )
        
        compliance_reqs = constraints.get("compliance_requirements", [])
        if compliance_reqs:
            additional_risks.append(
                RiskItem("compliance", "Regulatory compliance requirements", "medium", "high", "high",
                        "Ensure all compliance checks are integrated into migration process", "Compliance Team")
            )
        
        return base_risks + additional_risks
    
    def _generate_rollback_plan(self, phases: List[MigrationPhase]) -> Dict[str, Any]:
        """Generate comprehensive rollback plan"""
        rollback_phases = []
        
        for phase in reversed(phases):
            rollback_phase = {
                "phase": phase.name,
                "rollback_actions": [
                    f"Revert {phase.name} changes",
                    f"Restore pre-{phase.name} state",
                    f"Validate {phase.name} rollback success"
                ],
                "validation_criteria": [
                    f"System restored to pre-{phase.name} state",
                    f"All {phase.name} changes successfully reverted",
                    "System functionality confirmed"
                ],
                "estimated_time_minutes": phase.duration_hours * 15  # 25% of original phase time
            }
            rollback_phases.append(rollback_phase)
        
        return {
            "rollback_phases": rollback_phases,
            "rollback_triggers": [
                "Critical system failure",
                "Data corruption detected",
                "Migration timeline exceeded by > 50%",
                "Business-critical functionality unavailable",
                "Security breach detected",
                "Stakeholder decision to abort"
            ],
            "rollback_decision_matrix": {
                "low_severity": "Continue with monitoring",
                "medium_severity": "Assess and decide within 15 minutes",
                "high_severity": "Immediate rollback initiation",
                "critical_severity": "Emergency rollback - all hands"
            },
            "rollback_contacts": [
                "Migration Lead",
                "Technical Lead", 
                "Business Owner",
                "On-call Engineer"
            ]
        }
    
    def generate_plan(self, spec: Dict[str, Any]) -> MigrationPlan:
        """Generate complete migration plan from specification"""
        migration_id = hashlib.md5(json.dumps(spec, sort_keys=True).encode()).hexdigest()[:12]
        complexity = self._calculate_complexity(spec)
        phases = self._generate_phases(spec)
        risks = self._assess_risks(spec)
        total_duration = sum(phase.duration_hours for phase in phases)
        rollback_plan = self._generate_rollback_plan(phases)
        
        success_criteria = [
            "All data successfully migrated with 100% integrity",
            "System performance meets or exceeds baseline",
            "All business processes functioning normally",
            "No critical security vulnerabilities introduced",
            "Stakeholder acceptance criteria met",
            "Documentation and runbooks updated"
        ]
        
        stakeholders = [
            "Business Owner",
            "Technical Lead",
            "DevOps Team",
            "QA Team", 
            "Security Team",
            "End Users"
        ]
        
        return MigrationPlan(
            migration_id=migration_id,
            source_system=spec.get("source", "Unknown"),
            target_system=spec.get("target", "Unknown"),
            migration_type=spec.get("type", "Unknown"),
            complexity=complexity,
            estimated_duration_hours=total_duration,
            phases=phases,
            risks=risks,
            success_criteria=success_criteria,
            rollback_plan=rollback_plan,
            stakeholders=stakeholders,
            created_at=datetime.datetime.now().isoformat()
        )
    
    def generate_human_readable_plan(self, plan: MigrationPlan) -> str:
        """Generate human-readable migration plan"""
        output = []
        output.append("=" * 80)
        output.append(f"MIGRATION PLAN: {plan.migration_id}")
        output.append("=" * 80)
        output.append(f"Source System: {plan.source_system}")
        output.append(f"Target System: {plan.target_system}")
        output.append(f"Migration Type: {plan.migration_type.upper()}")
        output.append(f"Complexity Level: {plan.complexity.upper()}")
        output.append(f"Estimated Duration: {plan.estimated_duration_hours} hours ({plan.estimated_duration_hours/24:.1f} days)")
        output.append(f"Created: {plan.created_at}")
        output.append("")
        
        # Phases
        output.append("MIGRATION PHASES")
        output.append("-" * 40)
        for i, phase in enumerate(plan.phases, 1):
            output.append(f"{i}. {phase.name.upper()} ({phase.duration_hours}h)")
            output.append(f"   Description: {phase.description}")
            output.append(f"   Risk Level: {phase.risk_level.upper()}")
            if phase.dependencies:
                output.append(f"   Dependencies: {', '.join(phase.dependencies)}")
            output.append("   Tasks:")
            for task in phase.tasks:
                output.append(f"     • {task}")
            output.append("   Success Criteria:")
            for criteria in phase.validation_criteria:
                output.append(f"     ✓ {criteria}")
            output.append("")
        
        # Risk Assessment
        output.append("RISK ASSESSMENT")
        output.append("-" * 40)
        risk_by_severity = {}
        for risk in plan.risks:
            if risk.severity not in risk_by_severity:
                risk_by_severity[risk.severity] = []
            risk_by_severity[risk.severity].append(risk)
        
        for severity in ["critical", "high", "medium", "low"]:
            if severity in risk_by_severity:
                output.append(f"{severity.upper()} SEVERITY RISKS:")
                for risk in risk_by_severity[severity]:
                    output.append(f"  • {risk.description}")
                    output.append(f"    Category: {risk.category}")
                    output.append(f"    Probability: {risk.probability} | Impact: {risk.impact}")
                    output.append(f"    Mitigation: {risk.mitigation}")
                    output.append(f"    Owner: {risk.owner}")
                    output.append("")
        
        # Rollback Plan
        output.append("ROLLBACK STRATEGY")
        output.append("-" * 40)
        output.append("Rollback Triggers:")
        for trigger in plan.rollback_plan["rollback_triggers"]:
            output.append(f"  • {trigger}")
        output.append("")
        
        output.append("Rollback Phases:")
        for rb_phase in plan.rollback_plan["rollback_phases"]:
            output.append(f"  {rb_phase['phase'].upper()}:")
            for action in rb_phase["rollback_actions"]:
                output.append(f"    - {action}")
            output.append(f"    Estimated Time: {rb_phase['estimated_time_minutes']} minutes")
            output.append("")
        
        # Success Criteria
        output.append("SUCCESS CRITERIA")
        output.append("-" * 40)
        for criteria in plan.success_criteria:
            output.append(f"✓ {criteria}")
        output.append("")
        
        # Stakeholders
        output.append("STAKEHOLDERS")
        output.append("-" * 40)
        for stakeholder in plan.stakeholders:
            output.append(f"• {stakeholder}")
        output.append("")
        
        return "\n".join(output)


def main():
    """Main function with command line interface"""
    parser = argparse.ArgumentParser(description="Generate comprehensive migration plans")
    parser.add_argument("--input", "-i", required=True, help="Input migration specification file (JSON)")
    parser.add_argument("--output", "-o", help="Output file for migration plan (JSON)")
    parser.add_argument("--format", "-f", choices=["json", "text", "both"], default="both",
                       help="Output format")
    parser.add_argument("--validate", action="store_true", help="Validate migration specification only")
    
    args = parser.parse_args()
    
    try:
        # Load migration specification
        with open(args.input, 'r') as f:
            spec = json.load(f)
        
        # Validate required fields
        required_fields = ["type", "source", "target"]
        for field in required_fields:
            if field not in spec:
                print(f"Error: Missing required field '{field}' in specification", file=sys.stderr)
                return 1
        
        if args.validate:
            print("Migration specification is valid")
            return 0
        
        # Generate migration plan
        planner = MigrationPlanner()
        plan = planner.generate_plan(spec)
        
        # Output results
        if args.format in ["json", "both"]:
            plan_dict = asdict(plan)
            if args.output:
                with open(args.output, 'w') as f:
                    json.dump(plan_dict, f, indent=2)
                print(f"Migration plan saved to {args.output}")
            else:
                print(json.dumps(plan_dict, indent=2))
        
        if args.format in ["text", "both"]:
            human_plan = planner.generate_human_readable_plan(plan)
            text_output = args.output.replace('.json', '.txt') if args.output else None
            if text_output:
                with open(text_output, 'w') as f:
                    f.write(human_plan)
                print(f"Human-readable plan saved to {text_output}")
            else:
                print("\n" + "="*80)
                print("HUMAN-READABLE MIGRATION PLAN")
                print("="*80)
                print(human_plan)
        
    except FileNotFoundError:
        print(f"Error: Input file '{args.input}' not found", file=sys.stderr)
        return 1
    except json.JSONDecodeError as e:
        print(f"Error: Invalid JSON in input file: {e}", file=sys.stderr)
        return 1
    except Exception as e:
        print(f"Error: {e}", file=sys.stderr)
        return 1
    
    return 0


if __name__ == "__main__":
    sys.exit(main())

path: scripts/compatibility_checker.py
content: #!/usr/bin/env python3
"""
Compatibility Checker - Analyze schema and API compatibility between versions

This tool analyzes schema and API changes between versions and identifies backward
compatibility issues including breaking changes, data type mismatches, missing fields,
constraint violations, and generates migration scripts suggestions.

Author: Migration Architect Skill
Version: 1.0.0
License: MIT
"""

import json
import argparse
import sys
import re
import datetime
from typing import Dict, List, Any, Optional, Tuple, Set
from dataclasses import dataclass, asdict
from enum import Enum


class ChangeType(Enum):
    """Types of changes detected"""
    BREAKING = "breaking"
    POTENTIALLY_BREAKING = "potentially_breaking"
    NON_BREAKING = "non_breaking"
    ADDITIVE = "additive"


class CompatibilityLevel(Enum):
    """Compatibility assessment levels"""
    FULLY_COMPATIBLE = "fully_compatible"
    BACKWARD_COMPATIBLE = "backward_compatible"
    POTENTIALLY_INCOMPATIBLE = "potentially_incompatible"
    BREAKING_CHANGES = "breaking_changes"


@dataclass
class CompatibilityIssue:
    """Individual compatibility issue"""
    type: str
    severity: str
    description: str
    field_path: str
    old_value: Any
    new_value: Any
    impact: str
    suggested_migration: str
    affected_operations: List[str]


@dataclass
class MigrationScript:
    """Migration script suggestion"""
    script_type: str  # sql, api, config
    description: str
    script_content: str
    rollback_script: str
    dependencies: List[str]
    validation_query: str


@dataclass
class CompatibilityReport:
    """Complete compatibility analysis report"""
    schema_before: str
    schema_after: str
    analysis_date: str
    overall_compatibility: str
    breaking_changes_count: int
    potentially_breaking_count: int
    non_breaking_changes_count: int
    additive_changes_count: int
    issues: List[CompatibilityIssue]
    migration_scripts: List[MigrationScript]
    risk_assessment: Dict[str, Any]
    recommendations: List[str]


class SchemaCompatibilityChecker:
    """Main schema compatibility checker class"""
    
    def __init__(self):
        self.type_compatibility_matrix = self._build_type_compatibility_matrix()
        self.constraint_implications = self._build_constraint_implications()
    
    def _build_type_compatibility_matrix(self) -> Dict[str, Dict[str, str]]:
        """Build data type compatibility matrix"""
        return {
            # SQL data types compatibility
            "varchar": {
                "text": "compatible",
                "char": "potentially_breaking",  # length might be different
                "nvarchar": "compatible",
                "int": "breaking",
                "bigint": "breaking",
                "decimal": "breaking",
                "datetime": "breaking",
                "boolean": "breaking"
            },
            "int": {
                "bigint": "compatible",
                "smallint": "potentially_breaking",  # range reduction
                "decimal": "compatible",
                "float": "potentially_breaking",  # precision loss
                "varchar": "breaking",
                "boolean": "breaking"
            },
            "bigint": {
                "int": "potentially_breaking",  # range reduction
                "decimal": "compatible",
                "varchar": "breaking",
                "boolean": "breaking"
            },
            "decimal": {
                "float": "potentially_breaking",  # precision loss
                "int": "potentially_breaking",  # precision loss
                "bigint": "potentially_breaking",  # precision loss
                "varchar": "breaking",
                "boolean": "breaking"
            },
            "datetime": {
                "timestamp": "compatible",
                "date": "potentially_breaking",  # time component lost
                "varchar": "breaking",
                "int": "breaking"
            },
            "boolean": {
                "tinyint": "compatible",
                "varchar": "breaking",
                "int": "breaking"
            },
            # JSON/API field types
            "string": {
                "number": "breaking",
                "boolean": "breaking",
                "array": "breaking",
                "object": "breaking",
                "null": "potentially_breaking"
            },
            "number": {
                "string": "breaking",
                "boolean": "breaking",
                "array": "breaking",
                "object": "breaking",
                "null": "potentially_breaking"
            },
            "boolean": {
                "string": "breaking",
                "number": "breaking",
                "array": "breaking",
                "object": "breaking",
                "null": "potentially_breaking"
            },
            "array": {
                "string": "breaking",
                "number": "breaking",
                "boolean": "breaking",
                "object": "breaking",
                "null": "potentially_breaking"
            },
            "object": {
                "string": "breaking",
                "number": "breaking",
                "boolean": "breaking",
                "array": "breaking",
                "null": "potentially_breaking"
            }
        }
    
    def _build_constraint_implications(self) -> Dict[str, Dict[str, str]]:
        """Build constraint change implications"""
        return {
            "required": {
                "added": "breaking",  # Previously optional field now required
                "removed": "non_breaking"  # Previously required field now optional
            },
            "not_null": {
                "added": "breaking",  # Previously nullable now NOT NULL
                "removed": "non_breaking"  # Previously NOT NULL now nullable
            },
            "unique": {
                "added": "potentially_breaking",  # May fail if duplicates exist
                "removed": "non_breaking"  # No longer enforcing uniqueness
            },
            "primary_key": {
                "added": "breaking",  # Major structural change
                "removed": "breaking",  # Major structural change
                "modified": "breaking"  # Primary key change is always breaking
            },
            "foreign_key": {
                "added": "potentially_breaking",  # May fail if referential integrity violated
                "removed": "potentially_breaking",  # May allow orphaned records
                "modified": "breaking"  # Reference change is breaking
            },
            "check": {
                "added": "potentially_breaking",  # May fail if existing data violates check
                "removed": "non_breaking",  # No longer enforcing check
                "modified": "potentially_breaking"  # Different validation rules
            },
            "index": {
                "added": "non_breaking",  # Performance improvement
                "removed": "non_breaking",  # Performance impact only
                "modified": "non_breaking"  # Performance impact only
            }
        }
    
    def analyze_database_schema(self, before_schema: Dict[str, Any], 
                              after_schema: Dict[str, Any]) -> CompatibilityReport:
        """Analyze database schema compatibility"""
        issues = []
        migration_scripts = []
        
        before_tables = before_schema.get("tables", {})
        after_tables = after_schema.get("tables", {})
        
        # Check for removed tables
        for table_name in before_tables:
            if table_name not in after_tables:
                issues.append(CompatibilityIssue(
                    type="table_removed",
                    severity="breaking",
                    description=f"Table '{table_name}' has been removed",
                    field_path=f"tables.{table_name}",
                    old_value=before_tables[table_name],
                    new_value=None,
                    impact="All operations on this table will fail",
                    suggested_migration=f"CREATE VIEW {table_name} AS SELECT * FROM replacement_table;",
                    affected_operations=["SELECT", "INSERT", "UPDATE", "DELETE"]
                ))
        
        # Check for added tables
        for table_name in after_tables:
            if table_name not in before_tables:
                migration_scripts.append(MigrationScript(
                    script_type="sql",
                    description=f"Create new table {table_name}",
                    script_content=self._generate_create_table_sql(table_name, after_tables[table_name]),
                    rollback_script=f"DROP TABLE IF EXISTS {table_name};",
                    dependencies=[],
                    validation_query=f"SELECT COUNT(*) FROM information_schema.tables WHERE table_name = '{table_name}';"
                ))
        
        # Check for modified tables
        for table_name in set(before_tables.keys()) & set(after_tables.keys()):
            table_issues, table_scripts = self._analyze_table_changes(
                table_name, before_tables[table_name], after_tables[table_name]
            )
            issues.extend(table_issues)
            migration_scripts.extend(table_scripts)
        
        return self._build_compatibility_report(
            before_schema, after_schema, issues, migration_scripts
        )
    
    def analyze_api_schema(self, before_schema: Dict[str, Any], 
                          after_schema: Dict[str, Any]) -> CompatibilityReport:
        """Analyze REST API schema compatibility"""
        issues = []
        migration_scripts = []
        
        # Analyze endpoints
        before_paths = before_schema.get("paths", {})
        after_paths = after_schema.get("paths", {})
        
        # Check for removed endpoints
        for path in before_paths:
            if path not in after_paths:
                for method in before_paths[path]:
                    issues.append(CompatibilityIssue(
                        type="endpoint_removed",
                        severity="breaking",
                        description=f"Endpoint {method.upper()} {path} has been removed",
                        field_path=f"paths.{path}.{method}",
                        old_value=before_paths[path][method],
                        new_value=None,
                        impact="Client requests to this endpoint will fail with 404",
                        suggested_migration=f"Implement redirect to replacement endpoint or maintain backward compatibility stub",
                        affected_operations=[f"{method.upper()} {path}"]
                    ))
        
        # Check for modified endpoints
        for path in set(before_paths.keys()) & set(after_paths.keys()):
            path_issues, path_scripts = self._analyze_endpoint_changes(
                path, before_paths[path], after_paths[path]
            )
            issues.extend(path_issues)
            migration_scripts.extend(path_scripts)
        
        # Analyze data models
        before_components = before_schema.get("components", {}).get("schemas", {})
        after_components = after_schema.get("components", {}).get("schemas", {})
        
        for model_name in set(before_components.keys()) & set(after_components.keys()):
            model_issues, model_scripts = self._analyze_model_changes(
                model_name, before_components[model_name], after_components[model_name]
            )
            issues.extend(model_issues)
            migration_scripts.extend(model_scripts)
        
        return self._build_compatibility_report(
            before_schema, after_schema, issues, migration_scripts
        )
    
    def _analyze_table_changes(self, table_name: str, before_table: Dict[str, Any], 
                             after_table: Dict[str, Any]) -> Tuple[List[CompatibilityIssue], List[MigrationScript]]:
        """Analyze changes to a specific table"""
        issues = []
        scripts = []
        
        before_columns = before_table.get("columns", {})
        after_columns = after_table.get("columns", {})
        
        # Check for removed columns
        for col_name in before_columns:
            if col_name not in after_columns:
                issues.append(CompatibilityIssue(
                    type="column_removed",
                    severity="breaking",
                    description=f"Column '{col_name}' removed from table '{table_name}'",
                    field_path=f"tables.{table_name}.columns.{col_name}",
                    old_value=before_columns[col_name],
                    new_value=None,
                    impact="SELECT statements including this column will fail",
                    suggested_migration=f"ALTER TABLE {table_name} ADD COLUMN {col_name}_deprecated AS computed_value;",
                    affected_operations=["SELECT", "INSERT", "UPDATE"]
                ))
        
        # Check for added columns
        for col_name in after_columns:
            if col_name not in before_columns:
                col_def = after_columns[col_name]
                is_required = col_def.get("nullable", True) == False and col_def.get("default") is None
                
                if is_required:
                    issues.append(CompatibilityIssue(
                        type="required_column_added",
                        severity="breaking",
                        description=f"Required column '{col_name}' added to table '{table_name}'",
                        field_path=f"tables.{table_name}.columns.{col_name}",
                        old_value=None,
                        new_value=col_def,
                        impact="INSERT statements without this column will fail",
                        suggested_migration=f"Add default value or make column nullable initially",
                        affected_operations=["INSERT"]
                    ))
                
                scripts.append(MigrationScript(
                    script_type="sql",
                    description=f"Add column {col_name} to table {table_name}",
                    script_content=f"ALTER TABLE {table_name} ADD COLUMN {self._generate_column_definition(col_name, col_def)};",
                    rollback_script=f"ALTER TABLE {table_name} DROP COLUMN {col_name};",
                    dependencies=[],
                    validation_query=f"SELECT COUNT(*) FROM information_schema.columns WHERE table_name = '{table_name}' AND column_name = '{col_name}';"
                ))
        
        # Check for modified columns
        for col_name in set(before_columns.keys()) & set(after_columns.keys()):
            col_issues, col_scripts = self._analyze_column_changes(
                table_name, col_name, before_columns[col_name], after_columns[col_name]
            )
            issues.extend(col_issues)
            scripts.extend(col_scripts)
        
        # Check constraint changes
        before_constraints = before_table.get("constraints", {})
        after_constraints = after_table.get("constraints", {})
        
        constraint_issues, constraint_scripts = self._analyze_constraint_changes(
            table_name, before_constraints, after_constraints
        )
        issues.extend(constraint_issues)
        scripts.extend(constraint_scripts)
        
        return issues, scripts
    
    def _analyze_column_changes(self, table_name: str, col_name: str, 
                              before_col: Dict[str, Any], after_col: Dict[str, Any]) -> Tuple[List[CompatibilityIssue], List[MigrationScript]]:
        """Analyze changes to a specific column"""
        issues = []
        scripts = []
        
        # Check data type changes
        before_type = before_col.get("type", "").lower()
        after_type = after_col.get("type", "").lower()
        
        if before_type != after_type:
            compatibility = self.type_compatibility_matrix.get(before_type, {}).get(after_type, "breaking")
            
            if compatibility == "breaking":
                issues.append(CompatibilityIssue(
                    type="incompatible_type_change",
                    severity="breaking",
                    description=f"Column '{col_name}' type changed from {before_type} to {after_type}",
                    field_path=f"tables.{table_name}.columns.{col_name}.type",
                    old_value=before_type,
                    new_value=after_type,
                    impact="Data conversion may fail or lose precision",
                    suggested_migration=f"Add conversion logic and validate data integrity",
                    affected_operations=["SELECT", "INSERT", "UPDATE", "WHERE clauses"]
                ))
                
                scripts.append(MigrationScript(
                    script_type="sql",
                    description=f"Convert column {col_name} from {before_type} to {after_type}",
                    script_content=f"ALTER TABLE {table_name} ALTER COLUMN {col_name} TYPE {after_type} USING {col_name}::{after_type};",
                    rollback_script=f"ALTER TABLE {table_name} ALTER COLUMN {col_name} TYPE {before_type};",
                    dependencies=[f"backup_{table_name}"],
                    validation_query=f"SELECT COUNT(*) FROM {table_name} WHERE {col_name} IS NOT NULL;"
                ))
            
            elif compatibility == "potentially_breaking":
                issues.append(CompatibilityIssue(
                    type="risky_type_change",
                    severity="potentially_breaking",
                    description=f"Column '{col_name}' type changed from {before_type} to {after_type} - may lose data",
                    field_path=f"tables.{table_name}.columns.{col_name}.type",
                    old_value=before_type,
                    new_value=after_type,
                    impact="Potential data loss or precision reduction",
                    suggested_migration=f"Validate all existing data can be converted safely",
                    affected_operations=["Data integrity"]
                ))
        
        # Check nullability changes
        before_nullable = before_col.get("nullable", True)
        after_nullable = after_col.get("nullable", True)
        
        if before_nullable != after_nullable:
            if before_nullable and not after_nullable:  # null -> not null
                issues.append(CompatibilityIssue(
                    type="nullability_restriction",
                    severity="breaking",
                    description=f"Column '{col_name}' changed from nullable to NOT NULL",
                    field_path=f"tables.{table_name}.columns.{col_name}.nullable",
                    old_value=before_nullable,
                    new_value=after_nullable,
                    impact="Existing NULL values will cause constraint violations",
                    suggested_migration=f"Update NULL values to valid defaults before applying NOT NULL constraint",
                    affected_operations=["INSERT", "UPDATE"]
                ))
                
                scripts.append(MigrationScript(
                    script_type="sql",
                    description=f"Make column {col_name} NOT NULL",
                    script_content=f"""
                    -- Update NULL values first
                    UPDATE {table_name} SET {col_name} = 'DEFAULT_VALUE' WHERE {col_name} IS NULL;
                    -- Add NOT NULL constraint
                    ALTER TABLE {table_name} ALTER COLUMN {col_name} SET NOT NULL;
                    """,
                    rollback_script=f"ALTER TABLE {table_name} ALTER COLUMN {col_name} DROP NOT NULL;",
                    dependencies=[],
                    validation_query=f"SELECT COUNT(*) FROM {table_name} WHERE {col_name} IS NULL;"
                ))
        
        # Check length/precision changes
        before_length = before_col.get("length")
        after_length = after_col.get("length")
        
        if before_length and after_length and before_length != after_length:
            if after_length < before_length:
                issues.append(CompatibilityIssue(
                    type="length_reduction",
                    severity="potentially_breaking",
                    description=f"Column '{col_name}' length reduced from {before_length} to {after_length}",
                    field_path=f"tables.{table_name}.columns.{col_name}.length",
                    old_value=before_length,
                    new_value=after_length,
                    impact="Data truncation may occur for values exceeding new length",
                    suggested_migration=f"Validate no existing data exceeds new length limit",
                    affected_operations=["INSERT", "UPDATE"]
                ))
        
        return issues, scripts
    
    def _analyze_constraint_changes(self, table_name: str, before_constraints: Dict[str, Any], 
                                  after_constraints: Dict[str, Any]) -> Tuple[List[CompatibilityIssue], List[MigrationScript]]:
        """Analyze constraint changes"""
        issues = []
        scripts = []
        
        for constraint_type in ["primary_key", "foreign_key", "unique", "check"]:
            before_constraint = before_constraints.get(constraint_type, [])
            after_constraint = after_constraints.get(constraint_type, [])
            
            # Convert to sets for comparison
            before_set = set(str(c) for c in before_constraint) if isinstance(before_constraint, list) else {str(before_constraint)} if before_constraint else set()
            after_set = set(str(c) for c in after_constraint) if isinstance(after_constraint, list) else {str(after_constraint)} if after_constraint else set()
            
            # Check for removed constraints
            for constraint in before_set - after_set:
                implication = self.constraint_implications.get(constraint_type, {}).get("removed", "non_breaking")
                issues.append(CompatibilityIssue(
                    type=f"{constraint_type}_removed",
                    severity=implication,
                    description=f"{constraint_type.replace('_', ' ').title()} constraint '{constraint}' removed from table '{table_name}'",
                    field_path=f"tables.{table_name}.constraints.{constraint_type}",
                    old_value=constraint,
                    new_value=None,
                    impact=f"No longer enforcing {constraint_type} constraint",
                    suggested_migration=f"Consider application-level validation for removed constraint",
                    affected_operations=["INSERT", "UPDATE", "DELETE"]
                ))
            
            # Check for added constraints
            for constraint in after_set - before_set:
                implication = self.constraint_implications.get(constraint_type, {}).get("added", "potentially_breaking")
                issues.append(CompatibilityIssue(
                    type=f"{constraint_type}_added",
                    severity=implication,
                    description=f"New {constraint_type.replace('_', ' ')} constraint '{constraint}' added to table '{table_name}'",
                    field_path=f"tables.{table_name}.constraints.{constraint_type}",
                    old_value=None,
                    new_value=constraint,
                    impact=f"New {constraint_type} constraint may reject existing data",
                    suggested_migration=f"Validate existing data complies with new constraint",
                    affected_operations=["INSERT", "UPDATE"]
                ))
                
                scripts.append(MigrationScript(
                    script_type="sql",
                    description=f"Add {constraint_type} constraint to {table_name}",
                    script_content=f"ALTER TABLE {table_name} ADD CONSTRAINT {constraint_type}_{table_name} {constraint_type.upper()} ({constraint});",
                    rollback_script=f"ALTER TABLE {table_name} DROP CONSTRAINT {constraint_type}_{table_name};",
                    dependencies=[],
                    validation_query=f"SELECT COUNT(*) FROM information_schema.table_constraints WHERE table_name = '{table_name}' AND constraint_type = '{constraint_type.upper()}';"
                ))
        
        return issues, scripts
    
    def _analyze_endpoint_changes(self, path: str, before_endpoint: Dict[str, Any], 
                                after_endpoint: Dict[str, Any]) -> Tuple[List[CompatibilityIssue], List[MigrationScript]]:
        """Analyze changes to an API endpoint"""
        issues = []
        scripts = []
        
        for method in set(before_endpoint.keys()) & set(after_endpoint.keys()):
            before_method = before_endpoint[method]
            after_method = after_endpoint[method]
            
            # Check parameter changes
            before_params = before_method.get("parameters", [])
            after_params = after_method.get("parameters", [])
            
            before_param_names = {p["name"] for p in before_params}
            after_param_names = {p["name"] for p in after_params}
            
            # Check for removed required parameters
            for param_name in before_param_names - after_param_names:
                param = next(p for p in before_params if p["name"] == param_name)
                if param.get("required", False):
                    issues.append(CompatibilityIssue(
                        type="required_parameter_removed",
                        severity="breaking",
                        description=f"Required parameter '{param_name}' removed from {method.upper()} {path}",
                        field_path=f"paths.{path}.{method}.parameters",
                        old_value=param,
                        new_value=None,
                        impact="Client requests with this parameter will fail",
                        suggested_migration="Implement parameter validation with backward compatibility",
                        affected_operations=[f"{method.upper()} {path}"]
                    ))
            
            # Check for added required parameters
            for param_name in after_param_names - before_param_names:
                param = next(p for p in after_params if p["name"] == param_name)
                if param.get("required", False):
                    issues.append(CompatibilityIssue(
                        type="required_parameter_added",
                        severity="breaking",
                        description=f"New required parameter '{param_name}' added to {method.upper()} {path}",
                        field_path=f"paths.{path}.{method}.parameters",
                        old_value=None,
                        new_value=param,
                        impact="Client requests without this parameter will fail",
                        suggested_migration="Provide default value or make parameter optional initially",
                        affected_operations=[f"{method.upper()} {path}"]
                    ))
            
            # Check response schema changes
            before_responses = before_method.get("responses", {})
            after_responses = after_method.get("responses", {})
            
            for status_code in before_responses:
                if status_code in after_responses:
                    before_schema = before_responses[status_code].get("content", {}).get("application/json", {}).get("schema", {})
                    after_schema = after_responses[status_code].get("content", {}).get("application/json", {}).get("schema", {})
                    
                    if before_schema != after_schema:
                        issues.append(CompatibilityIssue(
                            type="response_schema_changed",
                            severity="potentially_breaking",
                            description=f"Response schema changed for {method.upper()} {path} (status {status_code})",
                            field_path=f"paths.{path}.{method}.responses.{status_code}",
                            old_value=before_schema,
                            new_value=after_schema,
                            impact="Client response parsing may fail",
                            suggested_migration="Implement versioned API responses",
                            affected_operations=[f"{method.upper()} {path}"]
                        ))
        
        return issues, scripts
    
    def _analyze_model_changes(self, model_name: str, before_model: Dict[str, Any], 
                             after_model: Dict[str, Any]) -> Tuple[List[CompatibilityIssue], List[MigrationScript]]:
        """Analyze changes to an API data model"""
        issues = []
        scripts = []
        
        before_props = before_model.get("properties", {})
        after_props = after_model.get("properties", {})
        before_required = set(before_model.get("required", []))
        after_required = set(after_model.get("required", []))
        
        # Check for removed properties
        for prop_name in set(before_props.keys()) - set(after_props.keys()):
            issues.append(CompatibilityIssue(
                type="property_removed",
                severity="breaking",
                description=f"Property '{prop_name}' removed from model '{model_name}'",
                field_path=f"components.schemas.{model_name}.properties.{prop_name}",
                old_value=before_props[prop_name],
                new_value=None,
                impact="Client code expecting this property will fail",
                suggested_migration="Use API versioning to maintain backward compatibility",
                affected_operations=["Serialization", "Deserialization"]
            ))
        
        # Check for newly required properties
        for prop_name in after_required - before_required:
            issues.append(CompatibilityIssue(
                type="property_made_required",
                severity="breaking",
                description=f"Property '{prop_name}' is now required in model '{model_name}'",
                field_path=f"components.schemas.{model_name}.required",
                old_value=list(before_required),
                new_value=list(after_required),
                impact="Client requests without this property will fail validation",
                suggested_migration="Provide default values or implement gradual rollout",
                affected_operations=["Request validation"]
            ))
        
        # Check for property type changes
        for prop_name in set(before_props.keys()) & set(after_props.keys()):
            before_type = before_props[prop_name].get("type")
            after_type = after_props[prop_name].get("type")
            
            if before_type != after_type:
                compatibility = self.type_compatibility_matrix.get(before_type, {}).get(after_type, "breaking")
                issues.append(CompatibilityIssue(
                    type="property_type_changed",
                    severity=compatibility,
                    description=f"Property '{prop_name}' type changed from {before_type} to {after_type} in model '{model_name}'",
                    field_path=f"components.schemas.{model_name}.properties.{prop_name}.type",
                    old_value=before_type,
                    new_value=after_type,
                    impact="Client serialization/deserialization may fail",
                    suggested_migration="Implement type coercion or API versioning",
                    affected_operations=["Serialization", "Deserialization"]
                ))
        
        return issues, scripts
    
    def _build_compatibility_report(self, before_schema: Dict[str, Any], after_schema: Dict[str, Any],
                                  issues: List[CompatibilityIssue], migration_scripts: List[MigrationScript]) -> CompatibilityReport:
        """Build the final compatibility report"""
        # Count issues by severity
        breaking_count = sum(1 for issue in issues if issue.severity == "breaking")
        potentially_breaking_count = sum(1 for issue in issues if issue.severity == "potentially_breaking")
        non_breaking_count = sum(1 for issue in issues if issue.severity == "non_breaking")
        additive_count = sum(1 for issue in issues if issue.type == "additive")
        
        # Determine overall compatibility
        if breaking_count > 0:
            overall_compatibility = "breaking_changes"
        elif potentially_breaking_count > 0:
            overall_compatibility = "potentially_incompatible"
        elif non_breaking_count > 0:
            overall_compatibility = "backward_compatible"
        else:
            overall_compatibility = "fully_compatible"
        
        # Generate risk assessment
        risk_assessment = {
            "overall_risk": "high" if breaking_count > 0 else "medium" if potentially_breaking_count > 0 else "low",
            "deployment_risk": "requires_coordinated_deployment" if breaking_count > 0 else "safe_independent_deployment",
            "rollback_complexity": "high" if breaking_count > 3 else "medium" if breaking_count > 0 else "low",
            "testing_requirements": ["integration_testing", "regression_testing"] + 
                                  (["data_migration_testing"] if any(s.script_type == "sql" for s in migration_scripts) else [])
        }
        
        # Generate recommendations
        recommendations = []
        if breaking_count > 0:
            recommendations.append("Implement API versioning to maintain backward compatibility")
            recommendations.append("Plan for coordinated deployment with all clients")
            recommendations.append("Implement comprehensive rollback procedures")
        
        if potentially_breaking_count > 0:
            recommendations.append("Conduct thorough testing with realistic data volumes")
            recommendations.append("Implement monitoring for migration success metrics")
        
        if migration_scripts:
            recommendations.append("Test all migration scripts in staging environment")
            recommendations.append("Implement migration progress monitoring")
        
        recommendations.append("Create detailed communication plan for stakeholders")
        recommendations.append("Implement feature flags for gradual rollout")
        
        return CompatibilityReport(
            schema_before=json.dumps(before_schema, indent=2)[:500] + "..." if len(json.dumps(before_schema)) > 500 else json.dumps(before_schema, indent=2),
            schema_after=json.dumps(after_schema, indent=2)[:500] + "..." if len(json.dumps(after_schema)) > 500 else json.dumps(after_schema, indent=2),
            analysis_date=datetime.datetime.now().isoformat(),
            overall_compatibility=overall_compatibility,
            breaking_changes_count=breaking_count,
            potentially_breaking_count=potentially_breaking_count,
            non_breaking_changes_count=non_breaking_count,
            additive_changes_count=additive_count,
            issues=issues,
            migration_scripts=migration_scripts,
            risk_assessment=risk_assessment,
            recommendations=recommendations
        )
    
    def _generate_create_table_sql(self, table_name: str, table_def: Dict[str, Any]) -> str:
        """Generate CREATE TABLE SQL statement"""
        columns = []
        for col_name, col_def in table_def.get("columns", {}).items():
            columns.append(self._generate_column_definition(col_name, col_def))
        
        return f"CREATE TABLE {table_name} (\n  " + ",\n  ".join(columns) + "\n);"
    
    def _generate_column_definition(self, col_name: str, col_def: Dict[str, Any]) -> str:
        """Generate column definition for SQL"""
        col_type = col_def.get("type", "VARCHAR(255)")
        nullable = "" if col_def.get("nullable", True) else " NOT NULL"
        default = f" DEFAULT {col_def.get('default')}" if col_def.get("default") is not None else ""
        
        return f"{col_name} {col_type}{nullable}{default}"
    
    def generate_human_readable_report(self, report: CompatibilityReport) -> str:
        """Generate human-readable compatibility report"""
        output = []
        output.append("=" * 80)
        output.append("COMPATIBILITY ANALYSIS REPORT")
        output.append("=" * 80)
        output.append(f"Analysis Date: {report.analysis_date}")
        output.append(f"Overall Compatibility: {report.overall_compatibility.upper()}")
        output.append("")
        
        # Summary
        output.append("SUMMARY")
        output.append("-" * 40)
        output.append(f"Breaking Changes: {report.breaking_changes_count}")
        output.append(f"Potentially Breaking: {report.potentially_breaking_count}")
        output.append(f"Non-Breaking Changes: {report.non_breaking_changes_count}")
        output.append(f"Additive Changes: {report.additive_changes_count}")
        output.append(f"Total Issues Found: {len(report.issues)}")
        output.append("")
        
        # Risk Assessment
        output.append("RISK ASSESSMENT")
        output.append("-" * 40)
        for key, value in report.risk_assessment.items():
            output.append(f"{key.replace('_', ' ').title()}: {value}")
        output.append("")
        
        # Issues by Severity
        issues_by_severity = {}
        for issue in report.issues:
            if issue.severity not in issues_by_severity:
                issues_by_severity[issue.severity] = []
            issues_by_severity[issue.severity].append(issue)
        
        for severity in ["breaking", "potentially_breaking", "non_breaking"]:
            if severity in issues_by_severity:
                output.append(f"{severity.upper().replace('_', ' ')} ISSUES")
                output.append("-" * 40)
                for issue in issues_by_severity[severity]:
                    output.append(f"• {issue.description}")
                    output.append(f"  Field: {issue.field_path}")
                    output.append(f"  Impact: {issue.impact}")
                    output.append(f"  Migration: {issue.suggested_migration}")
                    if issue.affected_operations:
                        output.append(f"  Affected Operations: {', '.join(issue.affected_operations)}")
                    output.append("")
        
        # Migration Scripts
        if report.migration_scripts:
            output.append("SUGGESTED MIGRATION SCRIPTS")
            output.append("-" * 40)
            for i, script in enumerate(report.migration_scripts, 1):
                output.append(f"{i}. {script.description}")
                output.append(f"   Type: {script.script_type}")
                output.append("   Script:")
                for line in script.script_content.split('\n'):
                    output.append(f"     {line}")
                output.append("")
        
        # Recommendations
        output.append("RECOMMENDATIONS")
        output.append("-" * 40)
        for i, rec in enumerate(report.recommendations, 1):
            output.append(f"{i}. {rec}")
        output.append("")
        
        return "\n".join(output)


def main():
    """Main function with command line interface"""
    parser = argparse.ArgumentParser(description="Analyze schema and API compatibility between versions")
    parser.add_argument("--before", required=True, help="Before schema file (JSON)")
    parser.add_argument("--after", required=True, help="After schema file (JSON)")
    parser.add_argument("--type", choices=["database", "api"], default="database", help="Schema type to analyze")
    parser.add_argument("--output", "-o", help="Output file for compatibility report (JSON)")
    parser.add_argument("--format", "-f", choices=["json", "text", "both"], default="both", help="Output format")
    
    args = parser.parse_args()
    
    try:
        # Load schemas
        with open(args.before, 'r') as f:
            before_schema = json.load(f)
        
        with open(args.after, 'r') as f:
            after_schema = json.load(f)
        
        # Analyze compatibility
        checker = SchemaCompatibilityChecker()
        
        if args.type == "database":
            report = checker.analyze_database_schema(before_schema, after_schema)
        else:  # api
            report = checker.analyze_api_schema(before_schema, after_schema)
        
        # Output results
        if args.format in ["json", "both"]:
            report_dict = asdict(report)
            if args.output:
                with open(args.output, 'w') as f:
                    json.dump(report_dict, f, indent=2)
                print(f"Compatibility report saved to {args.output}")
            else:
                print(json.dumps(report_dict, indent=2))
        
        if args.format in ["text", "both"]:
            human_report = checker.generate_human_readable_report(report)
            text_output = args.output.replace('.json', '.txt') if args.output else None
            if text_output:
                with open(text_output, 'w') as f:
                    f.write(human_report)
                print(f"Human-readable report saved to {text_output}")
            else:
                print("\n" + "="*80)
                print("HUMAN-READABLE COMPATIBILITY REPORT")
                print("="*80)
                print(human_report)
        
        # Return exit code based on compatibility
        if report.breaking_changes_count > 0:
            return 2  # Breaking changes found
        elif report.potentially_breaking_count > 0:
            return 1  # Potentially breaking changes found
        else:
            return 0  # No compatibility issues
            
    except FileNotFoundError as e:
        print(f"Error: File not found: {e}", file=sys.stderr)
        return 1
    except json.JSONDecodeError as e:
        print(f"Error: Invalid JSON: {e}", file=sys.stderr)
        return 1
    except Exception as e:
        print(f"Error: {e}", file=sys.stderr)
        return 1


if __name__ == "__main__":
    sys.exit(main())

path: scripts/rollback_generator.py
content: #!/usr/bin/env python3
"""
Rollback Generator - Generate comprehensive rollback procedures for migrations

This tool takes a migration plan and generates detailed rollback procedures for each phase,
including data rollback scripts, service rollback steps, validation checks, and communication
templates to ensure safe and reliable migration reversals.

Author: Migration Architect Skill
Version: 1.0.0
License: MIT
"""

import json
import argparse
import sys
import datetime
import hashlib
from typing import Dict, List, Any, Optional, Tuple
from dataclasses import dataclass, asdict
from enum import Enum


class RollbackTrigger(Enum):
    """Types of rollback triggers"""
    MANUAL = "manual"
    AUTOMATED = "automated"
    THRESHOLD_BASED = "threshold_based"
    TIME_BASED = "time_based"


class RollbackUrgency(Enum):
    """Rollback urgency levels"""
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"
    EMERGENCY = "emergency"


@dataclass
class RollbackStep:
    """Individual rollback step"""
    step_id: str
    name: str
    description: str
    script_type: str  # sql, bash, api, manual
    script_content: str
    estimated_duration_minutes: int
    dependencies: List[str]
    validation_commands: List[str]
    success_criteria: List[str]
    failure_escalation: str
    rollback_order: int


@dataclass
class RollbackPhase:
    """Rollback phase containing multiple steps"""
    phase_name: str
    description: str
    urgency_level: str
    estimated_duration_minutes: int
    prerequisites: List[str]
    steps: List[RollbackStep]
    validation_checkpoints: List[str]
    communication_requirements: List[str]
    risk_level: str


@dataclass
class RollbackTriggerCondition:
    """Conditions that trigger automatic rollback"""
    trigger_id: str
    name: str
    condition: str
    metric_threshold: Optional[Dict[str, Any]]
    evaluation_window_minutes: int
    auto_execute: bool
    escalation_contacts: List[str]


@dataclass
class DataRecoveryPlan:
    """Data recovery and restoration plan"""
    recovery_method: str  # backup_restore, point_in_time, event_replay
    backup_location: str
    recovery_scripts: List[str]
    data_validation_queries: List[str]
    estimated_recovery_time_minutes: int
    recovery_dependencies: List[str]


@dataclass
class CommunicationTemplate:
    """Communication template for rollback scenarios"""
    template_type: str  # start, progress, completion, escalation
    audience: str  # technical, business, executive, customers
    subject: str
    body: str
    urgency: str
    delivery_methods: List[str]


@dataclass
class RollbackRunbook:
    """Complete rollback runbook"""
    runbook_id: str
    migration_id: str
    created_at: str
    rollback_phases: List[RollbackPhase]
    trigger_conditions: List[RollbackTriggerCondition]
    data_recovery_plan: DataRecoveryPlan
    communication_templates: List[CommunicationTemplate]
    escalation_matrix: Dict[str, Any]
    validation_checklist: List[str]
    post_rollback_procedures: List[str]
    emergency_contacts: List[Dict[str, str]]


class RollbackGenerator:
    """Main rollback generator class"""
    
    def __init__(self):
        self.rollback_templates = self._load_rollback_templates()
        self.validation_templates = self._load_validation_templates()
        self.communication_templates = self._load_communication_templates()
    
    def _load_rollback_templates(self) -> Dict[str, Any]:
        """Load rollback script templates for different migration types"""
        return {
            "database": {
                "schema_rollback": {
                    "drop_table": "DROP TABLE IF EXISTS {table_name};",
                    "drop_column": "ALTER TABLE {table_name} DROP COLUMN IF EXISTS {column_name};",
                    "restore_column": "ALTER TABLE {table_name} ADD COLUMN {column_definition};",
                    "revert_type": "ALTER TABLE {table_name} ALTER COLUMN {column_name} TYPE {original_type};",
                    "drop_constraint": "ALTER TABLE {table_name} DROP CONSTRAINT {constraint_name};",
                    "add_constraint": "ALTER TABLE {table_name} ADD CONSTRAINT {constraint_name} {constraint_definition};"
                },
                "data_rollback": {
                    "restore_backup": "pg_restore -d {database_name} -c {backup_file}",
                    "point_in_time_recovery": "SELECT pg_create_restore_point('pre_migration_{timestamp}');",
                    "delete_migrated_data": "DELETE FROM {table_name} WHERE migration_batch_id = '{batch_id}';",
                    "restore_original_values": "UPDATE {table_name} SET {column_name} = backup_{column_name} WHERE migration_flag = true;"
                }
            },
            "service": {
                "deployment_rollback": {
                    "rollback_blue_green": "kubectl patch service {service_name} -p '{\"spec\":{\"selector\":{\"version\":\"blue\"}}}'",
                    "rollback_canary": "kubectl scale deployment {service_name}-canary --replicas=0",
                    "restore_previous_version": "kubectl rollout undo deployment/{service_name} --to-revision={revision_number}",
                    "update_load_balancer": "aws elbv2 modify-rule --rule-arn {rule_arn} --actions Type=forward,TargetGroupArn={original_target_group}"
                },
                "configuration_rollback": {
                    "restore_config_map": "kubectl apply -f {original_config_file}",
                    "revert_feature_flags": "curl -X PUT {feature_flag_api}/flags/{flag_name} -d '{\"enabled\": false}'",
                    "restore_environment_vars": "kubectl set env deployment/{deployment_name} {env_var_name}={original_value}"
                }
            },
            "infrastructure": {
                "cloud_rollback": {
                    "revert_terraform": "terraform apply -target={resource_name} {rollback_plan_file}",
                    "restore_dns": "aws route53 change-resource-record-sets --hosted-zone-id {zone_id} --change-batch file://{rollback_dns_changes}",
                    "rollback_security_groups": "aws ec2 authorize-security-group-ingress --group-id {group_id} --protocol {protocol} --port {port} --cidr {cidr}",
                    "restore_iam_policies": "aws iam put-role-policy --role-name {role_name} --policy-name {policy_name} --policy-document file://{original_policy}"
                },
                "network_rollback": {
                    "restore_routing": "aws ec2 replace-route --route-table-id {route_table_id} --destination-cidr-block {cidr} --gateway-id {original_gateway}",
                    "revert_load_balancer": "aws elbv2 modify-load-balancer --load-balancer-arn {lb_arn} --scheme {original_scheme}",
                    "restore_firewall_rules": "aws ec2 revoke-security-group-ingress --group-id {group_id} --protocol {protocol} --port {port} --source-group {source_group}"
                }
            }
        }
    
    def _load_validation_templates(self) -> Dict[str, List[str]]:
        """Load validation command templates"""
        return {
            "database": [
                "SELECT COUNT(*) FROM {table_name};",
                "SELECT COUNT(*) FROM information_schema.tables WHERE table_name = '{table_name}';",
                "SELECT COUNT(*) FROM information_schema.columns WHERE table_name = '{table_name}' AND column_name = '{column_name}';",
                "SELECT COUNT(DISTINCT {primary_key}) FROM {table_name};",
                "SELECT MAX({timestamp_column}) FROM {table_name};"
            ],
            "service": [
                "curl -f {health_check_url}",
                "kubectl get pods -l app={service_name} --field-selector=status.phase=Running",
                "kubectl logs deployment/{service_name} --tail=100 | grep -i error",
                "curl -f {service_endpoint}/api/v1/status"
            ],
            "infrastructure": [
                "aws ec2 describe-instances --instance-ids {instance_id} --query 'Reservations[*].Instances[*].State.Name'",
                "nslookup {domain_name}",
                "curl -I {load_balancer_url}",
                "aws elbv2 describe-target-health --target-group-arn {target_group_arn}"
            ]
        }
    
    def _load_communication_templates(self) -> Dict[str, Dict[str, str]]:
        """Load communication templates"""
        return {
            "rollback_start": {
                "technical": {
                    "subject": "ROLLBACK INITIATED: {migration_name}",
                    "body": """Team,

We have initiated rollback for migration: {migration_name}
Rollback ID: {rollback_id}
Start Time: {start_time}
Estimated Duration: {estimated_duration}

Reason: {rollback_reason}

Current Status: Rolling back phase {current_phase}

Next Updates: Every 15 minutes or upon phase completion

Actions Required:
- Monitor system health dashboards
- Stand by for escalation if needed
- Do not make manual changes during rollback

Incident Commander: {incident_commander}
"""
                },
                "business": {
                    "subject": "System Rollback In Progress - {system_name}",
                    "body": """Business Stakeholders,

We are currently performing a planned rollback of the {system_name} migration due to {rollback_reason}.

Impact: {business_impact}
Expected Resolution: {estimated_completion_time}
Affected Services: {affected_services}

We will provide updates every 30 minutes.

Contact: {business_contact}
"""
                },
                "executive": {
                    "subject": "EXEC ALERT: Critical System Rollback - {system_name}",
                    "body": """Executive Team,

A critical rollback is in progress for {system_name}.

Summary:
- Rollback Reason: {rollback_reason}
- Business Impact: {business_impact}
- Expected Resolution: {estimated_completion_time}
- Customer Impact: {customer_impact}

We are following established procedures and will update hourly.

Escalation: {escalation_contact}
"""
                }
            },
            "rollback_complete": {
                "technical": {
                    "subject": "ROLLBACK COMPLETED: {migration_name}",
                    "body": """Team,

Rollback has been successfully completed for migration: {migration_name}

Summary:
- Start Time: {start_time}
- End Time: {end_time}
- Duration: {actual_duration}
- Phases Completed: {completed_phases}

Validation Results:
{validation_results}

System Status: {system_status}

Next Steps:
- Continue monitoring for 24 hours
- Post-rollback review scheduled for {review_date}
- Root cause analysis to begin

All clear to resume normal operations.

Incident Commander: {incident_commander}
"""
                }
            }
        }
    
    def generate_rollback_runbook(self, migration_plan: Dict[str, Any]) -> RollbackRunbook:
        """Generate comprehensive rollback runbook from migration plan"""
        runbook_id = f"rb_{hashlib.md5(str(migration_plan).encode()).hexdigest()[:8]}"
        migration_id = migration_plan.get("migration_id", "unknown")
        migration_type = migration_plan.get("migration_type", "unknown")
        
        # Generate rollback phases (reverse order of migration phases)
        rollback_phases = self._generate_rollback_phases(migration_plan)
        
        # Generate trigger conditions
        trigger_conditions = self._generate_trigger_conditions(migration_plan)
        
        # Generate data recovery plan
        data_recovery_plan = self._generate_data_recovery_plan(migration_plan)
        
        # Generate communication templates
        communication_templates = self._generate_communication_templates(migration_plan)
        
        # Generate escalation matrix
        escalation_matrix = self._generate_escalation_matrix(migration_plan)
        
        # Generate validation checklist
        validation_checklist = self._generate_validation_checklist(migration_plan)
        
        # Generate post-rollback procedures
        post_rollback_procedures = self._generate_post_rollback_procedures(migration_plan)
        
        # Generate emergency contacts
        emergency_contacts = self._generate_emergency_contacts(migration_plan)
        
        return RollbackRunbook(
            runbook_id=runbook_id,
            migration_id=migration_id,
            created_at=datetime.datetime.now().isoformat(),
            rollback_phases=rollback_phases,
            trigger_conditions=trigger_conditions,
            data_recovery_plan=data_recovery_plan,
            communication_templates=communication_templates,
            escalation_matrix=escalation_matrix,
            validation_checklist=validation_checklist,
            post_rollback_procedures=post_rollback_procedures,
            emergency_contacts=emergency_contacts
        )
    
    def _generate_rollback_phases(self, migration_plan: Dict[str, Any]) -> List[RollbackPhase]:
        """Generate rollback phases from migration plan"""
        migration_phases = migration_plan.get("phases", [])
        migration_type = migration_plan.get("migration_type", "unknown")
        rollback_phases = []
        
        # Reverse the order of migration phases for rollback
        for i, phase in enumerate(reversed(migration_phases)):
            if isinstance(phase, dict):
                phase_name = phase.get("name", f"phase_{i}")
                phase_duration = phase.get("duration_hours", 2) * 60  # Convert to minutes
                phase_risk = phase.get("risk_level", "medium")
            else:
                phase_name = str(phase)
                phase_duration = 120  # Default 2 hours
                phase_risk = "medium"
            
            rollback_steps = self._generate_rollback_steps(phase_name, migration_type, i)
            
            rollback_phase = RollbackPhase(
                phase_name=f"rollback_{phase_name}",
                description=f"Rollback changes made during {phase_name} phase",
                urgency_level=self._calculate_urgency(phase_risk),
                estimated_duration_minutes=phase_duration // 2,  # Rollback typically faster
                prerequisites=self._get_rollback_prerequisites(phase_name, i),
                steps=rollback_steps,
                validation_checkpoints=self._get_validation_checkpoints(phase_name, migration_type),
                communication_requirements=self._get_communication_requirements(phase_name, phase_risk),
                risk_level=phase_risk
            )
            
            rollback_phases.append(rollback_phase)
        
        return rollback_phases
    
    def _generate_rollback_steps(self, phase_name: str, migration_type: str, phase_index: int) -> List[RollbackStep]:
        """Generate specific rollback steps for a phase"""
        steps = []
        templates = self.rollback_templates.get(migration_type, {})
        
        if migration_type == "database":
            if "migration" in phase_name.lower() or "cutover" in phase_name.lower():
                # Data rollback steps
                steps.extend([
                    RollbackStep(
                        step_id=f"rb_data_{phase_index}_01",
                        name="Stop data migration processes",
                        description="Halt all ongoing data migration processes",
                        script_type="sql",
                        script_content="-- Stop migration processes\nSELECT pg_cancel_backend(pid) FROM pg_stat_activity WHERE query LIKE '%migration%';",
                        estimated_duration_minutes=5,
                        dependencies=[],
                        validation_commands=["SELECT COUNT(*) FROM pg_stat_activity WHERE query LIKE '%migration%';"],
                        success_criteria=["No active migration processes"],
                        failure_escalation="Contact DBA immediately",
                        rollback_order=1
                    ),
                    RollbackStep(
                        step_id=f"rb_data_{phase_index}_02",
                        name="Restore from backup",
                        description="Restore database from pre-migration backup",
                        script_type="bash",
                        script_content=templates.get("data_rollback", {}).get("restore_backup", "pg_restore -d {database_name} -c {backup_file}"),
                        estimated_duration_minutes=30,
                        dependencies=[f"rb_data_{phase_index}_01"],
                        validation_commands=["SELECT COUNT(*) FROM information_schema.tables;"],
                        success_criteria=["Database restored successfully", "All expected tables present"],
                        failure_escalation="Escalate to senior DBA and infrastructure team",
                        rollback_order=2
                    )
                ])
            
            if "preparation" in phase_name.lower():
                # Schema rollback steps
                steps.append(
                    RollbackStep(
                        step_id=f"rb_schema_{phase_index}_01",
                        name="Drop migration artifacts",
                        description="Remove temporary migration tables and procedures",
                        script_type="sql",
                        script_content="-- Drop migration artifacts\nDROP TABLE IF EXISTS migration_log;\nDROP PROCEDURE IF EXISTS migrate_data();",
                        estimated_duration_minutes=5,
                        dependencies=[],
                        validation_commands=["SELECT COUNT(*) FROM information_schema.tables WHERE table_name LIKE '%migration%';"],
                        success_criteria=["No migration artifacts remain"],
                        failure_escalation="Manual cleanup required",
                        rollback_order=1
                    )
                )
        
        elif migration_type == "service":
            if "cutover" in phase_name.lower():
                # Service rollback steps
                steps.extend([
                    RollbackStep(
                        step_id=f"rb_service_{phase_index}_01",
                        name="Redirect traffic back to old service",
                        description="Update load balancer to route traffic back to previous service version",
                        script_type="bash",
                        script_content=templates.get("deployment_rollback", {}).get("update_load_balancer", "aws elbv2 modify-rule --rule-arn {rule_arn} --actions Type=forward,TargetGroupArn={original_target_group}"),
                        estimated_duration_minutes=2,
                        dependencies=[],
                        validation_commands=["curl -f {health_check_url}"],
                        success_criteria=["Traffic routing to original service", "Health checks passing"],
                        failure_escalation="Emergency procedure - manual traffic routing",
                        rollback_order=1
                    ),
                    RollbackStep(
                        step_id=f"rb_service_{phase_index}_02",
                        name="Rollback service deployment",
                        description="Revert to previous service deployment version",
                        script_type="bash",
                        script_content=templates.get("deployment_rollback", {}).get("restore_previous_version", "kubectl rollout undo deployment/{service_name} --to-revision={revision_number}"),
                        estimated_duration_minutes=10,
                        dependencies=[f"rb_service_{phase_index}_01"],
                        validation_commands=["kubectl get pods -l app={service_name} --field-selector=status.phase=Running"],
                        success_criteria=["Previous version deployed", "All pods running"],
                        failure_escalation="Manual pod management required",
                        rollback_order=2
                    )
                ])
        
        elif migration_type == "infrastructure":
            steps.extend([
                RollbackStep(
                    step_id=f"rb_infra_{phase_index}_01",
                    name="Revert infrastructure changes",
                    description="Apply terraform plan to revert infrastructure to previous state",
                    script_type="bash",
                    script_content=templates.get("cloud_rollback", {}).get("revert_terraform", "terraform apply -target={resource_name} {rollback_plan_file}"),
                    estimated_duration_minutes=15,
                    dependencies=[],
                    validation_commands=["terraform plan -detailed-exitcode"],
                    success_criteria=["Infrastructure matches previous state", "No planned changes"],
                    failure_escalation="Manual infrastructure review required",
                    rollback_order=1
                ),
                RollbackStep(
                    step_id=f"rb_infra_{phase_index}_02",
                    name="Restore DNS configuration",
                    description="Revert DNS changes to point back to original infrastructure",
                    script_type="bash",
                    script_content=templates.get("cloud_rollback", {}).get("restore_dns", "aws route53 change-resource-record-sets --hosted-zone-id {zone_id} --change-batch file://{rollback_dns_changes}"),
                    estimated_duration_minutes=10,
                    dependencies=[f"rb_infra_{phase_index}_01"],
                    validation_commands=["nslookup {domain_name}"],
                    success_criteria=["DNS resolves to original endpoints"],
                    failure_escalation="Contact DNS administrator",
                    rollback_order=2
                )
            ])
        
        # Add generic validation step for all migration types
        steps.append(
            RollbackStep(
                step_id=f"rb_validate_{phase_index}_final",
                name="Validate rollback completion",
                description=f"Comprehensive validation that {phase_name} rollback completed successfully",
                script_type="manual",
                script_content="Execute validation checklist for this phase",
                estimated_duration_minutes=10,
                dependencies=[step.step_id for step in steps],
                validation_commands=self.validation_templates.get(migration_type, []),
                success_criteria=[f"{phase_name} fully rolled back", "All validation checks pass"],
                failure_escalation=f"Investigate {phase_name} rollback failures",
                rollback_order=99
            )
        )
        
        return steps
    
    def _generate_trigger_conditions(self, migration_plan: Dict[str, Any]) -> List[RollbackTriggerCondition]:
        """Generate automatic rollback trigger conditions"""
        triggers = []
        migration_type = migration_plan.get("migration_type", "unknown")
        
        # Generic triggers for all migration types
        triggers.extend([
            RollbackTriggerCondition(
                trigger_id="error_rate_spike",
                name="Error Rate Spike",
                condition="error_rate > baseline * 5 for 5 minutes",
                metric_threshold={
                    "metric": "error_rate",
                    "operator": "greater_than",
                    "value": "baseline_error_rate * 5",
                    "duration_minutes": 5
                },
                evaluation_window_minutes=5,
                auto_execute=True,
                escalation_contacts=["on_call_engineer", "migration_lead"]
            ),
            RollbackTriggerCondition(
                trigger_id="response_time_degradation",
                name="Response Time Degradation",
                condition="p95_response_time > baseline * 3 for 10 minutes",
                metric_threshold={
                    "metric": "p95_response_time",
                    "operator": "greater_than",
                    "value": "baseline_p95 * 3",
                    "duration_minutes": 10
                },
                evaluation_window_minutes=10,
                auto_execute=False,
                escalation_contacts=["performance_team", "migration_lead"]
            ),
            RollbackTriggerCondition(
                trigger_id="availability_drop",
                name="Service Availability Drop",
                condition="availability < 95% for 2 minutes",
                metric_threshold={
                    "metric": "availability",
                    "operator": "less_than",
                    "value": 0.95,
                    "duration_minutes": 2
                },
                evaluation_window_minutes=2,
                auto_execute=True,
                escalation_contacts=["sre_team", "incident_commander"]
            )
        ])
        
        # Migration-type specific triggers
        if migration_type == "database":
            triggers.extend([
                RollbackTriggerCondition(
                    trigger_id="data_integrity_failure",
                    name="Data Integrity Check Failure",
                    condition="data_validation_failures > 0",
                    metric_threshold={
                        "metric": "data_validation_failures",
                        "operator": "greater_than",
                        "value": 0,
                        "duration_minutes": 1
                    },
                    evaluation_window_minutes=1,
                    auto_execute=True,
                    escalation_contacts=["dba_team", "data_team"]
                ),
                RollbackTriggerCondition(
                    trigger_id="migration_progress_stalled",
                    name="Migration Progress Stalled",
                    condition="migration_progress unchanged for 30 minutes",
                    metric_threshold={
                        "metric": "migration_progress_rate",
                        "operator": "equals",
                        "value": 0,
                        "duration_minutes": 30
                    },
                    evaluation_window_minutes=30,
                    auto_execute=False,
                    escalation_contacts=["migration_team", "dba_team"]
                )
            ])
        
        elif migration_type == "service":
            triggers.extend([
                RollbackTriggerCondition(
                    trigger_id="cpu_utilization_spike",
                    name="CPU Utilization Spike",
                    condition="cpu_utilization > 90% for 15 minutes",
                    metric_threshold={
                        "metric": "cpu_utilization",
                        "operator": "greater_than",
                        "value": 0.90,
                        "duration_minutes": 15
                    },
                    evaluation_window_minutes=15,
                    auto_execute=False,
                    escalation_contacts=["devops_team", "infrastructure_team"]
                ),
                RollbackTriggerCondition(
                    trigger_id="memory_leak_detected",
                    name="Memory Leak Detected",
                    condition="memory_usage increasing continuously for 20 minutes",
                    metric_threshold={
                        "metric": "memory_growth_rate",
                        "operator": "greater_than",
                        "value": "1MB/minute",
                        "duration_minutes": 20
                    },
                    evaluation_window_minutes=20,
                    auto_execute=True,
                    escalation_contacts=["development_team", "sre_team"]
                )
            ])
        
        return triggers
    
    def _generate_data_recovery_plan(self, migration_plan: Dict[str, Any]) -> DataRecoveryPlan:
        """Generate data recovery plan"""
        migration_type = migration_plan.get("migration_type", "unknown")
        
        if migration_type == "database":
            return DataRecoveryPlan(
                recovery_method="point_in_time",
                backup_location="/backups/pre_migration_{migration_id}_{timestamp}.sql",
                recovery_scripts=[
                    "pg_restore -d production -c /backups/pre_migration_backup.sql",
                    "SELECT pg_create_restore_point('rollback_point');",
                    "VACUUM ANALYZE; -- Refresh statistics after restore"
                ],
                data_validation_queries=[
                    "SELECT COUNT(*) FROM critical_business_table;",
                    "SELECT MAX(created_at) FROM audit_log;",
                    "SELECT COUNT(DISTINCT user_id) FROM user_sessions;",
                    "SELECT SUM(amount) FROM financial_transactions WHERE date = CURRENT_DATE;"
                ],
                estimated_recovery_time_minutes=45,
                recovery_dependencies=["database_instance_running", "backup_file_accessible"]
            )
        else:
            return DataRecoveryPlan(
                recovery_method="backup_restore",
                backup_location="/backups/pre_migration_state",
                recovery_scripts=[
                    "# Restore configuration files from backup",
                    "cp -r /backups/pre_migration_state/config/* /app/config/",
                    "# Restart services with previous configuration",
                    "systemctl restart application_service"
                ],
                data_validation_queries=[
                    "curl -f http://localhost:8080/health",
                    "curl -f http://localhost:8080/api/status"
                ],
                estimated_recovery_time_minutes=20,
                recovery_dependencies=["service_stopped", "backup_accessible"]
            )
    
    def _generate_communication_templates(self, migration_plan: Dict[str, Any]) -> List[CommunicationTemplate]:
        """Generate communication templates for rollback scenarios"""
        templates = []
        base_templates = self.communication_templates
        
        # Rollback start notifications
        for audience in ["technical", "business", "executive"]:
            if audience in base_templates["rollback_start"]:
                template_data = base_templates["rollback_start"][audience]
                templates.append(CommunicationTemplate(
                    template_type="rollback_start",
                    audience=audience,
                    subject=template_data["subject"],
                    body=template_data["body"],
                    urgency="high" if audience == "executive" else "medium",
                    delivery_methods=["email", "slack"] if audience == "technical" else ["email"]
                ))
        
        # Rollback completion notifications
        for audience in ["technical", "business"]:
            if audience in base_templates.get("rollback_complete", {}):
                template_data = base_templates["rollback_complete"][audience]
                templates.append(CommunicationTemplate(
                    template_type="rollback_complete",
                    audience=audience,
                    subject=template_data["subject"],
                    body=template_data["body"],
                    urgency="medium",
                    delivery_methods=["email", "slack"] if audience == "technical" else ["email"]
                ))
        
        # Emergency escalation template
        templates.append(CommunicationTemplate(
            template_type="emergency_escalation",
            audience="executive",
            subject="CRITICAL: Rollback Emergency - {migration_name}",
            body="""CRITICAL SITUATION - IMMEDIATE ATTENTION REQUIRED

Migration: {migration_name}
Issue: Rollback procedure has encountered critical failures

Current Status: {current_status}
Failed Components: {failed_components}
Business Impact: {business_impact}
Customer Impact: {customer_impact}

Immediate Actions:
1. Emergency response team activated
2. {emergency_action_1}
3. {emergency_action_2}

War Room: {war_room_location}
Bridge Line: {conference_bridge}

Next Update: {next_update_time}

Incident Commander: {incident_commander}
Executive On-Call: {executive_on_call}
""",
            urgency="emergency",
            delivery_methods=["email", "sms", "phone_call"]
        ))
        
        return templates
    
    def _generate_escalation_matrix(self, migration_plan: Dict[str, Any]) -> Dict[str, Any]:
        """Generate escalation matrix for different failure scenarios"""
        return {
            "level_1": {
                "trigger": "Single component failure",
                "response_time_minutes": 5,
                "contacts": ["on_call_engineer", "migration_lead"],
                "actions": ["Investigate issue", "Attempt automated remediation", "Monitor closely"]
            },
            "level_2": {
                "trigger": "Multiple component failures or single critical failure",
                "response_time_minutes": 2,
                "contacts": ["senior_engineer", "team_lead", "devops_lead"],
                "actions": ["Initiate rollback", "Establish war room", "Notify stakeholders"]
            },
            "level_3": {
                "trigger": "System-wide failure or data corruption",
                "response_time_minutes": 1,
                "contacts": ["engineering_manager", "cto", "incident_commander"],
                "actions": ["Emergency rollback", "All hands on deck", "Executive notification"]
            },
            "emergency": {
                "trigger": "Business-critical failure with customer impact",
                "response_time_minutes": 0,
                "contacts": ["ceo", "cto", "head_of_operations"],
                "actions": ["Emergency procedures", "Customer communication", "Media preparation if needed"]
            }
        }
    
    def _generate_validation_checklist(self, migration_plan: Dict[str, Any]) -> List[str]:
        """Generate comprehensive validation checklist"""
        migration_type = migration_plan.get("migration_type", "unknown")
        
        base_checklist = [
            "Verify system is responding to health checks",
            "Confirm error rates are within normal parameters",
            "Validate response times meet SLA requirements",
            "Check all critical business processes are functioning",
            "Verify monitoring and alerting systems are operational",
            "Confirm no data corruption has occurred",
            "Validate security controls are functioning properly",
            "Check backup systems are working correctly",
            "Verify integration points with downstream systems",
            "Confirm user authentication and authorization working"
        ]
        
        if migration_type == "database":
            base_checklist.extend([
                "Validate database schema matches expected state",
                "Confirm referential integrity constraints",
                "Check database performance metrics",
                "Verify data consistency across related tables",
                "Validate indexes and statistics are optimal",
                "Confirm transaction logs are clean",
                "Check database connections and connection pooling"
            ])
        
        elif migration_type == "service":
            base_checklist.extend([
                "Verify service discovery is working correctly",
                "Confirm load balancing is distributing traffic properly",
                "Check service-to-service communication",
                "Validate API endpoints are responding correctly",
                "Confirm feature flags are in correct state",
                "Check resource utilization (CPU, memory, disk)",
                "Verify container orchestration is healthy"
            ])
        
        elif migration_type == "infrastructure":
            base_checklist.extend([
                "Verify network connectivity between components",
                "Confirm DNS resolution is working correctly",
                "Check firewall rules and security groups",
                "Validate load balancer configuration",
                "Confirm SSL/TLS certificates are valid",
                "Check storage systems are accessible",
                "Verify backup and disaster recovery systems"
            ])
        
        return base_checklist
    
    def _generate_post_rollback_procedures(self, migration_plan: Dict[str, Any]) -> List[str]:
        """Generate post-rollback procedures"""
        return [
            "Monitor system stability for 24-48 hours post-rollback",
            "Conduct thorough post-rollback testing of all critical paths",
            "Review and analyze rollback metrics and timing",
            "Document lessons learned and rollback procedure improvements",
            "Schedule post-mortem meeting with all stakeholders",
            "Update rollback procedures based on actual experience",
            "Communicate rollback completion to all stakeholders",
            "Archive rollback logs and artifacts for future reference",
            "Review and update monitoring thresholds if needed",
            "Plan for next migration attempt with improved procedures",
            "Conduct security review to ensure no vulnerabilities introduced",
            "Update disaster recovery procedures if affected by rollback",
            "Review capacity planning based on rollback resource usage",
            "Update documentation with rollback experience and timings"
        ]
    
    def _generate_emergency_contacts(self, migration_plan: Dict[str, Any]) -> List[Dict[str, str]]:
        """Generate emergency contact list"""
        return [
            {
                "role": "Incident Commander",
                "name": "TBD - Assigned during migration",
                "primary_phone": "+1-XXX-XXX-XXXX",
                "email": "[REDACTED_SECRET]",
                "backup_contact": "[REDACTED_SECRET]"
            },
            {
                "role": "Technical Lead",
                "name": "TBD - Migration technical owner",
                "primary_phone": "+1-XXX-XXX-XXXX",
                "email": "[REDACTED_SECRET]",
                "backup_contact": "[REDACTED_SECRET]"
            },
            {
                "role": "Business Owner",
                "name": "TBD - Business stakeholder",
                "primary_phone": "+1-XXX-XXX-XXXX",
                "email": "[REDACTED_SECRET]",
                "backup_contact": "[REDACTED_SECRET]"
            },
            {
                "role": "On-Call Engineer",
                "name": "Current on-call rotation",
                "primary_phone": "+1-XXX-XXX-XXXX",
                "email": "[REDACTED_SECRET]",
                "backup_contact": "[REDACTED_SECRET]"
            },
            {
                "role": "Executive Escalation",
                "name": "CTO/VP Engineering",
                "primary_phone": "+1-XXX-XXX-XXXX",
                "email": "[REDACTED_SECRET]",
                "backup_contact": "[REDACTED_SECRET]"
            }
        ]
    
    def _calculate_urgency(self, risk_level: str) -> str:
        """Calculate rollback urgency based on risk level"""
        risk_to_urgency = {
            "low": "low",
            "medium": "medium", 
            "high": "high",
            "critical": "emergency"
        }
        return risk_to_urgency.get(risk_level, "medium")
    
    def _get_rollback_prerequisites(self, phase_name: str, phase_index: int) -> List[str]:
        """Get prerequisites for rollback phase"""
        prerequisites = [
            "Incident commander assigned and briefed",
            "All team members notified of rollback initiation",
            "Monitoring systems confirmed operational",
            "Backup systems verified and accessible"
        ]
        
        if phase_index > 0:
            prerequisites.append("Previous rollback phase completed successfully")
        
        if "cutover" in phase_name.lower():
            prerequisites.extend([
                "Traffic redirection capabilities confirmed",
                "Load balancer configuration backed up",
                "DNS changes prepared for quick execution"
            ])
        
        if "data" in phase_name.lower() or "migration" in phase_name.lower():
            prerequisites.extend([
                "Database backup verified and accessible",
                "Data validation queries prepared",
                "Database administrator on standby"
            ])
        
        return prerequisites
    
    def _get_validation_checkpoints(self, phase_name: str, migration_type: str) -> List[str]:
        """Get validation checkpoints for rollback phase"""
        checkpoints = [
            f"{phase_name} rollback steps completed",
            "System health checks passing",
            "No critical errors in logs",
            "Key metrics within acceptable ranges"
        ]
        
        validation_commands = self.validation_templates.get(migration_type, [])
        checkpoints.extend([f"Validation command passed: {cmd[:50]}..." for cmd in validation_commands[:3]])
        
        return checkpoints
    
    def _get_communication_requirements(self, phase_name: str, risk_level: str) -> List[str]:
        """Get communication requirements for rollback phase"""
        base_requirements = [
            "Notify incident commander of phase start/completion",
            "Update rollback status dashboard",
            "Log all actions and decisions"
        ]
        
        if risk_level in ["high", "critical"]:
            base_requirements.extend([
                "Notify all stakeholders of phase progress",
                "Update executive team if rollback extends beyond expected time",
                "Prepare customer communication if needed"
            ])
        
        if "cutover" in phase_name.lower():
            base_requirements.append("Immediate notification when traffic is redirected")
        
        return base_requirements
    
    def generate_human_readable_runbook(self, runbook: RollbackRunbook) -> str:
        """Generate human-readable rollback runbook"""
        output = []
        output.append("=" * 80)
        output.append(f"ROLLBACK RUNBOOK: {runbook.runbook_id}")
        output.append("=" * 80)
        output.append(f"Migration ID: {runbook.migration_id}")
        output.append(f"Created: {runbook.created_at}")
        output.append("")
        
        # Emergency Contacts
        output.append("EMERGENCY CONTACTS")
        output.append("-" * 40)
        for contact in runbook.emergency_contacts:
            output.append(f"{contact['role']}: {contact['name']}")
            output.append(f"  Phone: {contact['primary_phone']}")
            output.append(f"  Email: {contact['email']}")
            output.append(f"  Backup: {contact['backup_contact']}")
            output.append("")
        
        # Escalation Matrix
        output.append("ESCALATION MATRIX")
        output.append("-" * 40)
        for level, details in runbook.escalation_matrix.items():
            output.append(f"{level.upper()}:")
            output.append(f"  Trigger: {details['trigger']}")
            output.append(f"  Response Time: {details['response_time_minutes']} minutes")
            output.append(f"  Contacts: {', '.join(details['contacts'])}")
            output.append(f"  Actions: {', '.join(details['actions'])}")
            output.append("")
        
        # Rollback Trigger Conditions
        output.append("AUTOMATIC ROLLBACK TRIGGERS")
        output.append("-" * 40)
        for trigger in runbook.trigger_conditions:
            output.append(f"• {trigger.name}")
            output.append(f"  Condition: {trigger.condition}")
            output.append(f"  Auto-Execute: {'Yes' if trigger.auto_execute else 'No'}")
            output.append(f"  Evaluation Window: {trigger.evaluation_window_minutes} minutes")
            output.append(f"  Contacts: {', '.join(trigger.escalation_contacts)}")
            output.append("")
        
        # Rollback Phases
        output.append("ROLLBACK PHASES")
        output.append("-" * 40)
        for i, phase in enumerate(runbook.rollback_phases, 1):
            output.append(f"{i}. {phase.phase_name.upper()}")
            output.append(f"   Description: {phase.description}")
            output.append(f"   Urgency: {phase.urgency_level.upper()}")
            output.append(f"   Duration: {phase.estimated_duration_minutes} minutes")
            output.append(f"   Risk Level: {phase.risk_level.upper()}")
            
            if phase.prerequisites:
                output.append("   Prerequisites:")
                for prereq in phase.prerequisites:
                    output.append(f"     ✓ {prereq}")
            
            output.append("   Steps:")
            for step in sorted(phase.steps, key=lambda x: x.rollback_order):
                output.append(f"     {step.rollback_order}. {step.name}")
                output.append(f"        Duration: {step.estimated_duration_minutes} min")
                output.append(f"        Type: {step.script_type}")
                if step.script_content and step.script_type != "manual":
                    output.append("        Script:")
                    for line in step.script_content.split('\n')[:3]:  # Show first 3 lines
                        output.append(f"          {line}")
                    if len(step.script_content.split('\n')) > 3:
                        output.append("          ...")
                output.append(f"        Success Criteria: {', '.join(step.success_criteria)}")
                output.append("")
            
            if phase.validation_checkpoints:
                output.append("   Validation Checkpoints:")
                for checkpoint in phase.validation_checkpoints:
                    output.append(f"     ☐ {checkpoint}")
            output.append("")
        
        # Data Recovery Plan
        output.append("DATA RECOVERY PLAN")
        output.append("-" * 40)
        drp = runbook.data_recovery_plan
        output.append(f"Recovery Method: {drp.recovery_method}")
        output.append(f"Backup Location: {drp.backup_location}")
        output.append(f"Estimated Recovery Time: {drp.estimated_recovery_time_minutes} minutes")
        output.append("Recovery Scripts:")
        for script in drp.recovery_scripts:
            output.append(f"  • {script}")
        output.append("Validation Queries:")
        for query in drp.data_validation_queries:
            output.append(f"  • {query}")
        output.append("")
        
        # Validation Checklist
        output.append("POST-ROLLBACK VALIDATION CHECKLIST")
        output.append("-" * 40)
        for i, item in enumerate(runbook.validation_checklist, 1):
            output.append(f"{i:2d}. ☐ {item}")
        output.append("")
        
        # Post-Rollback Procedures
        output.append("POST-ROLLBACK PROCEDURES")
        output.append("-" * 40)
        for i, procedure in enumerate(runbook.post_rollback_procedures, 1):
            output.append(f"{i:2d}. {procedure}")
        output.append("")
        
        return "\n".join(output)


def main():
    """Main function with command line interface"""
    parser = argparse.ArgumentParser(description="Generate comprehensive rollback runbooks from migration plans")
    parser.add_argument("--input", "-i", required=True, help="Input migration plan file (JSON)")
    parser.add_argument("--output", "-o", help="Output file for rollback runbook (JSON)")
    parser.add_argument("--format", "-f", choices=["json", "text", "both"], default="both", help="Output format")
    
    args = parser.parse_args()
    
    try:
        # Load migration plan
        with open(args.input, 'r') as f:
            migration_plan = json.load(f)
        
        # Validate required fields
        if "migration_id" not in migration_plan and "source" not in migration_plan:
            print("Error: Migration plan must contain migration_id or source field", file=sys.stderr)
            return 1
        
        # Generate rollback runbook
        generator = RollbackGenerator()
        runbook = generator.generate_rollback_runbook(migration_plan)
        
        # Output results
        if args.format in ["json", "both"]:
            runbook_dict = asdict(runbook)
            if args.output:
                with open(args.output, 'w') as f:
                    json.dump(runbook_dict, f, indent=2)
                print(f"Rollback runbook saved to {args.output}")
            else:
                print(json.dumps(runbook_dict, indent=2))
        
        if args.format in ["text", "both"]:
            human_runbook = generator.generate_human_readable_runbook(runbook)
            text_output = args.output.replace('.json', '.txt') if args.output else None
            if text_output:
                with open(text_output, 'w') as f:
                    f.write(human_runbook)
                print(f"Human-readable runbook saved to {text_output}")
            else:
                print("\n" + "="*80)
                print("HUMAN-READABLE ROLLBACK RUNBOOK")
                print("="*80)
                print(human_runbook)
        
    except FileNotFoundError:
        print(f"Error: Input file '{args.input}' not found", file=sys.stderr)
        return 1
    except json.JSONDecodeError as e:
        print(f"Error: Invalid JSON in input file: {e}", file=sys.stderr)
        return 1
    except Exception as e:
        print(f"Error: {e}", file=sys.stderr)
        return 1
    
    return 0


if __name__ == "__main__":
    sys.exit(main())

path: references/migration_patterns_catalog.md
content: # Migration Patterns Catalog

## Overview

This catalog provides detailed descriptions of proven migration patterns, their use cases, implementation guidelines, and best practices. Each pattern includes code examples, diagrams, and lessons learned from real-world implementations.

## Database Migration Patterns

### 1. Expand-Contract Pattern

**Use Case:** Schema evolution with zero downtime
**Complexity:** Medium
**Risk Level:** Low-Medium

#### Description
The Expand-Contract pattern allows for schema changes without downtime by following a three-phase approach:

1. **Expand:** Add new schema elements alongside existing ones
2. **Migrate:** Dual-write to both old and new schema during transition
3. **Contract:** Remove old schema elements after validation

#### Implementation Steps

```sql
-- Phase 1: Expand
ALTER TABLE users ADD COLUMN email_new VARCHAR(255);
CREATE INDEX CONCURRENTLY idx_users_email_new ON users(email_new);

-- Phase 2: Migrate (Application Code)
-- Write to both columns during transition period
INSERT INTO users (name, email, email_new) VALUES (?, ?, ?);

-- Backfill existing data
UPDATE users SET email_new = email WHERE email_new IS NULL;

-- Phase 3: Contract (after validation)
ALTER TABLE users DROP COLUMN email;
ALTER TABLE users RENAME COLUMN email_new TO email;
```

#### Pros and Cons
**Pros:**
- Zero downtime deployments
- Safe rollback at any point
- Gradual transition with validation

**Cons:**
- Increased storage during transition
- More complex application logic
- Extended migration timeline

### 2. Parallel Schema Pattern

**Use Case:** Major database restructuring
**Complexity:** High
**Risk Level:** Medium

#### Description
Run new and old schemas in parallel, using feature flags to gradually route traffic to the new schema while maintaining the ability to rollback quickly.

#### Implementation Example

```python
class DatabaseRouter:
    def __init__(self, feature_flag_service):
        self.feature_flags = feature_flag_service
        self.old_db = OldDatabaseConnection()
        self.new_db = NewDatabaseConnection()
    
    def route_query(self, user_id, query_type):
        if self.feature_flags.is_enabled("new_schema", user_id):
            return self.new_db.execute(query_type)
        else:
            return self.old_db.execute(query_type)
    
    def dual_write(self, data):
        # Write to both databases for consistency
        success_old = self.old_db.write(data)
        success_new = self.new_db.write(transform_data(data))
        
        if not (success_old and success_new):
            # Handle partial failures
            self.handle_dual_write_failure(data, success_old, success_new)
```

#### Best Practices
- Implement data consistency checks between schemas
- Use circuit breakers for automatic failover
- Monitor performance impact of dual writes
- Plan for data reconciliation processes

### 3. Event Sourcing Migration

**Use Case:** Migrating systems with complex business logic
**Complexity:** High
**Risk Level:** Medium-High

#### Description
Capture all changes as events during migration, enabling replay and reconciliation capabilities.

#### Event Store Schema
```sql
CREATE TABLE migration_events (
    event_id UUID PRIMARY KEY,
    aggregate_id UUID NOT NULL,
    event_type VARCHAR(100) NOT NULL,
    event_data JSONB NOT NULL,
    event_version INTEGER NOT NULL,
    occurred_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    processed_at TIMESTAMP WITH TIME ZONE
);
```

#### Migration Event Handler
```python
class MigrationEventHandler:
    def __init__(self, old_store, new_store):
        self.old_store = old_store
        self.new_store = new_store
        self.event_log = []
    
    def handle_update(self, entity_id, old_data, new_data):
        # Log the change as an event
        event = MigrationEvent(
            entity_id=entity_id,
            event_type="entity_migrated",
            old_data=old_data,
            new_data=new_data,
            timestamp=datetime.now()
        )
        
        self.event_log.append(event)
        
        # Apply to new store
        success = self.new_store.update(entity_id, new_data)
        
        if not success:
            # Mark for retry
            event.status = "failed"
            self.schedule_retry(event)
        
        return success
    
    def replay_events(self, from_timestamp=None):
        """Replay events for reconciliation"""
        events = self.get_events_since(from_timestamp)
        for event in events:
            self.apply_event(event)
```

## Service Migration Patterns

### 1. Strangler Fig Pattern

**Use Case:** Legacy system replacement
**Complexity:** Medium-High
**Risk Level:** Medium

#### Description
Gradually replace legacy functionality by intercepting calls and routing them to new services, eventually "strangling" the legacy system.

#### Implementation Architecture

```yaml
# API Gateway Configuration
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: user-service-migration
spec:
  http:
  - match:
    - headers:
        migration-flag:
          exact: "new"
    route:
    - destination:
        host: user-service-v2
  - route:
    - destination:
        host: user-service-v1
```

#### Strangler Proxy Implementation

```python
class StranglerProxy:
    def __init__(self):
        self.legacy_service = LegacyUserService()
        self.new_service = NewUserService()
        self.feature_flags = FeatureFlagService()
    
    def handle_request(self, request):
        route = self.determine_route(request)
        
        if route == "new":
            return self.handle_with_new_service(request)
        elif route == "both":
            return self.handle_with_both_services(request)
        else:
            return self.handle_with_legacy_service(request)
    
    def determine_route(self, request):
        user_id = request.get('user_id')
        
        if self.feature_flags.is_enabled("new_user_service", user_id):
            if self.feature_flags.is_enabled("dual_write", user_id):
                return "both"
            else:
                return "new"
        else:
            return "legacy"
```

### 2. Parallel Run Pattern

**Use Case:** Risk mitigation for critical services
**Complexity:** Medium
**Risk Level:** Low-Medium

#### Description
Run both old and new services simultaneously, comparing outputs to validate correctness before switching traffic.

#### Implementation

```python
class ParallelRunManager:
    def __init__(self):
        self.primary_service = PrimaryService()
        self.candidate_service = CandidateService()
        self.comparator = ResponseComparator()
        self.metrics = MetricsCollector()
    
    async def parallel_execute(self, request):
        # Execute both services concurrently
        primary_task = asyncio.create_task(
            self.primary_service.process(request)
        )
        candidate_task = asyncio.create_task(
            self.candidate_service.process(request)
        )
        
        # Always wait for primary
        primary_result = await primary_task
        
        try:
            # Wait for candidate with timeout
            candidate_result = await asyncio.wait_for(
                candidate_task, timeout=5.0
            )
            
            # Compare results
            comparison = self.comparator.compare(
                primary_result, candidate_result
            )
            
            # Record metrics
            self.metrics.record_comparison(comparison)
            
        except asyncio.TimeoutError:
            self.metrics.record_timeout("candidate")
        except Exception as e:
            self.metrics.record_error("candidate", str(e))
        
        # Always return primary result
        return primary_result
```

### 3. Blue-Green Deployment Pattern

**Use Case:** Zero-downtime service updates
**Complexity:** Low-Medium
**Risk Level:** Low

#### Description
Maintain two identical production environments (blue and green), switching traffic between them for deployments.

#### Kubernetes Implementation

```yaml
# Blue Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-blue
  labels:
    version: blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: blue
  template:
    metadata:
      labels:
        app: myapp
        version: blue
    spec:
      containers:
      - name: app
        image: myapp:v1.0.0

---
# Green Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-green
  labels:
    version: green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: green
  template:
    metadata:
      labels:
        app: myapp
        version: green
    spec:
      containers:
      - name: app
        image: myapp:v2.0.0

---
# Service (switches between blue and green)
apiVersion: v1
kind: Service
metadata:
  name: app-service
spec:
  selector:
    app: myapp
    version: blue  # Change to green for deployment
  ports:
  - port: 80
    targetPort: 8080
```

## Infrastructure Migration Patterns

### 1. Lift and Shift Pattern

**Use Case:** Quick cloud migration with minimal changes
**Complexity:** Low-Medium
**Risk Level:** Low

#### Description
Migrate applications to cloud infrastructure with minimal or no code changes, focusing on infrastructure compatibility.

#### Migration Checklist

```yaml
Pre-Migration Assessment:
  - inventory_current_infrastructure:
      - servers_and_specifications
      - network_configuration
      - storage_requirements
      - security_configurations
  - identify_dependencies:
      - database_connections
      - external_service_integrations
      - file_system_dependencies
  - assess_compatibility:
      - operating_system_versions
      - runtime_dependencies
      - license_requirements

Migration Execution:
  - provision_target_infrastructure:
      - compute_instances
      - storage_volumes
      - network_configuration
      - security_groups
  - migrate_data:
      - database_backup_restore
      - file_system_replication
      - configuration_files
  - update_configurations:
      - connection_strings
      - environment_variables
      - dns_records
  - validate_functionality:
      - application_health_checks
      - end_to_end_testing
      - performance_validation
```

### 2. Hybrid Cloud Migration

**Use Case:** Gradual cloud adoption with on-premises integration
**Complexity:** High
**Risk Level:** Medium-High

#### Description
Maintain some components on-premises while migrating others to cloud, requiring secure connectivity and data synchronization.

#### Network Architecture

```hcl
# Terraform configuration for hybrid connectivity
resource "aws_vpc" "main" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true
}

resource "aws_vpn_gateway" "main" {
  vpc_id = aws_vpc.main.id
  
  tags = {
    Name = "hybrid-vpn-gateway"
  }
}

resource "aws_customer_gateway" "main" {
  bgp_asn    = 65000
  ip_address = var.on_premises_public_ip
  type       = "ipsec.1"
  
  tags = {
    Name = "on-premises-gateway"
  }
}

resource "aws_vpn_connection" "main" {
  vpn_gateway_id      = aws_vpn_gateway.main.id
  customer_gateway_id = aws_customer_gateway.main.id
  type                = "ipsec.1"
  static_routes_only  = true
}
```

#### Data Synchronization Pattern

```python
class HybridDataSync:
    def __init__(self):
        self.on_prem_db = OnPremiseDatabase()
        self.cloud_db = CloudDatabase()
        self.sync_log = SyncLogManager()
    
    async def bidirectional_sync(self):
        """Synchronize data between on-premises and cloud"""
        
        # Get last sync timestamp
        last_sync = self.sync_log.get_last_sync_time()
        
        # Sync on-prem changes to cloud
        on_prem_changes = self.on_prem_db.get_changes_since(last_sync)
        for change in on_prem_changes:
            await self.apply_change_to_cloud(change)
        
        # Sync cloud changes to on-prem
        cloud_changes = self.cloud_db.get_changes_since(last_sync)
        for change in cloud_changes:
            await self.apply_change_to_on_prem(change)
        
        # Handle conflicts
        conflicts = self.detect_conflicts(on_prem_changes, cloud_changes)
        for conflict in conflicts:
            await self.resolve_conflict(conflict)
        
        # Update sync timestamp
        self.sync_log.record_sync_completion()
    
    async def apply_change_to_cloud(self, change):
        """Apply on-premises change to cloud database"""
        try:
            if change.operation == "INSERT":
                await self.cloud_db.insert(change.table, change.data)
            elif change.operation == "UPDATE":
                await self.cloud_db.update(change.table, change.key, change.data)
            elif change.operation == "DELETE":
                await self.cloud_db.delete(change.table, change.key)
                
            self.sync_log.record_success(change.id, "cloud")
            
        except Exception as e:
            self.sync_log.record_failure(change.id, "cloud", str(e))
            raise
```

### 3. Multi-Cloud Migration

**Use Case:** Avoiding vendor lock-in or regulatory requirements
**Complexity:** Very High
**Risk Level:** High

#### Description
Distribute workloads across multiple cloud providers for resilience, compliance, or cost optimization.

#### Service Mesh Configuration

```yaml
# Istio configuration for multi-cloud service mesh
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: aws-service
spec:
  hosts:
  - aws-service.company.com
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: multi-cloud-routing
spec:
  hosts:
  - user-service
  http:
  - match:
    - headers:
        region:
          exact: "us-east"
    route:
    - destination:
        host: aws-service.company.com
      weight: 100
  - match:
    - headers:
        region:
          exact: "eu-west"
    route:
    - destination:
        host: gcp-service.company.com
      weight: 100
  - route:  # Default routing
    - destination:
        host: user-service
        subset: local
      weight: 80
    - destination:
        host: aws-service.company.com
      weight: 20
```

## Feature Flag Patterns

### 1. Progressive Rollout Pattern

**Use Case:** Gradual feature deployment with risk mitigation
**Implementation:**

```python
class ProgressiveRollout:
    def __init__(self, feature_name):
        self.feature_name = feature_name
        self.rollout_percentage = 0
        self.user_buckets = {}
        
    def is_enabled_for_user(self, user_id):
        # Consistent user bucketing
        user_hash = hashlib.md5(f"{self.feature_name}:{user_id}".encode()).hexdigest()
        bucket = int(user_hash, 16) % 100
        
        return bucket < self.rollout_percentage
    
    def increase_rollout(self, target_percentage, step_size=10):
        """Gradually increase rollout percentage"""
        while self.rollout_percentage < target_percentage:
            self.rollout_percentage = min(
                self.rollout_percentage + step_size,
                target_percentage
            )
            
            # Monitor metrics before next increase
            yield self.rollout_percentage
            time.sleep(300)  # Wait 5 minutes between increases
```

### 2. Circuit Breaker Pattern

**Use Case:** Automatic fallback during migration issues

```python
class MigrationCircuitBreaker:
    def __init__(self, failure_threshold=5, timeout=60):
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.timeout = timeout
        self.last_failure_time = None
        self.state = 'CLOSED'  # CLOSED, OPEN, HALF_OPEN
    
    def call_new_service(self, request):
        if self.state == 'OPEN':
            if self.should_attempt_reset():
                self.state = 'HALF_OPEN'
            else:
                return self.fallback_to_legacy(request)
        
        try:
            response = self.new_service.process(request)
            self.on_success()
            return response
        except Exception as e:
            self.on_failure()
            return self.fallback_to_legacy(request)
    
    def on_success(self):
        self.failure_count = 0
        self.state = 'CLOSED'
    
    def on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        
        if self.failure_count >= self.failure_threshold:
            self.state = 'OPEN'
    
    def should_attempt_reset(self):
        return (time.time() - self.last_failure_time) >= self.timeout
```

## Migration Anti-Patterns

### 1. Big Bang Migration (Anti-Pattern)

**Why to Avoid:**
- High risk of complete system failure
- Difficult to rollback
- Extended downtime
- All-or-nothing deployment

**Better Alternative:** Use incremental migration patterns like Strangler Fig or Parallel Run.

### 2. No Rollback Plan (Anti-Pattern)

**Why to Avoid:**
- Cannot recover from failures
- Increases business risk
- Panic-driven decisions during issues

**Better Alternative:** Always implement comprehensive rollback procedures before migration.

### 3. Insufficient Testing (Anti-Pattern)

**Why to Avoid:**
- Unknown compatibility issues
- Performance degradation
- Data corruption risks

**Better Alternative:** Implement comprehensive testing at each migration phase.

## Pattern Selection Matrix

| Migration Type | Complexity | Downtime Tolerance | Recommended Pattern |
|---------------|------------|-------------------|-------------------|
| Schema Change | Low | Zero | Expand-Contract |
| Schema Change | High | Zero | Parallel Schema |
| Service Replace | Medium | Zero | Strangler Fig |
| Service Update | Low | Zero | Blue-Green |
| Data Migration | High | Some | Event Sourcing |
| Infrastructure | Low | Some | Lift and Shift |
| Infrastructure | High | Zero | Hybrid Cloud |

## Success Metrics

### Technical Metrics
- Migration completion rate
- System availability during migration
- Performance impact (response time, throughput)
- Error rate changes
- Rollback execution time

### Business Metrics
- Customer impact score
- Revenue protection
- Time to value realization
- Stakeholder satisfaction

### Operational Metrics
- Team efficiency
- Knowledge transfer effectiveness
- Post-migration support requirements
- Documentation completeness

## Lessons Learned

### Common Pitfalls
1. **Underestimating data dependencies** - Always map all data relationships
2. **Insufficient monitoring** - Implement comprehensive observability before migration
3. **Poor communication** - Keep all stakeholders informed throughout the process
4. **Rushed timelines** - Allow adequate time for testing and validation
5. **Ignoring performance impact** - Benchmark before and after migration

### Best Practices
1. **Start with low-risk migrations** - Build confidence and experience
2. **Automate everything possible** - Reduce human error and increase repeatability
3. **Test rollback procedures** - Ensure you can recover from any failure
4. **Monitor continuously** - Use real-time dashboards and alerting
5. **Document everything** - Create comprehensive runbooks and documentation

This catalog serves as a reference for selecting appropriate migration patterns based on specific requirements, risk tolerance, and technical constraints.

path: references/data_reconciliation_strategies.md
content: # Data Reconciliation Strategies

## Overview

Data reconciliation is the process of ensuring data consistency and integrity across systems during and after migrations. This document provides comprehensive strategies, tools, and implementation patterns for detecting, measuring, and correcting data discrepancies in migration scenarios.

## Core Principles

### 1. Eventually Consistent
Accept that perfect real-time consistency may not be achievable during migrations, but ensure eventual consistency through reconciliation processes.

### 2. Idempotent Operations
All reconciliation operations must be safe to run multiple times without causing additional issues.

### 3. Audit Trail
Maintain detailed logs of all reconciliation actions for compliance and debugging.

### 4. Non-Destructive
Reconciliation should prefer addition over deletion, and always maintain backups before corrections.

## Types of Data Inconsistencies

### 1. Missing Records
Records that exist in source but not in target system.

### 2. Extra Records
Records that exist in target but not in source system.

### 3. Field Mismatches
Records exist in both systems but with different field values.

### 4. Referential Integrity Violations
Foreign key relationships that are broken during migration.

### 5. Temporal Inconsistencies
Data with incorrect timestamps or ordering.

### 6. Schema Drift
Structural differences between source and target schemas.

## Detection Strategies

### 1. Row Count Validation

#### Simple Count Comparison
```sql
-- Compare total row counts
SELECT 
    'source' as system, 
    COUNT(*) as row_count 
FROM source_table
UNION ALL
SELECT 
    'target' as system, 
    COUNT(*) as row_count 
FROM target_table;
```

#### Filtered Count Comparison
```sql
-- Compare counts with business logic filters
WITH source_counts AS (
    SELECT 
        status,
        created_date::date as date,
        COUNT(*) as count
    FROM source_orders
    WHERE created_date >= '2024-01-01'
    GROUP BY status, created_date::date
),
target_counts AS (
    SELECT 
        status,
        created_date::date as date,
        COUNT(*) as count
    FROM target_orders
    WHERE created_date >= '2024-01-01'
    GROUP BY status, created_date::date
)
SELECT 
    COALESCE(s.status, t.status) as status,
    COALESCE(s.date, t.date) as date,
    COALESCE(s.count, 0) as source_count,
    COALESCE(t.count, 0) as target_count,
    COALESCE(s.count, 0) - COALESCE(t.count, 0) as difference
FROM source_counts s
FULL OUTER JOIN target_counts t 
    ON s.status = t.status AND s.date = t.date
WHERE COALESCE(s.count, 0) != COALESCE(t.count, 0);
```

### 2. Checksum-Based Validation

#### Record-Level Checksums
```python
import hashlib
import json

class RecordChecksum:
    def __init__(self, exclude_fields=None):
        self.exclude_fields = exclude_fields or ['updated_at', 'version']
    
    def calculate_checksum(self, record):
        """Calculate MD5 checksum for a database record"""
        # Remove excluded fields and sort for consistency
        filtered_record = {
            k: v for k, v in record.items() 
            if k not in self.exclude_fields
        }
        
        # Convert to sorted JSON string for consistent hashing
        normalized = json.dumps(filtered_record, sort_keys=True, default=str)
        
        return hashlib.md5(normalized.encode('utf-8')).hexdigest()
    
    def compare_records(self, source_record, target_record):
        """Compare two records using checksums"""
        source_checksum = self.calculate_checksum(source_record)
        target_checksum = self.calculate_checksum(target_record)
        
        return {
            'match': source_checksum == target_checksum,
            'source_checksum': source_checksum,
            'target_checksum': target_checksum
        }

# Usage example
checksum_calculator = RecordChecksum(exclude_fields=['updated_at', 'migration_flag'])

source_records = fetch_records_from_source()
target_records = fetch_records_from_target()

mismatches = []
for source_id, source_record in source_records.items():
    if source_id in target_records:
        comparison = checksum_calculator.compare_records(
            source_record, target_records[source_id]
        )
        if not comparison['match']:
            mismatches.append({
                'record_id': source_id,
                'source_checksum': comparison['source_checksum'],
                'target_checksum': comparison['target_checksum']
            })
```

#### Aggregate Checksums
```sql
-- Calculate aggregate checksums for data validation
WITH source_aggregates AS (
    SELECT 
        DATE_TRUNC('day', created_at) as day,
        status,
        COUNT(*) as record_count,
        SUM(amount) as total_amount,
        MD5(STRING_AGG(CAST(id AS VARCHAR) || ':' || CAST(amount AS VARCHAR), '|' ORDER BY id)) as checksum
    FROM source_transactions
    GROUP BY DATE_TRUNC('day', created_at), status
),
target_aggregates AS (
    SELECT 
        DATE_TRUNC('day', created_at) as day,
        status,
        COUNT(*) as record_count,
        SUM(amount) as total_amount,
        MD5(STRING_AGG(CAST(id AS VARCHAR) || ':' || CAST(amount AS VARCHAR), '|' ORDER BY id)) as checksum
    FROM target_transactions
    GROUP BY DATE_TRUNC('day', created_at), status
)
SELECT 
    COALESCE(s.day, t.day) as day,
    COALESCE(s.status, t.status) as status,
    COALESCE(s.record_count, 0) as source_count,
    COALESCE(t.record_count, 0) as target_count,
    COALESCE(s.total_amount, 0) as source_amount,
    COALESCE(t.total_amount, 0) as target_amount,
    s.checksum as source_checksum,
    t.checksum as target_checksum,
    CASE WHEN s.checksum = t.checksum THEN 'MATCH' ELSE 'MISMATCH' END as status
FROM source_aggregates s
FULL OUTER JOIN target_aggregates t 
    ON s.day = t.day AND s.status = t.status
WHERE s.checksum != t.checksum OR s.checksum IS NULL OR t.checksum IS NULL;
```

### 3. Delta Detection

#### Change Data Capture (CDC) Based
```python
class CDCReconciler:
    def __init__(self, kafka_client, database_client):
        self.kafka = kafka_client
        self.db = database_client
        self.processed_changes = set()
    
    def process_cdc_stream(self, topic_name):
        """Process CDC events and track changes for reconciliation"""
        
        consumer = self.kafka.consumer(topic_name)
        
        for message in consumer:
            change_event = json.loads(message.value)
            
            change_id = f"{change_event['table']}:{change_event['key']}:{change_event['timestamp']}"
            
            if change_id in self.processed_changes:
                continue  # Skip duplicate events
            
            try:
                self.apply_change(change_event)
                self.processed_changes.add(change_id)
                
                # Commit offset only after successful processing
                consumer.commit()
                
            except Exception as e:
                # Log failure and continue - will be caught by reconciliation
                self.log_processing_failure(change_id, str(e))
    
    def apply_change(self, change_event):
        """Apply CDC change to target system"""
        
        table = change_event['table']
        operation = change_event['operation']
        key = change_event['key']
        data = change_event.get('data', {})
        
        if operation == 'INSERT':
            self.db.insert(table, data)
        elif operation == 'UPDATE':
            self.db.update(table, key, data)
        elif operation == 'DELETE':
            self.db.delete(table, key)
    
    def reconcile_missed_changes(self, start_timestamp, end_timestamp):
        """Find and apply changes that may have been missed"""
        
        # Query source database for changes in time window
        source_changes = self.db.get_changes_in_window(
            start_timestamp, end_timestamp
        )
        
        missed_changes = []
        
        for change in source_changes:
            change_id = f"{change['table']}:{change['key']}:{change['timestamp']}"
            
            if change_id not in self.processed_changes:
                missed_changes.append(change)
        
        # Apply missed changes
        for change in missed_changes:
            try:
                self.apply_change(change)
                print(f"Applied missed change: {change['table']}:{change['key']}")
            except Exception as e:
                print(f"Failed to apply missed change: {e}")
```

### 4. Business Logic Validation

#### Critical Business Rules Validation
```python
class BusinessLogicValidator:
    def __init__(self, source_db, target_db):
        self.source_db = source_db
        self.target_db = target_db
    
    def validate_financial_consistency(self):
        """Validate critical financial calculations"""
        
        validation_rules = [
            {
                'name': 'daily_transaction_totals',
                'source_query': """
                    SELECT DATE(created_at) as date, SUM(amount) as total
                    FROM source_transactions
                    WHERE created_at >= CURRENT_DATE - INTERVAL '30 days'
                    GROUP BY DATE(created_at)
                """,
                'target_query': """
                    SELECT DATE(created_at) as date, SUM(amount) as total
                    FROM target_transactions
                    WHERE created_at >= CURRENT_DATE - INTERVAL '30 days'
                    GROUP BY DATE(created_at)
                """,
                'tolerance': 0.01  # Allow $0.01 difference for rounding
            },
            {
                'name': 'customer_balance_totals',
                'source_query': """
                    SELECT customer_id, SUM(balance) as total_balance
                    FROM source_accounts
                    GROUP BY customer_id
                    HAVING SUM(balance) > 0
                """,
                'target_query': """
                    SELECT customer_id, SUM(balance) as total_balance
                    FROM target_accounts
                    GROUP BY customer_id
                    HAVING SUM(balance) > 0
                """,
                'tolerance': 0.01
            }
        ]
        
        validation_results = []
        
        for rule in validation_rules:
            source_data = self.source_db.execute_query(rule['source_query'])
            target_data = self.target_db.execute_query(rule['target_query'])
            
            differences = self.compare_financial_data(
                source_data, target_data, rule['tolerance']
            )
            
            validation_results.append({
                'rule_name': rule['name'],
                'differences_found': len(differences),
                'differences': differences[:10],  # First 10 differences
                'status': 'PASS' if len(differences) == 0 else 'FAIL'
            })
        
        return validation_results
    
    def compare_financial_data(self, source_data, target_data, tolerance):
        """Compare financial data with tolerance for rounding differences"""
        
        source_dict = {
            tuple(row[:-1]): row[-1] for row in source_data
        }  # Last column is the amount
        
        target_dict = {
            tuple(row[:-1]): row[-1] for row in target_data
        }
        
        differences = []
        
        # Check for missing records and value differences
        for key, source_value in source_dict.items():
            if key not in target_dict:
                differences.append({
                    'key': key,
                    'source_value': source_value,
                    'target_value': None,
                    'difference_type': 'MISSING_IN_TARGET'
                })
            else:
                target_value = target_dict[key]
                if abs(float(source_value) - float(target_value)) > tolerance:
                    differences.append({
                        'key': key,
                        'source_value': source_value,
                        'target_value': target_value,
                        'difference': float(source_value) - float(target_value),
                        'difference_type': 'VALUE_MISMATCH'
                    })
        
        # Check for extra records in target
        for key, target_value in target_dict.items():
            if key not in source_dict:
                differences.append({
                    'key': key,
                    'source_value': None,
                    'target_value': target_value,
                    'difference_type': 'EXTRA_IN_TARGET'
                })
        
        return differences
```

## Correction Strategies

### 1. Automated Correction

#### Missing Record Insertion
```python
class AutoCorrector:
    def __init__(self, source_db, target_db, dry_run=True):
        self.source_db = source_db
        self.target_db = target_db
        self.dry_run = dry_run
        self.correction_log = []
    
    def correct_missing_records(self, table_name, key_field):
        """Add missing records from source to target"""
        
        # Find records in source but not in target
        missing_query = f"""
            SELECT s.* 
            FROM source_{table_name} s
            LEFT JOIN target_{table_name} t ON s.{key_field} = t.{key_field}
            WHERE t.{key_field} IS NULL
        """
        
        missing_records = self.source_db.execute_query(missing_query)
        
        for record in missing_records:
            correction = {
                'table': table_name,
                'operation': 'INSERT',
                'key': record[key_field],
                'data': record,
                'timestamp': datetime.utcnow()
            }
            
            if not self.dry_run:
                try:
                    self.target_db.insert(table_name, record)
                    correction['status'] = 'SUCCESS'
                except Exception as e:
                    correction['status'] = 'FAILED'
                    correction['error'] = str(e)
            else:
                correction['status'] = 'DRY_RUN'
            
            self.correction_log.append(correction)
        
        return len(missing_records)
    
    def correct_field_mismatches(self, table_name, key_field, fields_to_correct):
        """Correct field value mismatches"""
        
        mismatch_query = f"""
            SELECT s.{key_field}, {', '.join([f's.{f} as source_{f}, t.{f} as target_{f}' for f in fields_to_correct])}
            FROM source_{table_name} s
            JOIN target_{table_name} t ON s.{key_field} = t.{key_field}
            WHERE {' OR '.join([f's.{f} != t.{f}' for f in fields_to_correct])}
        """
        
        mismatched_records = self.source_db.execute_query(mismatch_query)
        
        for record in mismatched_records:
            key_value = record[key_field]
            updates = {}
            
            for field in fields_to_correct:
                source_value = record[f'source_{field}']
                target_value = record[f'target_{field}']
                
                if source_value != target_value:
                    updates[field] = source_value
            
            if updates:
                correction = {
                    'table': table_name,
                    'operation': 'UPDATE',
                    'key': key_value,
                    'updates': updates,
                    'timestamp': datetime.utcnow()
                }
                
                if not self.dry_run:
                    try:
                        self.target_db.update(table_name, {key_field: key_value}, updates)
                        correction['status'] = 'SUCCESS'
                    except Exception as e:
                        correction['status'] = 'FAILED'
                        correction['error'] = str(e)
                else:
                    correction['status'] = 'DRY_RUN'
                
                self.correction_log.append(correction)
        
        return len(mismatched_records)
```

### 2. Manual Review Process

#### Correction Workflow
```python
class ManualReviewSystem:
    def __init__(self, database_client):
        self.db = database_client
        self.review_queue = []
    
    def queue_for_review(self, discrepancy):
        """Add discrepancy to manual review queue"""
        
        review_item = {
            'id': str(uuid.uuid4()),
            'discrepancy_type': discrepancy['type'],
            'table': discrepancy['table'],
            'record_key': discrepancy['key'],
            'source_data': discrepancy.get('source_data'),
            'target_data': discrepancy.get('target_data'),
            'description': discrepancy['description'],
            'severity': discrepancy.get('severity', 'medium'),
            'status': 'PENDING',
            'created_at': datetime.utcnow(),
            'reviewed_by': None,
            'reviewed_at': None,
            'resolution': None
        }
        
        self.review_queue.append(review_item)
        
        # Persist to review database
        self.db.insert('manual_review_queue', review_item)
        
        return review_item['id']
    
    def process_review(self, review_id, reviewer, action, notes=None):
        """Process manual review decision"""
        
        review_item = self.get_review_item(review_id)
        
        if not review_item:
            raise ValueError(f"Review item {review_id} not found")
        
        review_item.update({
            'status': 'REVIEWED',
            'reviewed_by': reviewer,
            'reviewed_at': datetime.utcnow(),
            'resolution': {
                'action': action,  # 'APPLY_SOURCE', 'KEEP_TARGET', 'CUSTOM_FIX'
                'notes': notes
            }
        })
        
        # Apply the resolution
        if action == 'APPLY_SOURCE':
            self.apply_source_data(review_item)
        elif action == 'KEEP_TARGET':
            pass  # No action needed
        elif action == 'CUSTOM_FIX':
            # Custom fix would be applied separately
            pass
        
        # Update review record
        self.db.update('manual_review_queue', 
                      {'id': review_id}, 
                      review_item)
        
        return review_item
    
    def generate_review_report(self):
        """Generate summary report of manual reviews"""
        
        reviews = self.db.query("""
            SELECT 
                discrepancy_type,
                severity,
                status,
                COUNT(*) as count,
                MIN(created_at) as oldest_review,
                MAX(created_at) as newest_review
            FROM manual_review_queue
            GROUP BY discrepancy_type, severity, status
            ORDER BY severity DESC, discrepancy_type
        """)
        
        return reviews
```

### 3. Reconciliation Scheduling

#### Automated Reconciliation Jobs
```python
import schedule
import time
from datetime import datetime, timedelta

class ReconciliationScheduler:
    def __init__(self, reconciler):
        self.reconciler = reconciler
        self.job_history = []
    
    def setup_schedules(self):
        """Set up automated reconciliation schedules"""
        
        # Quick reconciliation every 15 minutes during migration
        schedule.every(15).minutes.do(self.quick_reconciliation)
        
        # Comprehensive reconciliation every 4 hours
        schedule.every(4).hours.do(self.comprehensive_reconciliation)
        
        # Deep validation daily
        schedule.every().day.at("02:00").do(self.deep_validation)
        
        # Weekly business logic validation
        schedule.every().sunday.at("03:00").do(self.business_logic_validation)
    
    def quick_reconciliation(self):
        """Quick count-based reconciliation"""
        
        job_start = datetime.utcnow()
        
        try:
            # Check critical tables only
            critical_tables = [
                'transactions', 'orders', 'customers', 'accounts'
            ]
            
            results = []
            for table in critical_tables:
                count_diff = self.reconciler.check_row_counts(table)
                if abs(count_diff) > 0:
                    results.append({
                        'table': table,
                        'count_difference': count_diff,
                        'severity': 'high' if abs(count_diff) > 100 else 'medium'
                    })
            
            job_result = {
                'job_type': 'quick_reconciliation',
                'start_time': job_start,
                'end_time': datetime.utcnow(),
                'status': 'completed',
                'issues_found': len(results),
                'details': results
            }
            
            # Alert if significant issues found
            if any(r['severity'] == 'high' for r in results):
                self.send_alert(job_result)
        
        except Exception as e:
            job_result = {
                'job_type': 'quick_reconciliation',
                'start_time': job_start,
                'end_time': datetime.utcnow(),
                'status': 'failed',
                'error': str(e)
            }
        
        self.job_history.append(job_result)
    
    def comprehensive_reconciliation(self):
        """Comprehensive checksum-based reconciliation"""
        
        job_start = datetime.utcnow()
        
        try:
            tables_to_check = self.get_migration_tables()
            issues = []
            
            for table in tables_to_check:
                # Sample-based checksum validation
                sample_issues = self.reconciler.validate_sample_checksums(
                    table, sample_size=1000
                )
                issues.extend(sample_issues)
            
            # Auto-correct simple issues
            auto_corrections = 0
            for issue in issues:
                if issue['auto_correctable']:
                    self.reconciler.auto_correct_issue(issue)
                    auto_corrections += 1
                else:
                    # Queue for manual review
                    self.reconciler.queue_for_manual_review(issue)
            
            job_result = {
                'job_type': 'comprehensive_reconciliation',
                'start_time': job_start,
                'end_time': datetime.utcnow(),
                'status': 'completed',
                'total_issues': len(issues),
                'auto_corrections': auto_corrections,
                'manual_reviews_queued': len(issues) - auto_corrections
            }
        
        except Exception as e:
            job_result = {
                'job_type': 'comprehensive_reconciliation',
                'start_time': job_start,
                'end_time': datetime.utcnow(),
                'status': 'failed',
                'error': str(e)
            }
        
        self.job_history.append(job_result)
    
    def run_scheduler(self):
        """Run the reconciliation scheduler"""
        
        print("Starting reconciliation scheduler...")
        
        while True:
            schedule.run_pending()
            time.sleep(60)  # Check every minute
```

## Monitoring and Reporting

### 1. Reconciliation Metrics

```python
class ReconciliationMetrics:
    def __init__(self, prometheus_client):
        self.prometheus = prometheus_client
        
        # Define metrics
        self.inconsistencies_found = Counter(
            'reconciliation_inconsistencies_total',
            'Number of inconsistencies found',
            ['table', 'type', 'severity']
        )
        
        self.reconciliation_duration = Histogram(
            'reconciliation_duration_seconds',
            'Time spent on reconciliation jobs',
            ['job_type']
        )
        
        self.auto_corrections = Counter(
            'reconciliation_auto_corrections_total',
            'Number of automatically corrected inconsistencies',
            ['table', 'correction_type']
        )
        
        self.data_drift_gauge = Gauge(
            'data_drift_percentage',
            'Percentage of records with inconsistencies',
            ['table']
        )
    
    def record_inconsistency(self, table, inconsistency_type, severity):
        """Record a found inconsistency"""
        self.inconsistencies_found.labels(
            table=table,
            type=inconsistency_type,
            severity=severity
        ).inc()
    
    def record_auto_correction(self, table, correction_type):
        """Record an automatic correction"""
        self.auto_corrections.labels(
            table=table,
            correction_type=correction_type
        ).inc()
    
    def update_data_drift(self, table, drift_percentage):
        """Update data drift gauge"""
        self.data_drift_gauge.labels(table=table).set(drift_percentage)
    
    def record_job_duration(self, job_type, duration_seconds):
        """Record reconciliation job duration"""
        self.reconciliation_duration.labels(job_type=job_type).observe(duration_seconds)
```

### 2. Alerting Rules

```yaml
# Prometheus alerting rules for data reconciliation
groups:
  - name: data_reconciliation
    rules:
    - alert: HighDataInconsistency
      expr: reconciliation_inconsistencies_total > 100
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "High number of data inconsistencies detected"
        description: "{{ $value }} inconsistencies found in the last 5 minutes"
    
    - alert: DataDriftHigh
      expr: data_drift_percentage > 5
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Data drift percentage is high"
        description: "{{ $labels.table }} has {{ $value }}% data drift"
    
    - alert: ReconciliationJobFailed
      expr: up{job="reconciliation"} == 0
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Reconciliation job is down"
        description: "The data reconciliation service is not responding"
    
    - alert: AutoCorrectionRateHigh
      expr: rate(reconciliation_auto_corrections_total[10m]) > 10
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High rate of automatic corrections"
        description: "Auto-correction rate is {{ $value }} per second"
```

### 3. Dashboard and Reporting

```python
class ReconciliationDashboard:
    def __init__(self, database_client, metrics_client):
        self.db = database_client
        self.metrics = metrics_client
    
    def generate_daily_report(self, date=None):
        """Generate daily reconciliation report"""
        
        if not date:
            date = datetime.utcnow().date()
        
        # Query reconciliation results for the day
        daily_stats = self.db.query("""
            SELECT 
                table_name,
                inconsistency_type,
                COUNT(*) as count,
                AVG(CASE WHEN resolution = 'AUTO_CORRECTED' THEN 1 ELSE 0 END) as auto_correction_rate
            FROM reconciliation_log
            WHERE DATE(created_at) = %s
            GROUP BY table_name, inconsistency_type
        """, (date,))
        
        # Generate summary
        summary = {
            'date': date.isoformat(),
            'total_inconsistencies': sum(row['count'] for row in daily_stats),
            'auto_correction_rate': sum(row['auto_correction_rate'] * row['count'] for row in daily_stats) / max(sum(row['count'] for row in daily_stats), 1),
            'tables_affected': len(set(row['table_name'] for row in daily_stats)),
            'details_by_table': {}
        }
        
        # Group by table
        for row in daily_stats:
            table = row['table_name']
            if table not in summary['details_by_table']:
                summary['details_by_table'][table] = []
            
            summary['details_by_table'][table].append({
                'inconsistency_type': row['inconsistency_type'],
                'count': row['count'],
                'auto_correction_rate': row['auto_correction_rate']
            })
        
        return summary
    
    def generate_trend_analysis(self, days=7):
        """Generate trend analysis for reconciliation metrics"""
        
        end_date = datetime.utcnow().date()
        start_date = end_date - timedelta(days=days)
        
        trends = self.db.query("""
            SELECT 
                DATE(created_at) as date,
                table_name,
                COUNT(*) as inconsistencies,
                AVG(CASE WHEN resolution = 'AUTO_CORRECTED' THEN 1 ELSE 0 END) as auto_correction_rate
            FROM reconciliation_log
            WHERE DATE(created_at) BETWEEN %s AND %s
            GROUP BY DATE(created_at), table_name
            ORDER BY date, table_name
        """, (start_date, end_date))
        
        # Calculate trends
        trend_analysis = {
            'period': f"{start_date} to {end_date}",
            'trends': {},
            'overall_trend': 'stable'
        }
        
        for table in set(row['table_name'] for row in trends):
            table_data = [row for row in trends if row['table_name'] == table]
            
            if len(table_data) >= 2:
                first_count = table_data[0]['inconsistencies']
                last_count = table_data[-1]['inconsistencies']
                
                if last_count > first_count * 1.2:
                    trend = 'increasing'
                elif last_count < first_count * 0.8:
                    trend = 'decreasing'
                else:
                    trend = 'stable'
                
                trend_analysis['trends'][table] = {
                    'direction': trend,
                    'first_day_count': first_count,
                    'last_day_count': last_count,
                    'change_percentage': ((last_count - first_count) / max(first_count, 1)) * 100
                }
        
        return trend_analysis
```

## Advanced Reconciliation Techniques

### 1. Machine Learning-Based Anomaly Detection

```python
from sklearn.isolation import IsolationForest
from sklearn.preprocessing import StandardScaler
import numpy as np

class MLAnomalyDetector:
    def __init__(self):
        self.models = {}
        self.scalers = {}
    
    def train_anomaly_detector(self, table_name, training_data):
        """Train anomaly detection model for a specific table"""
        
        # Prepare features (convert records to numerical features)
        features = self.extract_features(training_data)
        
        # Scale features
        scaler = StandardScaler()
        scaled_features = scaler.fit_transform(features)
        
        # Train isolation forest
        model = IsolationForest(contamination=0.05, random_state=42)
        model.fit(scaled_features)
        
        # Store model and scaler
        self.models[table_name] = model
        self.scalers[table_name] = scaler
    
    def detect_anomalies(self, table_name, data):
        """Detect anomalous records that may indicate reconciliation issues"""
        
        if table_name not in self.models:
            raise ValueError(f"No trained model for table {table_name}")
        
        # Extract features
        features = self.extract_features(data)
        
        # Scale features
        scaled_features = self.scalers[table_name].transform(features)
        
        # Predict anomalies
        anomaly_scores = self.models[table_name].decision_function(scaled_features)
        anomaly_predictions = self.models[table_name].predict(scaled_features)
        
        # Return anomalous records with scores
        anomalies = []
        for i, (record, score, is_anomaly) in enumerate(zip(data, anomaly_scores, anomaly_predictions)):
            if is_anomaly == -1:  # Isolation forest returns -1 for anomalies
                anomalies.append({
                    'record_index': i,
                    'record': record,
                    'anomaly_score': score,
                    'severity': 'high' if score < -0.5 else 'medium'
                })
        
        return anomalies
    
    def extract_features(self, data):
        """Extract numerical features from database records"""
        
        features = []
        
        for record in data:
            record_features = []
            
            for key, value in record.items():
                if isinstance(value, (int, float)):
                    record_features.append(value)
                elif isinstance(value, str):
                    # Convert string to hash-based feature
                    record_features.append(hash(value) % 10000)
                elif isinstance(value, datetime):
                    # Convert datetime to timestamp
                    record_features.append(value.timestamp())
                else:
                    # Default value for other types
                    record_features.append(0)
            
            features.append(record_features)
        
        return np.array(features)
```

### 2. Probabilistic Reconciliation

```python
import random
from typing import List, Dict, Tuple

class ProbabilisticReconciler:
    def __init__(self, confidence_threshold=0.95):
        self.confidence_threshold = confidence_threshold
    
    def statistical_sampling_validation(self, table_name: str, population_size: int) -> Dict:
        """Use statistical sampling to validate large datasets"""
        
        # Calculate sample size for 95% confidence, 5% margin of error
        confidence_level = 0.95
        margin_of_error = 0.05
        
        z_score = 1.96  # for 95% confidence
        p = 0.5  # assume 50% error rate for maximum sample size
        
        sample_size = (z_score ** 2 * p * (1 - p)) / (margin_of_error ** 2)
        
        if population_size < 10000:
            # Finite population correction
            sample_size = sample_size / (1 + (sample_size - 1) / population_size)
        
        sample_size = min(int(sample_size), population_size)
        
        # Generate random sample
        sample_ids = self.generate_random_sample(table_name, sample_size)
        
        # Validate sample
        sample_results = self.validate_sample_records(table_name, sample_ids)
        
        # Calculate population estimates
        error_rate = sample_results['errors'] / sample_size
        estimated_errors = int(population_size * error_rate)
        
        # Calculate confidence interval
        standard_error = (error_rate * (1 - error_rate) / sample_size) ** 0.5
        margin_of_error_actual = z_score * standard_error
        
        confidence_interval = (
            max(0, error_rate - margin_of_error_actual),
            min(1, error_rate + margin_of_error_actual)
        )
        
        return {
            'table_name': table_name,
            'population_size': population_size,
            'sample_size': sample_size,
            'sample_error_rate': error_rate,
            'estimated_total_errors': estimated_errors,
            'confidence_interval': confidence_interval,
            'confidence_level': confidence_level,
            'recommendation': self.generate_recommendation(error_rate, confidence_interval)
        }
    
    def generate_random_sample(self, table_name: str, sample_size: int) -> List[int]:
        """Generate random sample of record IDs"""
        
        # Get total record count and ID range
        id_range = self.db.query(f"SELECT MIN(id), MAX(id) FROM {table_name}")[0]
        min_id, max_id = id_range
        
        # Generate random IDs
        sample_ids = []
        attempts = 0
        max_attempts = sample_size * 10  # Avoid infinite loop
        
        while len(sample_ids) < sample_size and attempts < max_attempts:
            candidate_id = random.randint(min_id, max_id)
            
            # Check if ID exists
            exists = self.db.query(f"SELECT 1 FROM {table_name} WHERE id = %s", (candidate_id,))
            
            if exists and candidate_id not in sample_ids:
                sample_ids.append(candidate_id)
            
            attempts += 1
        
        return sample_ids
    
    def validate_sample_records(self, table_name: str, sample_ids: List[int]) -> Dict:
        """Validate a sample of records"""
        
        validation_results = {
            'total_checked': len(sample_ids),
            'errors': 0,
            'error_details': []
        }
        
        for record_id in sample_ids:
            # Get record from both source and target
            source_record = self.source_db.get_record(table_name, record_id)
            target_record = self.target_db.get_record(table_name, record_id)
            
            if not target_record:
                validation_results['errors'] += 1
                validation_results['error_details'].append({
                    'id': record_id,
                    'error_type': 'MISSING_IN_TARGET'
                })
            elif not self.records_match(source_record, target_record):
                validation_results['errors'] += 1
                validation_results['error_details'].append({
                    'id': record_id,
                    'error_type': 'DATA_MISMATCH',
                    'differences': self.find_differences(source_record, target_record)
                })
        
        return validation_results
    
    def generate_recommendation(self, error_rate: float, confidence_interval: Tuple[float, float]) -> str:
        """Generate recommendation based on error rate and confidence"""
        
        if confidence_interval[1] < 0.01:  # Less than 1% error rate with confidence
            return "Data quality is excellent. Continue with normal reconciliation schedule."
        elif confidence_interval[1] < 0.05:  # Less than 5% error rate with confidence
            return "Data quality is acceptable. Monitor closely and investigate sample errors."
        elif confidence_interval[0] > 0.1:  # More than 10% error rate with confidence
            return "Data quality is poor. Immediate comprehensive reconciliation required."
        else:
            return "Data quality is uncertain. Increase sample size for better estimates."
```

## Performance Optimization

### 1. Parallel Processing

```python
import asyncio
import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor

class ParallelReconciler:
    def __init__(self, max_workers=None):
        self.max_workers = max_workers or mp.cpu_count()
    
    async def parallel_table_reconciliation(self, tables: List[str]):
        """Reconcile multiple tables in parallel"""
        
        async with asyncio.Semaphore(self.max_workers):
            tasks = [
                self.reconcile_table_async(table) 
                for table in tables
            ]
            
            results = await asyncio.gather(*tasks, return_exceptions=True)
            
            # Process results
            summary = {
                'total_tables': len(tables),
                'successful': 0,
                'failed': 0,
                'results': {}
            }
            
            for table, result in zip(tables, results):
                if isinstance(result, Exception):
                    summary['failed'] += 1
                    summary['results'][table] = {
                        'status': 'failed',
                        'error': str(result)
                    }
                else:
                    summary['successful'] += 1
                    summary['results'][table] = result
            
            return summary
    
    def parallel_chunk_processing(self, table_name: str, chunk_size: int = 10000):
        """Process table reconciliation in parallel chunks"""
        
        # Get total record count
        total_records = self.db.get_record_count(table_name)
        num_chunks = (total_records + chunk_size - 1) // chunk_size
        
        # Create chunk specifications
        chunks = []
        for i in range(num_chunks):
            start_id = i * chunk_size
            end_id = min((i + 1) * chunk_size - 1, total_records - 1)
            chunks.append({
                'table': table_name,
                'start_id': start_id,
                'end_id': end_id,
                'chunk_number': i + 1
            })
        
        # Process chunks in parallel
        with ProcessPoolExecutor(max_workers=self.max_workers) as executor:
            chunk_results = list(executor.map(self.process_chunk, chunks))
        
        # Aggregate results
        total_inconsistencies = sum(r['inconsistencies'] for r in chunk_results)
        total_corrections = sum(r['corrections'] for r in chunk_results)
        
        return {
            'table': table_name,
            'total_records': total_records,
            'chunks_processed': len(chunks),
            'total_inconsistencies': total_inconsistencies,
            'total_corrections': total_corrections,
            'chunk_details': chunk_results
        }
    
    def process_chunk(self, chunk_spec: Dict) -> Dict:
        """Process a single chunk of records"""
        
        # This runs in a separate process
        table = chunk_spec['table']
        start_id = chunk_spec['start_id']
        end_id = chunk_spec['end_id']
        
        # Initialize database connections for this process
        local_source_db = SourceDatabase()
        local_target_db = TargetDatabase()
        
        # Get records in chunk
        source_records = local_source_db.get_records_range(table, start_id, end_id)
        target_records = local_target_db.get_records_range(table, start_id, end_id)
        
        # Reconcile chunk
        inconsistencies = 0
        corrections = 0
        
        for source_record in source_records:
            target_record = target_records.get(source_record['id'])
            
            if not target_record:
                inconsistencies += 1
                # Auto-correct if possible
                try:
                    local_target_db.insert(table, source_record)
                    corrections += 1
                except Exception:
                    pass  # Log error in production
            elif not self.records_match(source_record, target_record):
                inconsistencies += 1
                # Auto-correct field mismatches
                try:
                    updates = self.calculate_updates(source_record, target_record)
                    local_target_db.update(table, source_record['id'], updates)
                    corrections += 1
                except Exception:
                    pass  # Log error in production
        
        return {
            'chunk_number': chunk_spec['chunk_number'],
            'start_id': start_id,
            'end_id': end_id,
            'records_processed': len(source_records),
            'inconsistencies': inconsistencies,
            'corrections': corrections
        }
```

### 2. Incremental Reconciliation

```python
class IncrementalReconciler:
    def __init__(self, source_db, target_db):
        self.source_db = source_db
        self.target_db = target_db
        self.last_reconciliation_times = {}
    
    def incremental_reconciliation(self, table_name: str):
        """Reconcile only records changed since last reconciliation"""
        
        last_reconciled = self.get_last_reconciliation_time(table_name)
        
        # Get records modified since last reconciliation
        modified_source = self.source_db.get_records_modified_since(
            table_name, last_reconciled
        )
        
        modified_target = self.target_db.get_records_modified_since(
            table_name, last_reconciled
        )
        
        # Create lookup dictionaries
        source_dict = {r['id']: r for r in modified_source}
        target_dict = {r['id']: r for r in modified_target}
        
        # Find all record IDs to check
        all_ids = set(source_dict.keys()) | set(target_dict.keys())
        
        inconsistencies = []
        
        for record_id in all_ids:
            source_record = source_dict.get(record_id)
            target_record = target_dict.get(record_id)
            
            if source_record and not target_record:
                inconsistencies.append({
                    'type': 'missing_in_target',
                    'table': table_name,
                    'id': record_id,
                    'source_record': source_record
                })
            elif not source_record and target_record:
                inconsistencies.append({
                    'type': 'extra_in_target',
                    'table': table_name,
                    'id': record_id,
                    'target_record': target_record
                })
            elif source_record and target_record:
                if not self.records_match(source_record, target_record):
                    inconsistencies.append({
                        'type': 'data_mismatch',
                        'table': table_name,
                        'id': record_id,
                        'source_record': source_record,
                        'target_record': target_record,
                        'differences': self.find_differences(source_record, target_record)
                    })
        
        # Update last reconciliation time
        self.update_last_reconciliation_time(table_name, datetime.utcnow())
        
        return {
            'table': table_name,
            'reconciliation_time': datetime.utcnow(),
            'records_checked': len(all_ids),
            'inconsistencies_found': len(inconsistencies),
            'inconsistencies': inconsistencies
        }
    
    def get_last_reconciliation_time(self, table_name: str) -> datetime:
        """Get the last reconciliation timestamp for a table"""
        
        result = self.source_db.query("""
            SELECT last_reconciled_at
            FROM reconciliation_metadata
            WHERE table_name = %s
        """, (table_name,))
        
        if result:
            return result[0]['last_reconciled_at']
        else:
            # First time reconciliation - start from beginning of migration
            return self.get_migration_start_time()
    
    def update_last_reconciliation_time(self, table_name: str, timestamp: datetime):
        """Update the last reconciliation timestamp"""
        
        self.source_db.execute("""
            INSERT INTO reconciliation_metadata (table_name, last_reconciled_at)
            VALUES (%s, %s)
            ON CONFLICT (table_name)
            DO UPDATE SET last_reconciled_at = %s
        """, (table_name, timestamp, timestamp))
```

This comprehensive guide provides the framework and tools necessary for implementing robust data reconciliation strategies during migrations, ensuring data integrity and consistency while minimizing business disruption.

path: references/zero_downtime_techniques.md
content: # Zero-Downtime Migration Techniques

## Overview

Zero-downtime migrations are critical for maintaining business continuity and user experience during system changes. This guide provides comprehensive techniques, patterns, and implementation strategies for achieving true zero-downtime migrations across different system components.

## Core Principles

### 1. Backward Compatibility
Every change must be backward compatible until all clients have migrated to the new version.

### 2. Incremental Changes
Break large changes into smaller, independent increments that can be deployed and validated separately.

### 3. Feature Flags
Use feature toggles to control the rollout of new functionality without code deployments.

### 4. Graceful Degradation
Ensure systems continue to function even when some components are unavailable or degraded.

## Database Zero-Downtime Techniques

### Schema Evolution Without Downtime

#### 1. Additive Changes Only
**Principle:** Only add new elements; never remove or modify existing ones directly.

```sql
-- ✅ Good: Additive change
ALTER TABLE users ADD COLUMN middle_name VARCHAR(50);

-- ❌ Bad: Breaking change
ALTER TABLE users DROP COLUMN email;
```

#### 2. Multi-Phase Schema Evolution

**Phase 1: Expand**
```sql
-- Add new column alongside existing one
ALTER TABLE users ADD COLUMN email_address VARCHAR(255);

-- Add index concurrently (PostgreSQL)
CREATE INDEX CONCURRENTLY idx_users_email_address ON users(email_address);
```

**Phase 2: Dual Write (Application Code)**
```python
class UserService:
    def create_user(self, name, email):
        # Write to both old and new columns
        user = User(
            name=name,
            email=email,           # Old column
            email_address=email    # New column
        )
        return user.save()
    
    def update_email(self, user_id, new_email):
        # Update both columns
        user = User.objects.get(id=user_id)
        user.email = new_email
        user.email_address = new_email
        user.save()
        return user
```

**Phase 3: Backfill Data**
```sql
-- Backfill existing data (in batches)
UPDATE users 
SET email_address = email 
WHERE email_address IS NULL 
  AND id BETWEEN ? AND ?;
```

**Phase 4: Switch Reads**
```python
class UserService:
    def get_user_email(self, user_id):
        user = User.objects.get(id=user_id)
        # Switch to reading from new column
        return user.email_address or user.email
```

**Phase 5: Contract**
```sql
-- After validation, remove old column
ALTER TABLE users DROP COLUMN email;
-- Rename new column if needed
ALTER TABLE users RENAME COLUMN email_address TO email;
```

### 3. Online Schema Changes

#### PostgreSQL Techniques

```sql
-- Safe column addition
ALTER TABLE orders ADD COLUMN status_new VARCHAR(20) DEFAULT 'pending';

-- Safe index creation
CREATE INDEX CONCURRENTLY idx_orders_status_new ON orders(status_new);

-- Safe constraint addition (after data validation)
ALTER TABLE orders ADD CONSTRAINT check_status_new 
CHECK (status_new IN ('pending', 'processing', 'completed', 'cancelled'));
```

#### MySQL Techniques

```sql
-- Use pt-online-schema-change for large tables
pt-online-schema-change \
  --alter "ADD COLUMN status VARCHAR(20) DEFAULT 'pending'" \
  --execute \
  D=mydb,t=orders

-- Online DDL (MySQL 5.6+)
ALTER TABLE orders 
ADD COLUMN priority INT DEFAULT 1,
ALGORITHM=INPLACE, 
LOCK=NONE;
```

### 4. Data Migration Strategies

#### Chunked Data Migration

```python
class DataMigrator:
    def __init__(self, source_table, target_table, chunk_size=1000):
        self.source_table = source_table
        self.target_table = target_table
        self.chunk_size = chunk_size
    
    def migrate_data(self):
        last_id = 0
        total_migrated = 0
        
        while True:
            # Get next chunk
            chunk = self.get_chunk(last_id, self.chunk_size)
            
            if not chunk:
                break
            
            # Transform and migrate chunk
            for record in chunk:
                transformed = self.transform_record(record)
                self.insert_or_update(transformed)
            
            last_id = chunk[-1]['id']
            total_migrated += len(chunk)
            
            # Brief pause to avoid overwhelming the database
            time.sleep(0.1)
            
            self.log_progress(total_migrated)
        
        return total_migrated
    
    def get_chunk(self, last_id, limit):
        return db.execute(f"""
            SELECT * FROM {self.source_table}
            WHERE id > %s
            ORDER BY id
            LIMIT %s
        """, (last_id, limit))
```

#### Change Data Capture (CDC)

```python
class CDCProcessor:
    def __init__(self):
        self.kafka_consumer = KafkaConsumer('db_changes')
        self.target_db = TargetDatabase()
    
    def process_changes(self):
        for message in self.kafka_consumer:
            change = json.loads(message.value)
            
            if change['operation'] == 'INSERT':
                self.handle_insert(change)
            elif change['operation'] == 'UPDATE':
                self.handle_update(change)
            elif change['operation'] == 'DELETE':
                self.handle_delete(change)
    
    def handle_insert(self, change):
        transformed_data = self.transform_data(change['after'])
        self.target_db.insert(change['table'], transformed_data)
    
    def handle_update(self, change):
        key = change['key']
        transformed_data = self.transform_data(change['after'])
        self.target_db.update(change['table'], key, transformed_data)
```

## Application Zero-Downtime Techniques

### 1. Blue-Green Deployments

#### Infrastructure Setup

```yaml
# Blue Environment (Current Production)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-blue
  labels:
    version: blue
    app: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: blue
  template:
    metadata:
      labels:
        app: myapp
        version: blue
    spec:
      containers:
      - name: app
        image: myapp:1.0.0
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 10

---
# Green Environment (New Version)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-green
  labels:
    version: green
    app: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: green
  template:
    metadata:
      labels:
        app: myapp
        version: green
    spec:
      containers:
      - name: app
        image: myapp:2.0.0
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
```

#### Service Switching

```yaml
# Service (switches between blue and green)
apiVersion: v1
kind: Service
metadata:
  name: app-service
spec:
  selector:
    app: myapp
    version: blue  # Switch to 'green' for deployment
  ports:
  - port: 80
    targetPort: 8080
  type: LoadBalancer
```

#### Automated Deployment Script

```bash
#!/bin/bash

# Blue-Green Deployment Script
NAMESPACE="production"
APP_NAME="myapp"
NEW_IMAGE="myapp:2.0.0"

# Determine current and target environments
CURRENT_VERSION=$(kubectl get service $APP_NAME-service -o jsonpath='{.spec.selector.version}')

if [ "$CURRENT_VERSION" = "blue" ]; then
    TARGET_VERSION="green"
else
    TARGET_VERSION="blue"
fi

echo "Current version: $CURRENT_VERSION"
echo "Target version: $TARGET_VERSION"

# Update target environment with new image
kubectl set image deployment/$APP_NAME-$TARGET_VERSION app=$NEW_IMAGE

# Wait for rollout to complete
kubectl rollout status deployment/$APP_NAME-$TARGET_VERSION --timeout=300s

# Run health checks
echo "Running health checks..."
TARGET_IP=$(kubectl get service $APP_NAME-$TARGET_VERSION -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

for i in {1..30}; do
    if curl -f http://$TARGET_IP/health; then
        echo "Health check passed"
        break
    fi
    
    if [ $i -eq 30 ]; then
        echo "Health check failed after 30 attempts"
        exit 1
    fi
    
    sleep 2
done

# Switch traffic to new version
kubectl patch service $APP_NAME-service -p '{"spec":{"selector":{"version":"'$TARGET_VERSION'"}}}'

echo "Traffic switched to $TARGET_VERSION"

# Monitor for 5 minutes
echo "Monitoring new version..."
sleep 300

# Check if rollback is needed
ERROR_RATE=$(curl -s "http://monitoring.company.com/api/error_rate?service=$APP_NAME" | jq '.error_rate')

if (( $(echo "$ERROR_RATE > 0.05" | bc -l) )); then
    echo "Error rate too high ($ERROR_RATE), rolling back..."
    kubectl patch service $APP_NAME-service -p '{"spec":{"selector":{"version":"'$CURRENT_VERSION'"}}}'
    exit 1
fi

echo "Deployment successful!"
```

### 2. Canary Deployments

#### Progressive Canary with Istio

```yaml
# Destination Rule
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: myapp-destination
spec:
  host: myapp
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

---
# Virtual Service for Canary
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp-canary
spec:
  hosts:
  - myapp
  http:
  - match:
    - headers:
        canary:
          exact: "true"
    route:
    - destination:
        host: myapp
        subset: v2
  - route:
    - destination:
        host: myapp
        subset: v1
      weight: 95
    - destination:
        host: myapp
        subset: v2
      weight: 5
```

#### Automated Canary Controller

```python
class CanaryController:
    def __init__(self, istio_client, prometheus_client):
        self.istio = istio_client
        self.prometheus = prometheus_client
        self.canary_weight = 5
        self.max_weight = 100
        self.weight_increment = 5
        self.validation_window = 300  # 5 minutes
    
    async def deploy_canary(self, app_name, new_version):
        """Deploy new version using canary strategy"""
        
        # Start with small percentage
        await self.update_traffic_split(app_name, self.canary_weight)
        
        while self.canary_weight < self.max_weight:
            # Monitor metrics for validation window
            await asyncio.sleep(self.validation_window)
            
            # Check canary health
            if not await self.is_canary_healthy(app_name, new_version):
                await self.rollback_canary(app_name)
                raise Exception("Canary deployment failed health checks")
            
            # Increase traffic to canary
            self.canary_weight = min(
                self.canary_weight + self.weight_increment,
                self.max_weight
            )
            
            await self.update_traffic_split(app_name, self.canary_weight)
            
            print(f"Canary traffic increased to {self.canary_weight}%")
        
        print("Canary deployment completed successfully")
    
    async def is_canary_healthy(self, app_name, version):
        """Check if canary version is healthy"""
        
        # Check error rate
        error_rate = await self.prometheus.query(
            f'rate(http_requests_total{{app="{app_name}", version="{version}", status=~"5.."}}'
            f'[5m]) / rate(http_requests_total{{app="{app_name}", version="{version}"}}[5m])'
        )
        
        if error_rate > 0.05:  # 5% error rate threshold
            return False
        
        # Check response time
        p95_latency = await self.prometheus.query(
            f'histogram_quantile(0.95, rate(http_request_duration_seconds_bucket'
            f'{{app="{app_name}", version="{version}"}}[5m]))'
        )
        
        if p95_latency > 2.0:  # 2 second p95 threshold
            return False
        
        return True
    
    async def update_traffic_split(self, app_name, canary_weight):
        """Update Istio virtual service with new traffic split"""
        
        stable_weight = 100 - canary_weight
        
        virtual_service = {
            "apiVersion": "networking.istio.io/v1beta1",
            "kind": "VirtualService",
            "metadata": {"name": f"{app_name}-canary"},
            "spec": {
                "hosts": [app_name],
                "http": [{
                    "route": [
                        {
                            "destination": {"host": app_name, "subset": "stable"},
                            "weight": stable_weight
                        },
                        {
                            "destination": {"host": app_name, "subset": "canary"},
                            "weight": canary_weight
                        }
                    ]
                }]
            }
        }
        
        await self.istio.apply_virtual_service(virtual_service)
```

### 3. Rolling Updates

#### Kubernetes Rolling Update Strategy

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rolling-update-app
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2         # Can have 2 extra pods during update
      maxUnavailable: 1   # At most 1 pod can be unavailable
  selector:
    matchLabels:
      app: rolling-update-app
  template:
    metadata:
      labels:
        app: rolling-update-app
    spec:
      containers:
      - name: app
        image: myapp:2.0.0
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 2
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /live
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
```

#### Custom Rolling Update Controller

```python
class RollingUpdateController:
    def __init__(self, k8s_client):
        self.k8s = k8s_client
        self.max_surge = 2
        self.max_unavailable = 1
        
    async def rolling_update(self, deployment_name, new_image):
        """Perform rolling update with custom logic"""
        
        deployment = await self.k8s.get_deployment(deployment_name)
        total_replicas = deployment.spec.replicas
        
        # Calculate batch size
        batch_size = min(self.max_surge, total_replicas // 5)  # Update 20% at a time
        
        updated_pods = []
        
        for i in range(0, total_replicas, batch_size):
            batch_end = min(i + batch_size, total_replicas)
            
            # Update batch of pods
            for pod_index in range(i, batch_end):
                old_pod = await self.get_pod_by_index(deployment_name, pod_index)
                
                # Create new pod with new image
                new_pod = await self.create_updated_pod(old_pod, new_image)
                
                # Wait for new pod to be ready
                await self.wait_for_pod_ready(new_pod.metadata.name)
                
                # Remove old pod
                await self.k8s.delete_pod(old_pod.metadata.name)
                
                updated_pods.append(new_pod)
                
                # Brief pause between pod updates
                await asyncio.sleep(2)
            
            # Validate batch health before continuing
            if not await self.validate_batch_health(updated_pods[-batch_size:]):
                # Rollback batch
                await self.rollback_batch(updated_pods[-batch_size:])
                raise Exception("Rolling update failed validation")
            
            print(f"Updated {batch_end}/{total_replicas} pods")
        
        print("Rolling update completed successfully")
```

## Load Balancer and Traffic Management

### 1. Weighted Routing

#### NGINX Configuration

```nginx
upstream backend {
    # Old version - 80% traffic
    server old-app-1:8080 weight=4;
    server old-app-2:8080 weight=4;
    
    # New version - 20% traffic
    server new-app-1:8080 weight=1;
    server new-app-2:8080 weight=1;
}

server {
    listen 80;
    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        
        # Health check headers
        proxy_set_header X-Health-Check-Timeout 5s;
    }
}
```

#### HAProxy Configuration

```haproxy
backend app_servers
    balance roundrobin
    option httpchk GET /health
    
    # Old version servers
    server old-app-1 old-app-1:8080 check weight 80
    server old-app-2 old-app-2:8080 check weight 80
    
    # New version servers
    server new-app-1 new-app-1:8080 check weight 20
    server new-app-2 new-app-2:8080 check weight 20

frontend app_frontend
    bind *:80
    default_backend app_servers
    
    # Custom health check endpoint
    acl health_check path_beg /health
    http-request return status 200 content-type text/plain string "OK" if health_check
```

### 2. Circuit Breaker Implementation

```python
class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=60, expected_exception=Exception):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.expected_exception = expected_exception
        self.failure_count = 0
        self.last_failure_time = None
        self.state = 'CLOSED'  # CLOSED, OPEN, HALF_OPEN
        
    def call(self, func, *args, **kwargs):
        """Execute function with circuit breaker protection"""
        
        if self.state == 'OPEN':
            if self._should_attempt_reset():
                self.state = 'HALF_OPEN'
            else:
                raise CircuitBreakerOpenException("Circuit breaker is OPEN")
        
        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except self.expected_exception as e:
            self._on_failure()
            raise
    
    def _should_attempt_reset(self):
        return (
            self.last_failure_time and
            time.time() - self.last_failure_time >= self.recovery_timeout
        )
    
    def _on_success(self):
        self.failure_count = 0
        self.state = 'CLOSED'
    
    def _on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        
        if self.failure_count >= self.failure_threshold:
            self.state = 'OPEN'

# Usage with service migration
@CircuitBreaker(failure_threshold=3, recovery_timeout=30)
def call_new_service(request):
    return new_service.process(request)

def handle_request(request):
    try:
        return call_new_service(request)
    except CircuitBreakerOpenException:
        # Fallback to old service
        return old_service.process(request)
```

## Monitoring and Validation

### 1. Health Check Implementation

```python
class HealthChecker:
    def __init__(self):
        self.checks = []
        
    def add_check(self, name, check_func, timeout=5):
        self.checks.append({
            'name': name,
            'func': check_func,
            'timeout': timeout
        })
    
    async def run_checks(self):
        """Run all health checks and return status"""
        results = {}
        overall_status = 'healthy'
        
        for check in self.checks:
            try:
                result = await asyncio.wait_for(
                    check['func'](),
                    timeout=check['timeout']
                )
                results[check['name']] = {
                    'status': 'healthy',
                    'result': result
                }
            except asyncio.TimeoutError:
                results[check['name']] = {
                    'status': 'unhealthy',
                    'error': 'timeout'
                }
                overall_status = 'unhealthy'
            except Exception as e:
                results[check['name']] = {
                    'status': 'unhealthy',
                    'error': str(e)
                }
                overall_status = 'unhealthy'
        
        return {
            'status': overall_status,
            'checks': results,
            'timestamp': datetime.utcnow().isoformat()
        }

# Example health checks
health_checker = HealthChecker()

async def database_check():
    """Check database connectivity"""
    result = await db.execute("SELECT 1")
    return result is not None

async def external_api_check():
    """Check external API availability"""
    response = await http_client.get("https://api.example.com/health")
    return response.status_code == 200

async def memory_check():
    """Check memory usage"""
    memory_usage = psutil.virtual_memory().percent
    if memory_usage > 90:
        raise Exception(f"Memory usage too high: {memory_usage}%")
    return f"Memory usage: {memory_usage}%"

health_checker.add_check("database", database_check)
health_checker.add_check("external_api", external_api_check)
health_checker.add_check("memory", memory_check)
```

### 2. Readiness vs Liveness Probes

```yaml
# Kubernetes Pod with proper health checks
apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  containers:
  - name: app
    image: myapp:2.0.0
    ports:
    - containerPort: 8080
    
    # Readiness probe - determines if pod should receive traffic
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 3
      timeoutSeconds: 2
      successThreshold: 1
      failureThreshold: 3
    
    # Liveness probe - determines if pod should be restarted
    livenessProbe:
      httpGet:
        path: /live
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 5
      successThreshold: 1
      failureThreshold: 3
    
    # Startup probe - gives app time to start before other probes
    startupProbe:
      httpGet:
        path: /startup
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 3
      successThreshold: 1
      failureThreshold: 30  # Allow up to 150 seconds for startup
```

### 3. Metrics and Alerting

```python
class MigrationMetrics:
    def __init__(self, prometheus_client):
        self.prometheus = prometheus_client
        
        # Define custom metrics
        self.migration_progress = Counter(
            'migration_progress_total',
            'Total migration operations completed',
            ['operation', 'status']
        )
        
        self.migration_duration = Histogram(
            'migration_operation_duration_seconds',
            'Time spent on migration operations',
            ['operation']
        )
        
        self.system_health = Gauge(
            'system_health_score',
            'Overall system health score (0-1)',
            ['component']
        )
        
        self.traffic_split = Gauge(
            'traffic_split_percentage',
            'Percentage of traffic going to each version',
            ['version']
        )
    
    def record_migration_step(self, operation, status, duration=None):
        """Record completion of a migration step"""
        self.migration_progress.labels(operation=operation, status=status).inc()
        
        if duration:
            self.migration_duration.labels(operation=operation).observe(duration)
    
    def update_health_score(self, component, score):
        """Update health score for a component"""
        self.system_health.labels(component=component).set(score)
    
    def update_traffic_split(self, version_weights):
        """Update traffic split metrics"""
        for version, weight in version_weights.items():
            self.traffic_split.labels(version=version).set(weight)

# Usage in migration
metrics = MigrationMetrics(prometheus_client)

def perform_migration_step(operation):
    start_time = time.time()
    
    try:
        # Perform migration operation
        result = execute_migration_operation(operation)
        
        # Record success
        duration = time.time() - start_time
        metrics.record_migration_step(operation, 'success', duration)
        
        return result
        
    except Exception as e:
        # Record failure
        duration = time.time() - start_time
        metrics.record_migration_step(operation, 'failure', duration)
        raise
```

## Rollback Strategies

### 1. Immediate Rollback Triggers

```python
class AutoRollbackSystem:
    def __init__(self, metrics_client, deployment_client):
        self.metrics = metrics_client
        self.deployment = deployment_client
        self.rollback_triggers = {
            'error_rate_spike': {
                'threshold': 0.05,  # 5% error rate
                'window': 300,      # 5 minutes
                'auto_rollback': True
            },
            'latency_increase': {
                'threshold': 2.0,   # 2x baseline latency
                'window': 600,      # 10 minutes
                'auto_rollback': False  # Manual confirmation required
            },
            'availability_drop': {
                'threshold': 0.95,  # Below 95% availability
                'window': 120,      # 2 minutes
                'auto_rollback': True
            }
        }
    
    async def monitor_and_rollback(self, deployment_name):
        """Monitor deployment and trigger rollback if needed"""
        
        while True:
            for trigger_name, config in self.rollback_triggers.items():
                if await self.check_trigger(trigger_name, config):
                    if config['auto_rollback']:
                        await self.execute_rollback(deployment_name, trigger_name)
                    else:
                        await self.alert_for_manual_rollback(deployment_name, trigger_name)
            
            await asyncio.sleep(30)  # Check every 30 seconds
    
    async def check_trigger(self, trigger_name, config):
        """Check if rollback trigger condition is met"""
        
        current_value = await self.metrics.get_current_value(trigger_name)
        baseline_value = await self.metrics.get_baseline_value(trigger_name)
        
        if trigger_name == 'error_rate_spike':
            return current_value > config['threshold']
        elif trigger_name == 'latency_increase':
            return current_value > baseline_value * config['threshold']
        elif trigger_name == 'availability_drop':
            return current_value < config['threshold']
        
        return False
    
    async def execute_rollback(self, deployment_name, reason):
        """Execute automatic rollback"""
        
        print(f"Executing automatic rollback for {deployment_name}. Reason: {reason}")
        
        # Get previous revision
        previous_revision = await self.deployment.get_previous_revision(deployment_name)
        
        # Perform rollback
        await self.deployment.rollback_to_revision(deployment_name, previous_revision)
        
        # Notify stakeholders
        await self.notify_rollback_executed(deployment_name, reason)
```

### 2. Data Rollback Strategies

```sql
-- Point-in-time recovery setup
-- Create restore point before migration
SELECT pg_create_restore_point('pre_migration_' || to_char(now(), 'YYYYMMDD_HH24MISS'));

-- Rollback using point-in-time recovery
-- (This would be executed on a separate recovery instance)
-- recovery.conf:
-- recovery_target_name = 'pre_migration_20240101_120000'
-- recovery_target_action = 'promote'
```

```python
class DataRollbackManager:
    def __init__(self, database_client, backup_service):
        self.db = database_client
        self.backup = backup_service
    
    async def create_rollback_point(self, migration_id):
        """Create a rollback point before migration"""
        
        rollback_point = {
            'migration_id': migration_id,
            'timestamp': datetime.utcnow(),
            'backup_location': None,
            'schema_snapshot': None
        }
        
        # Create database backup
        backup_path = await self.backup.create_backup(
            f"pre_migration_{migration_id}_{int(time.time())}"
        )
        rollback_point['backup_location'] = backup_path
        
        # Capture schema snapshot
        schema_snapshot = await self.capture_schema_snapshot()
        rollback_point['schema_snapshot'] = schema_snapshot
        
        # Store rollback point metadata
        await self.store_rollback_metadata(rollback_point)
        
        return rollback_point
    
    async def execute_rollback(self, migration_id):
        """Execute data rollback to specified point"""
        
        rollback_point = await self.get_rollback_metadata(migration_id)
        
        if not rollback_point:
            raise Exception(f"No rollback point found for migration {migration_id}")
        
        # Stop application traffic
        await self.stop_application_traffic()
        
        try:
            # Restore from backup
            await self.backup.restore_from_backup(
                rollback_point['backup_location']
            )
            
            # Validate data integrity
            await self.validate_data_integrity(
                rollback_point['schema_snapshot']
            )
            
            # Update application configuration
            await self.update_application_config(rollback_point)
            
            # Resume application traffic
            await self.resume_application_traffic()
            
            print(f"Data rollback completed successfully for migration {migration_id}")
            
        except Exception as e:
            # If rollback fails, we have a serious problem
            await self.escalate_rollback_failure(migration_id, str(e))
            raise
```

## Best Practices Summary

### 1. Pre-Migration Checklist
- [ ] Comprehensive backup strategy in place
- [ ] Rollback procedures tested in staging
- [ ] Monitoring and alerting configured
- [ ] Health checks implemented
- [ ] Feature flags configured
- [ ] Team communication plan established
- [ ] Load balancer configuration prepared
- [ ] Database connection pooling optimized

### 2. During Migration
- [ ] Monitor key metrics continuously
- [ ] Validate each phase before proceeding
- [ ] Maintain detailed logs of all actions
- [ ] Keep stakeholders informed of progress
- [ ] Have rollback trigger ready
- [ ] Monitor user experience metrics
- [ ] Watch for performance degradation
- [ ] Validate data consistency

### 3. Post-Migration
- [ ] Continue monitoring for 24-48 hours
- [ ] Validate all business processes
- [ ] Update documentation
- [ ] Conduct post-migration retrospective
- [ ] Archive migration artifacts
- [ ] Update disaster recovery procedures
- [ ] Plan for legacy system decommissioning

### 4. Common Pitfalls to Avoid
- Don't skip testing rollback procedures
- Don't ignore performance impact
- Don't rush through validation phases
- Don't forget to communicate with stakeholders
- Don't assume health checks are sufficient
- Don't neglect data consistency validation
- Don't underestimate time requirements
- Don't overlook dependency impacts

This comprehensive guide provides the foundation for implementing zero-downtime migrations across various system components while maintaining high availability and data integrity.

Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)