Skill: encryptshawn/mysql8-crm-schema-expert (version 1.0.0)

--- name: mysql8-design-crm version: "1.0.0" description: > MySQL 8 database schema design for CRM systems. Use this skill whenever the user needs to design, review, optimize, or generate database schemas for Customer Relationship Management systems. Triggers on: CRM database design, CRM schema, customer database, contact management database, sales pipeline database, lead tracking schema, opportunity management tables, CRM entity relationships, CRM data model, accounts/contacts/deals schema, activity tracking database, CRM audit trail, CRM custom fields, EAV pattern for CRM, CRM soft deletes, polymorphic relationships in CRM, sales funnel database, customer lifecycle database, CRM normalization, CRM indexing strategy, MySQL CRM tables, or any database design work involving customer relationship management concepts. Also trigger when the user mentions designing tables for contacts, accounts, opportunities, leads, deals, pipelines, activities, notes, tasks, campaigns, or any combination of these CRM entities. --- # MySQL 8 CRM Database Design Skill A comprehensive guide for designing production-quality MySQL 8 database schemas for CRM (Customer Relationship Management) systems. This skill covers everything from core entity design to advanced patterns like EAV custom fields, polymorphic activities, audit trails, and multi-tenant architectures. ## How to Use This Skill This skill is organized into a main guide (this file) and detailed reference documents. Read the relevant reference file before generating any SQL or making design decisions. ### Reference Files Read these from `references/` as needed: | File | When to Read | |------|-------------| | `core-entities.md` | Designing the foundational CRM tables (accounts, contacts, leads, opportunities, etc.) | | `relationships-and-normalization.md` | Establishing foreign keys, junction tables, and achieving proper normal forms | | `indexing-and-performance.md` | Creating indexes, query optimization, partitioning, and performance tuning | | `custom-fields-and-flexibility.md` | Implementing EAV patterns, JSON columns, or hybrid approaches for user-defined fields | | `audit-and-soft-deletes.md` | Change tracking, audit trails, soft delete patterns, and compliance logging | | `activities-and-timeline.md` | Polymorphic activity feeds, notes, tasks, emails, calls, and event tracking | | `security-and-multitenancy.md` | Row-level security, role-based access, tenant isolation, and data privacy | | `migrations-and-seeding.md` | Schema versioning, migration scripts, and realistic test data generation | | `reference-schemas.md` | Complete example schemas you can use as starting points | ## Core Design Principles When designing a CRM database on MySQL 8, always follow these principles: 1. **Relational integrity first.** Define FOREIGN KEY constraints at the database level. Never rely on application code alone to maintain referential integrity. 2. **Normalize to 3NF, then denormalize deliberately.** Start at Third Normal Form. Only denormalize when you have measured performance evidence, and document the reason. 3. **Consistent naming conventions.** Use `snake_case` for all identifiers. Table names are plural (`contacts`, `accounts`). Foreign keys follow the pattern `{singular_referenced_table}_id` (e.g., `account_id`). Timestamps are `created_at`, `updated_at`, `deleted_at`. 4. **Every table gets an audit baseline.** At minimum: `id` (BIGINT UNSIGNED AUTO_INCREMENT), `created_at`, `updated_at`. Most CRM tables also need `created_by` and `updated_by`. 5. **Soft deletes over hard deletes.** CRM data has legal, compliance, and historical reporting value. Use `deleted_at` (TIMESTAMP NULL) rather than DELETE statements. 6. **Use BIGINT UNSIGNED for primary keys.** INT runs out at ~2.1 billion. CRM tables like activities and audit logs grow fast. BIGINT UNSIGNED gives you headroom through 18.4 quintillion. 7. **UTF8MB4 everywhere.** Always `CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci`. Customer names, notes, and communications contain international characters and emoji. 8. **InnoDB only.** All tables use InnoDB for transaction support, row-level locking, foreign key enforcement, and crash recovery. 9. **Timestamps use DATETIME(3) or TIMESTAMP.** For CRM, prefer `DATETIME(3)` for event times (timezone-independent, millisecond precision). Use `TIMESTAMP` for `created_at`/`updated_at` with `DEFAULT CURRENT_TIMESTAMP` and `ON UPDATE CURRENT_TIMESTAMP`. 10. **Design for integration.** CRM systems connect to email, marketing, billing, and support tools. Include `external_id` or `external_source` columns on entities that sync with third-party systems. ## Standard Table Template Every CRM table should follow this baseline structure: ```sql CREATE TABLE `table_name` ( `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT, -- entity-specific columns here -- `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, `deleted_at` TIMESTAMP NULL DEFAULT NULL, `created_by` BIGINT UNSIGNED NULL DEFAULT NULL, `updated_by` BIGINT UNSIGNED NULL DEFAULT NULL, PRIMARY KEY (`id`), INDEX `idx_table_name_deleted_at` (`deleted_at`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; ``` ## Workflow for Designing a CRM Schema Follow this sequence when the user asks you to design a CRM database: 1. **Clarify scope.** Determine which CRM modules are needed: contacts/accounts, sales pipeline, marketing/campaigns, support/tickets, or all of the above. 2. **Read the relevant reference files.** Always start with `core-entities.md`. Add others based on the modules identified. 3. **Design entities first, relationships second.** List the tables and their columns, then define the foreign keys and junction tables. 4. **Apply indexing strategy.** Read `indexing-and-performance.md` and add indexes for every foreign key, every column used in WHERE/JOIN/ORDER BY, and composite indexes for common query patterns. 5. **Add flexibility layer.** If the user needs custom fields, read `custom-fields-and-flexibility.md` and choose between EAV, JSON columns, or a hybrid approach. 6. **Add audit and compliance.** Read `audit-and-soft-deletes.md` and implement the appropriate level of change tracking. 7. **Generate migration scripts.** Read `migrations-and-seeding.md` and output versioned, idempotent migration SQL. 8. **Review and validate.** Walk through the schema checking for: missing indexes on FKs, missing NOT NULL constraints, missing default values, orphan risk, and query patterns that would cause full table scans. ## MySQL 8 Features to Leverage These MySQL 8 specific features are particularly valuable for CRM schemas: - **JSON columns** for semi-structured data (custom fields, metadata, integration payloads). See `custom-fields-and-flexibility.md`. - **Generated columns** (VIRTUAL or STORED) to extract and index JSON values. - **Functional indexes** (8.0.13+) to index expressions without explicit generated columns. - **Multi-valued indexes** (8.0.17+) to index JSON arrays efficiently. - **Common Table Expressions (CTEs)** for recursive queries on hierarchical data (org charts, account hierarchies, nested categories). - **Window functions** for pipeline analytics (running totals, rank, lead/lag). - **CHECK constraints** for data validation at the database level. - **DEFAULT expressions** for computed defaults. - **Invisible indexes** for safe testing of index removal. - **Descending indexes** for optimizing ORDER BY ... DESC queries. ## Quick Decision Guide | Situation | Action | |-----------|--------| | User needs a full CRM from scratch | Read `core-entities.md` + `reference-schemas.md`, design all modules | | User needs just contacts + accounts | Read `core-entities.md`, design the contact/account module only | | User asks about custom fields | Read `custom-fields-and-flexibility.md` | | User has performance concerns | Read `indexing-and-performance.md` | | User needs GDPR/compliance support | Read `audit-and-soft-deletes.md` + `security-and-multitenancy.md` | | User is building multi-tenant SaaS CRM | Read `security-and-multitenancy.md` | | User wants to track all user activity | Read `activities-and-timeline.md` | | User needs migration scripts | Read `migrations-and-seeding.md` | | User wants a ready-to-use schema | Read `reference-schemas.md` |

path: references/security-and-multitenancy.md
content: # Security and Multi-Tenancy

CRM systems contain sensitive customer data. This reference covers database-level security patterns, role-based access control, and multi-tenant isolation strategies for MySQL 8.

## Role-Based Access Control (RBAC)

### Simple Role System

For most CRMs, an ENUM on the users table is sufficient:

```sql
ALTER TABLE `users`
    ADD COLUMN `role` ENUM('admin', 'manager', 'sales_rep', 'support_agent', 'read_only')
    NOT NULL DEFAULT 'sales_rep';
```

### Advanced RBAC with Permissions

For fine-grained control, implement a roles-and-permissions system:

```sql
CREATE TABLE `roles` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL,
    `description` TEXT NULL DEFAULT NULL,
    `is_system` TINYINT(1) NOT NULL DEFAULT 0 COMMENT 'System roles cannot be deleted',
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_roles_name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `permissions` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL COMMENT 'e.g., contacts.view, contacts.edit, contacts.delete',
    `entity_type` VARCHAR(50) NOT NULL COMMENT 'e.g., contact, account, opportunity',
    `action` VARCHAR(50) NOT NULL COMMENT 'e.g., view, create, edit, delete, export, import',
    `description` TEXT NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_permissions_name` (`name`),
    INDEX `idx_permissions_entity_action` (`entity_type`, `action`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `role_permissions` (
    `role_id` BIGINT UNSIGNED NOT NULL,
    `permission_id` BIGINT UNSIGNED NOT NULL,

    PRIMARY KEY (`role_id`, `permission_id`),
    INDEX `idx_rp_permission` (`permission_id`),

    CONSTRAINT `fk_rp_role` FOREIGN KEY (`role_id`) REFERENCES `roles` (`id`) ON DELETE CASCADE,
    CONSTRAINT `fk_rp_permission` FOREIGN KEY (`permission_id`) REFERENCES `permissions` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `user_roles` (
    `user_id` BIGINT UNSIGNED NOT NULL,
    `role_id` BIGINT UNSIGNED NOT NULL,

    PRIMARY KEY (`user_id`, `role_id`),
    INDEX `idx_ur_role` (`role_id`),

    CONSTRAINT `fk_ur_user` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE,
    CONSTRAINT `fk_ur_role` FOREIGN KEY (`role_id`) REFERENCES `roles` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

### Record-Level Access (Record Ownership)

CRM commonly restricts data visibility by ownership:
- **Private:** Users see only records they own
- **Team:** Users see records owned by anyone on their team
- **Hierarchy:** Managers see records owned by their direct reports and below
- **Public:** Everyone sees everything

Implement with a teams/territories table:

```sql
CREATE TABLE `teams` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL,
    `description` TEXT NULL DEFAULT NULL,
    `parent_team_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    INDEX `idx_teams_parent` (`parent_team_id`),
    CONSTRAINT `fk_teams_parent` FOREIGN KEY (`parent_team_id`) REFERENCES `teams` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `team_members` (
    `team_id` BIGINT UNSIGNED NOT NULL,
    `user_id` BIGINT UNSIGNED NOT NULL,
    `role_in_team` ENUM('member', 'lead', 'manager') NOT NULL DEFAULT 'member',

    PRIMARY KEY (`team_id`, `user_id`),
    INDEX `idx_tm_user` (`user_id`),

    CONSTRAINT `fk_tm_team` FOREIGN KEY (`team_id`) REFERENCES `teams` (`id`) ON DELETE CASCADE,
    CONSTRAINT `fk_tm_user` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Query pattern for team-based visibility:

```sql
-- Get contacts visible to a user based on their teams
SELECT c.* FROM contacts c
WHERE c.deleted_at IS NULL
    AND (
        c.owner_id = :current_user_id  -- own records
        OR c.owner_id IN (              -- team members' records
            SELECT tm2.user_id
            FROM team_members tm1
            INNER JOIN team_members tm2 ON tm2.team_id = tm1.team_id
            WHERE tm1.user_id = :current_user_id
        )
    );
```

For high-traffic CRM systems, cache the set of visible user IDs per user to avoid running this subquery on every request.

## Multi-Tenancy Patterns

### Option 1: Shared Database, Tenant Column (Most Common)

Add a `tenant_id` column to every table and filter every query by it.

```sql
-- Add tenant_id to all CRM tables
ALTER TABLE `accounts` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;
ALTER TABLE `contacts` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;
ALTER TABLE `opportunities` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;
ALTER TABLE `leads` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;
ALTER TABLE `activities` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;

-- Tenants table
CREATE TABLE `tenants` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(255) NOT NULL,
    `domain` VARCHAR(255) NULL DEFAULT NULL,
    `plan` ENUM('free', 'starter', 'professional', 'enterprise') NOT NULL DEFAULT 'free',
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `settings` JSON NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_tenants_domain` (`domain`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Critical rules for shared-database multi-tenancy:

1. **Every query includes tenant_id.** No exceptions. A missed filter exposes data across tenants.

2. **tenant_id is the first column in every composite index:**
```sql
-- Correct: tenant_id first
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_tenant_email` (`tenant_id`, `email`);
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_tenant_account` (`tenant_id`, `account_id`);

-- Wrong: tenant_id missing or not first
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_email` (`email`);  -- leaks across tenants
```

3. **Unique constraints include tenant_id:**
```sql
-- Correct: unique per tenant
ALTER TABLE `pipelines`
    ADD UNIQUE INDEX `uq_pipelines_tenant_name` (`tenant_id`, `name`);

-- Wrong: globally unique (tenant A and B can't both have "Sales Pipeline")
ALTER TABLE `pipelines`
    ADD UNIQUE INDEX `uq_pipelines_name` (`name`);
```

4. **Foreign keys should stay within a tenant.** Add application-level checks to ensure cross-table references share the same `tenant_id`.

### Option 2: Separate Schema per Tenant

Each tenant gets their own MySQL schema (database). The application connects to the right schema based on the tenant.

```sql
CREATE DATABASE `crm_tenant_acme`;
CREATE DATABASE `crm_tenant_globex`;
```

Pros:
- Complete isolation — no risk of cross-tenant data leakage
- Easy to back up, restore, or migrate individual tenants
- Per-tenant schema customization is possible

Cons:
- Schema migrations must be applied to every tenant database
- Connection pool management becomes complex
- Harder to run cross-tenant analytics or admin queries
- MySQL has limits on the number of open tables/databases

Best for: Enterprise CRM with strict isolation requirements and fewer than 100 tenants.

### Option 3: Separate Database Server per Tenant

Each tenant gets a dedicated MySQL instance. Maximum isolation but highest operational cost. Only appropriate for very large enterprise customers with regulatory requirements.

## Data Encryption

### Encryption at Rest

Enable InnoDB tablespace encryption:

```sql
-- Enable encryption for a table
ALTER TABLE `contacts` ENCRYPTION='Y';

-- Enable encryption for all new tables by default
SET GLOBAL default_table_encryption = ON;
```

Requires configuring a keyring plugin (e.g., `keyring_file`, `keyring_encrypted_file`, or a KMS-backed keyring like `keyring_aws`).

### Encryption of Sensitive Columns

For columns containing highly sensitive data (SSN, credit card info, health data), consider application-level encryption:

```sql
-- Store encrypted values
ALTER TABLE `contacts`
    ADD COLUMN `ssn_encrypted` VARBINARY(255) NULL DEFAULT NULL,
    ADD COLUMN `ssn_hash` VARCHAR(64) NULL DEFAULT NULL COMMENT 'SHA-256 hash for lookup';
```

Encrypt/decrypt in the application layer using AES-256-GCM or similar. Store a hash for indexed lookups.

**Never store plaintext:** passwords, social security numbers, credit card numbers, bank account numbers, API keys, or authentication tokens.

## Input Validation at Database Level

Use CHECK constraints (MySQL 8.0.16+) to enforce data quality:

```sql
ALTER TABLE `contacts`
    ADD CONSTRAINT `chk_contacts_email` CHECK (
        `email` IS NULL OR `email` REGEXP '^[^@]+@[^@]+\\.[^@]+$'
    ),
    ADD CONSTRAINT `chk_contacts_phone` CHECK (
        `phone` IS NULL OR LENGTH(`phone`) >= 7
    );

ALTER TABLE `opportunities`
    ADD CONSTRAINT `chk_opp_amount_positive` CHECK (
        `amount` IS NULL OR `amount` >= 0
    ),
    ADD CONSTRAINT `chk_opp_probability_range` CHECK (
        `probability` IS NULL OR (`probability` >= 0 AND `probability` <= 100)
    );

ALTER TABLE `accounts`
    ADD CONSTRAINT `chk_accounts_country_code` CHECK (
        `billing_country` IS NULL OR LENGTH(`billing_country`) = 2
    );
```

CHECK constraints are not a substitute for application-level validation, but they provide a last line of defense against bad data.

## MySQL User Privileges

Follow the principle of least privilege for database users:

```sql
-- Application user: read and write to CRM tables only
CREATE USER 'crm_app'@'%' IDENTIFIED BY '<strong_password>';
GRANT SELECT, INSERT, UPDATE, DELETE ON crm.* TO 'crm_app'@'%';

-- Read-only reporting user
CREATE USER 'crm_reports'@'%' IDENTIFIED BY '<strong_password>';
GRANT SELECT ON crm.* TO 'crm_reports'@'%';

-- Migration user: can alter schema
CREATE USER 'crm_migrate'@'%' IDENTIFIED BY '<strong_password>';
GRANT ALL PRIVILEGES ON crm.* TO 'crm_migrate'@'%';

-- Never use root for application connections
```


path: references/activities-and-timeline.md
content: # Activities and Timeline

A CRM's value comes from recording every interaction with customers. This reference covers designing the activity tracking system — the polymorphic timeline that captures calls, emails, meetings, notes, tasks, and any other touchpoint.

## The Activity Model

### Core Design: Polymorphic Activity Table

Activities can relate to any CRM entity (contact, account, opportunity, lead). Use a polymorphic reference pattern:

```sql
CREATE TABLE `activities` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,

    -- What type of activity
    `activity_type` ENUM('call', 'email', 'meeting', 'note', 'task',
                          'sms', 'chat', 'social', 'document', 'status_change',
                          'stage_change', 'assignment', 'system') NOT NULL,

    -- What entity this activity relates to (polymorphic)
    `entity_type` VARCHAR(50) NOT NULL COMMENT 'contact, account, opportunity, lead',
    `entity_id` BIGINT UNSIGNED NOT NULL,

    -- When it happened
    `activity_date` DATETIME(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),

    -- Who performed the activity
    `user_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'CRM user who performed this',

    -- Content
    `subject` VARCHAR(500) NULL DEFAULT NULL,
    `body` TEXT NULL DEFAULT NULL,
    `body_html` MEDIUMTEXT NULL DEFAULT NULL COMMENT 'Rich text version for emails',

    -- Activity-specific metadata stored as JSON
    `metadata` JSON NULL DEFAULT NULL,

    -- Duration tracking (for calls, meetings)
    `duration_minutes` INT UNSIGNED NULL DEFAULT NULL,

    -- Task-specific fields
    `is_completed` TINYINT(1) NULL DEFAULT NULL COMMENT 'Only for tasks',
    `due_date` DATETIME NULL DEFAULT NULL COMMENT 'Only for tasks',
    `priority` ENUM('low', 'medium', 'high', 'urgent') NULL DEFAULT NULL,

    -- Associations (an activity can also link to secondary entities)
    `account_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'Denormalized for fast account timeline queries',
    `contact_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'Specific contact involved',
    `opportunity_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'Related deal',

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_activities_entity` (`entity_type`, `entity_id`, `activity_date` DESC),
    INDEX `idx_activities_user` (`user_id`, `activity_date` DESC),
    INDEX `idx_activities_type` (`activity_type`, `activity_date` DESC),
    INDEX `idx_activities_account` (`account_id`, `activity_date` DESC),
    INDEX `idx_activities_contact` (`contact_id`, `activity_date` DESC),
    INDEX `idx_activities_opportunity` (`opportunity_id`, `activity_date` DESC),
    INDEX `idx_activities_due_date` (`due_date`, `is_completed`),
    INDEX `idx_activities_deleted_at` (`deleted_at`),

    CONSTRAINT `fk_activities_user` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_activities_account` FOREIGN KEY (`account_id`) REFERENCES `accounts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_activities_contact` FOREIGN KEY (`contact_id`) REFERENCES `contacts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_activities_opportunity` FOREIGN KEY (`opportunity_id`) REFERENCES `opportunities` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_activities_created_by` FOREIGN KEY (`created_by`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

### Why Denormalized Association Columns?

The `account_id`, `contact_id`, and `opportunity_id` columns on the activities table are intentionally denormalized. Here is why:

- The most common CRM query is "show me the timeline for account X" — having `account_id` directly on activities avoids a JOIN through the polymorphic entity reference.
- An activity might relate to multiple entities simultaneously (a call with Contact A about Opportunity B at Account C).
- The denormalized FKs enable proper foreign key constraints, which the polymorphic `entity_type` + `entity_id` pattern cannot.

### Metadata by Activity Type

The `metadata` JSON column stores activity-type-specific data. Document the expected structure per type:

**Call:**
```json
{
    "direction": "outbound",
    "phone_number": "+1-555-0123",
    "outcome": "connected",
    "recording_url": "https://...",
    "voicemail_left": false
}
```

**Email:**
```json
{
    "direction": "outbound",
    "from": "[REDACTED_SECRET]",
    "to": ["[REDACTED_SECRET]"],
    "cc": [],
    "message_id": "<abc123@mail>",
    "thread_id": "thread_456",
    "has_attachments": true,
    "opened": true,
    "opened_at": "2026-04-01T10:30:00Z",
    "clicked": false
}
```

**Meeting:**
```json
{
    "location": "Zoom",
    "meeting_url": "https://zoom.us/j/123",
    "attendees": [
        {"email": "[REDACTED_SECRET]", "status": "accepted"},
        {"email": "[REDACTED_SECRET]", "status": "accepted"}
    ],
    "calendar_event_id": "cal_abc123",
    "outcome": "completed"
}
```

**Stage Change (system-generated):**
```json
{
    "old_stage_id": 3,
    "new_stage_id": 4,
    "old_stage_name": "Proposal",
    "new_stage_name": "Negotiation",
    "pipeline_id": 1
}
```

### Multi-Entity Activity Associations

An activity can relate to multiple entities. For example, a meeting might involve three contacts from two accounts about one opportunity. Use a junction table for additional associations beyond the primary:

```sql
CREATE TABLE `activity_associations` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `activity_id` BIGINT UNSIGNED NOT NULL,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `association_type` VARCHAR(50) NULL DEFAULT NULL COMMENT 'e.g., attendee, mentioned, related',
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_activity_assoc` (`activity_id`, `entity_type`, `entity_id`),
    INDEX `idx_activity_assoc_entity` (`entity_type`, `entity_id`, `created_at` DESC),

    CONSTRAINT `fk_activity_assoc_activity` FOREIGN KEY (`activity_id`)
        REFERENCES `activities` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

## Notes

Notes are a subset of activities but are so commonly queried independently that some CRM systems give them their own table:

```sql
CREATE TABLE `notes` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `title` VARCHAR(255) NULL DEFAULT NULL,
    `body` TEXT NOT NULL,
    `is_pinned` TINYINT(1) NOT NULL DEFAULT 0,
    `user_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_notes_entity` (`entity_type`, `entity_id`, `is_pinned` DESC, `created_at` DESC),
    INDEX `idx_notes_user` (`user_id`),
    FULLTEXT INDEX `ft_notes_search` (`title`, `body`),

    CONSTRAINT `fk_notes_user` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

## Tasks

Tasks deserve their own table when the CRM has a dedicated task management workflow (assigned tasks, due dates, reminders, recurring tasks).

```sql
CREATE TABLE `tasks` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `title` VARCHAR(500) NOT NULL,
    `description` TEXT NULL DEFAULT NULL,
    `status` ENUM('open', 'in_progress', 'completed', 'cancelled') NOT NULL DEFAULT 'open',
    `priority` ENUM('low', 'medium', 'high', 'urgent') NOT NULL DEFAULT 'medium',
    `due_date` DATETIME NULL DEFAULT NULL,
    `completed_at` DATETIME NULL DEFAULT NULL,

    -- Who is responsible
    `assignee_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `assigned_by_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- What entity is this task related to
    `entity_type` VARCHAR(50) NULL DEFAULT NULL,
    `entity_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- Recurrence (optional)
    `is_recurring` TINYINT(1) NOT NULL DEFAULT 0,
    `recurrence_rule` VARCHAR(255) NULL DEFAULT NULL COMMENT 'iCal RRULE format',

    -- Reminders
    `reminder_at` DATETIME NULL DEFAULT NULL,

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_tasks_assignee` (`assignee_id`, `status`, `due_date`),
    INDEX `idx_tasks_entity` (`entity_type`, `entity_id`),
    INDEX `idx_tasks_due_date` (`due_date`, `status`),
    INDEX `idx_tasks_status` (`status`),
    INDEX `idx_tasks_reminder` (`reminder_at`, `status`),
    INDEX `idx_tasks_deleted_at` (`deleted_at`),

    CONSTRAINT `fk_tasks_assignee` FOREIGN KEY (`assignee_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_tasks_assigned_by` FOREIGN KEY (`assigned_by_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_tasks_created_by` FOREIGN KEY (`created_by`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

## Attachments / Files

Documents attached to any CRM entity:

```sql
CREATE TABLE `attachments` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `filename` VARCHAR(255) NOT NULL,
    `file_path` VARCHAR(1000) NOT NULL COMMENT 'S3 key or filesystem path',
    `file_size` BIGINT UNSIGNED NOT NULL COMMENT 'Size in bytes',
    `mime_type` VARCHAR(100) NOT NULL,
    `uploaded_by` BIGINT UNSIGNED NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_attachments_entity` (`entity_type`, `entity_id`),
    INDEX `idx_attachments_uploaded_by` (`uploaded_by`),

    CONSTRAINT `fk_attachments_uploaded_by` FOREIGN KEY (`uploaded_by`)
        REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- Never store file contents (BLOBs) in MySQL. Store files in object storage (S3, GCS) and keep only the path/key in the database.
- `mime_type` enables the UI to display previews or choose appropriate icons.
- `file_size` allows enforcing storage quotas without querying the file system.

## Building the Unified Timeline Query

The timeline view for any CRM entity aggregates activities, notes, and system events:

```sql
-- Unified timeline for an account
(
    SELECT
        a.id,
        'activity' AS record_type,
        a.activity_type AS event_type,
        a.subject AS title,
        a.body AS description,
        a.activity_date AS event_date,
        a.user_id,
        u.first_name AS user_first_name,
        u.last_name AS user_last_name
    FROM activities a
    LEFT JOIN users u ON u.id = a.user_id
    WHERE a.account_id = :account_id AND a.deleted_at IS NULL
)
UNION ALL
(
    SELECT
        n.id,
        'note' AS record_type,
        'note' AS event_type,
        n.title,
        LEFT(n.body, 200) AS description,
        n.created_at AS event_date,
        n.user_id,
        u.first_name,
        u.last_name
    FROM notes n
    LEFT JOIN users u ON u.id = n.user_id
    WHERE n.entity_type = 'account' AND n.entity_id = :account_id AND n.deleted_at IS NULL
)
ORDER BY event_date DESC
LIMIT 50;
```

For performance, consider a materialized timeline view updated by application events or triggers, especially for accounts with thousands of activities.


path: references/indexing-and-performance.md
content: # Indexing and Performance

This reference covers indexing strategies, query optimization, and performance tuning for CRM databases on MySQL 8.

## Indexing Fundamentals for CRM

### Index Every Foreign Key

MySQL does not automatically index foreign key columns (InnoDB indexes the referencing column in some cases, but you should always be explicit). Without an index on every FK column:
- JOINs between parent and child tables become full table scans
- DELETE on a parent row scans the entire child table to check references
- This can cause table-level locks and cascading performance problems

```sql
-- Ensure all FK columns are indexed
ALTER TABLE `contacts` ADD INDEX `idx_contacts_account_id` (`account_id`);
ALTER TABLE `contacts` ADD INDEX `idx_contacts_owner_id` (`owner_id`);
ALTER TABLE `opportunities` ADD INDEX `idx_opportunities_account_id` (`account_id`);
ALTER TABLE `opportunities` ADD INDEX `idx_opportunities_stage_id` (`stage_id`);
ALTER TABLE `opportunities` ADD INDEX `idx_opportunities_owner_id` (`owner_id`);
ALTER TABLE `activities` ADD INDEX `idx_activities_user_id` (`user_id`);
```

### Composite Indexes for Common CRM Queries

Design composite indexes around the actual queries your CRM runs most. The column order matters — put the most selective or most commonly filtered column first.

```sql
-- Pipeline board: "Show all active deals in pipeline X, grouped by stage"
ALTER TABLE `opportunities`
    ADD INDEX `idx_opp_pipeline_board` (`pipeline_id`, `stage_id`, `deleted_at`);

-- My deals: "Show deals owned by me, sorted by expected close date"
ALTER TABLE `opportunities`
    ADD INDEX `idx_opp_owner_close` (`owner_id`, `expected_close_date`, `deleted_at`);

-- Contact search: "Find contacts by last name within an account"
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_account_name` (`account_id`, `last_name`, `first_name`);

-- Activity timeline: "Recent activities for a given entity"
ALTER TABLE `activities`
    ADD INDEX `idx_activities_entity_timeline` (`entity_type`, `entity_id`, `activity_date` DESC);

-- Lead queue: "Unassigned new leads sorted by creation date"
ALTER TABLE `leads`
    ADD INDEX `idx_leads_new_unassigned` (`status`, `owner_id`, `created_at`);
```

### Covering Indexes

A covering index includes all columns a query needs, so MySQL can satisfy the query from the index alone (no table lookup). This is especially valuable for list views and dashboards.

```sql
-- Covering index for opportunity list view
ALTER TABLE `opportunities`
    ADD INDEX `idx_opp_list_view` (
        `owner_id`, `deleted_at`, `stage_id`,
        `name`, `amount`, `expected_close_date`
    );
```

Use `EXPLAIN` and check for "Using index" in the Extra column to verify a covering index is working.

### Prefix Indexes for Long Text Columns

For VARCHAR columns longer than 191 characters (the max for utf8mb4 with a 767-byte index limit on older row formats), use prefix indexes:

```sql
-- Index only the first 50 characters of description
ALTER TABLE `accounts` ADD INDEX `idx_accounts_description` (`description`(50));

-- For URLs, index a meaningful prefix
ALTER TABLE `contacts` ADD INDEX `idx_contacts_linkedin` (`linkedin_url`(100));
```

Prefer generated columns with indexes over prefix indexes when you need exact match queries.

## MySQL 8 Specific Index Features

### Functional Indexes (MySQL 8.0.13+)

Index an expression without creating an explicit generated column:

```sql
-- Index on email domain for account matching
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_email_domain` ((SUBSTRING_INDEX(`email`, '@', -1)));

-- Index on year of created_at for reporting
ALTER TABLE `opportunities`
    ADD INDEX `idx_opp_created_year` ((YEAR(`created_at`)));

-- Case-insensitive email search
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_email_lower` ((LOWER(`email`)));
```

Important: the query must use the exact same expression as the index definition for MySQL to use it.

### Descending Indexes

MySQL 8 supports true descending indexes (earlier versions ignored DESC in index definitions):

```sql
-- Optimize "most recent first" queries
ALTER TABLE `activities`
    ADD INDEX `idx_activities_recent` (`entity_type`, `entity_id`, `activity_date` DESC);

-- Dashboard: latest deals
ALTER TABLE `opportunities`
    ADD INDEX `idx_opp_latest` (`created_at` DESC, `deleted_at`);
```

### Invisible Indexes

Test the impact of removing an index without actually dropping it:

```sql
-- Make an index invisible (optimizer ignores it)
ALTER TABLE `contacts` ALTER INDEX `idx_contacts_lead_source` INVISIBLE;

-- Run your workload and measure impact...

-- Make it visible again if needed
ALTER TABLE `contacts` ALTER INDEX `idx_contacts_lead_source` VISIBLE;

-- Or drop it if performance was unaffected
DROP INDEX `idx_contacts_lead_source` ON `contacts`;
```

### JSON Indexing Strategies

CRM systems often store semi-structured data in JSON columns (custom fields, metadata, integration payloads). MySQL cannot directly index JSON columns, but offers three approaches:

**Approach 1: Generated columns (works in MySQL 5.7+)**

```sql
-- Add a generated column that extracts a JSON value
ALTER TABLE `contacts`
    ADD COLUMN `custom_industry` VARCHAR(100)
        GENERATED ALWAYS AS (`custom_fields` ->> '$.industry') VIRTUAL,
    ADD INDEX `idx_contacts_custom_industry` (`custom_industry`);
```

**Approach 2: Functional indexes (MySQL 8.0.13+)**

```sql
-- Index a JSON path directly
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_json_industry` ((
        CAST(`custom_fields` ->> '$.industry' AS CHAR(100)) COLLATE utf8mb4_bin
    ));
```

**Approach 3: Multi-valued indexes for JSON arrays (MySQL 8.0.17+)**

```sql
-- Index elements of a JSON array
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_interests` ((
        CAST(`custom_fields` -> '$.interests' AS CHAR(50) ARRAY)
    ));

-- Query using MEMBER OF
SELECT * FROM contacts
WHERE 'machine_learning' MEMBER OF (custom_fields -> '$.interests');
```

Recommendation: Use generated STORED columns for frequently queried JSON paths — they are the most reliable and performant approach. Use functional indexes for occasional queries. Use multi-valued indexes when you need to search within JSON arrays.

## Query Optimization Patterns

### The Soft Delete Filter

Almost every CRM query needs to filter out soft-deleted records. Always include `deleted_at IS NULL` and make sure your indexes account for it:

```sql
-- BAD: Index doesn't include deleted_at
SELECT * FROM contacts WHERE account_id = 42;

-- GOOD: Index on (account_id, deleted_at), query filters both
SELECT * FROM contacts WHERE account_id = 42 AND deleted_at IS NULL;
```

Consider putting `deleted_at` as the last column in composite indexes so the optimizer can use the index for both "all records" and "non-deleted records" queries.

### Pagination

CRM list views need efficient pagination. Avoid `OFFSET` for large datasets.

```sql
-- BAD: OFFSET becomes slow as page number grows
SELECT * FROM contacts WHERE deleted_at IS NULL ORDER BY id LIMIT 25 OFFSET 10000;

-- GOOD: Keyset pagination using the last seen ID
SELECT * FROM contacts
WHERE deleted_at IS NULL AND id > :last_seen_id
ORDER BY id
LIMIT 25;
```

For sorted pagination on non-unique columns, use a composite cursor:

```sql
-- Paginate contacts sorted by last_name
SELECT * FROM contacts
WHERE deleted_at IS NULL
    AND (last_name > :last_name OR (last_name = :last_name AND id > :last_id))
ORDER BY last_name, id
LIMIT 25;
```

### Counting Large Tables

Avoid `SELECT COUNT(*)` on large CRM tables for UI display. Instead:
- Use approximate counts: `SELECT TABLE_ROWS FROM information_schema.TABLES WHERE TABLE_NAME = 'contacts'`
- Cache counts in a summary table updated by triggers or scheduled jobs
- Use `SQL_CALC_FOUND_ROWS` or window functions for "page X of Y" display

### Full-Text Search

CRM users expect to search across names, emails, descriptions, and notes. Use MySQL full-text indexes:

```sql
ALTER TABLE `contacts`
    ADD FULLTEXT INDEX `ft_contacts_search` (`first_name`, `last_name`, `email`);

ALTER TABLE `accounts`
    ADD FULLTEXT INDEX `ft_accounts_search` (`name`, `description`, `domain`);

-- Search query
SELECT * FROM contacts
WHERE MATCH(first_name, last_name, email) AGAINST ('john smith' IN BOOLEAN MODE)
    AND deleted_at IS NULL;
```

For more sophisticated search (typo tolerance, relevance ranking, faceting), consider offloading to Elasticsearch or Meilisearch and keeping MySQL as the source of truth.

## Partitioning for Large CRM Tables

Partitioning is useful for tables that grow very large (tens of millions of rows), particularly for time-series data.

### Good Candidates for Partitioning

- `activities` — partitioned by `activity_date` (RANGE partitioning by month or year)
- `audit_logs` — partitioned by `created_at`
- `email_events` — partitioned by `event_date`

### Example: Partitioning Activities by Year

```sql
CREATE TABLE `activities` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `activity_type` VARCHAR(50) NOT NULL,
    `activity_date` DATETIME NOT NULL,
    `user_id` BIGINT UNSIGNED NULL,
    `summary` VARCHAR(500) NULL,
    `details` JSON NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`, `activity_date`),
    INDEX `idx_activities_entity` (`entity_type`, `entity_id`, `activity_date`),
    INDEX `idx_activities_user` (`user_id`, `activity_date`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
PARTITION BY RANGE (YEAR(`activity_date`)) (
    PARTITION p2023 VALUES LESS THAN (2024),
    PARTITION p2024 VALUES LESS THAN (2025),
    PARTITION p2025 VALUES LESS THAN (2026),
    PARTITION p2026 VALUES LESS THAN (2027),
    PARTITION p_future VALUES LESS THAN MAXVALUE
);
```

Important constraints:
- The partition key must be part of every unique index (including the primary key). That is why `activity_date` is included in the PK.
- Foreign keys are not supported on partitioned tables. Use application-level integrity checks.
- Add new partitions proactively before the year rolls over (automate this with a scheduled job).

### Tables NOT to Partition

Do not partition `accounts`, `contacts`, `opportunities`, or `users`. These are looked up by ID across all time periods, and partitioning would not help (and may hurt) these access patterns.

## Performance Monitoring Checklist

Run these periodically on your CRM database:

```sql
-- Find tables without primary keys (should be zero)
SELECT TABLE_NAME FROM information_schema.TABLES
WHERE TABLE_SCHEMA = DATABASE()
    AND TABLE_TYPE = 'BASE TABLE'
    AND TABLE_NAME NOT IN (
        SELECT TABLE_NAME FROM information_schema.TABLE_CONSTRAINTS
        WHERE CONSTRAINT_TYPE = 'PRIMARY KEY' AND TABLE_SCHEMA = DATABASE()
    );

-- Find foreign key columns without indexes
SELECT
    kcu.TABLE_NAME, kcu.COLUMN_NAME
FROM information_schema.KEY_COLUMN_USAGE kcu
LEFT JOIN information_schema.STATISTICS s
    ON s.TABLE_SCHEMA = kcu.TABLE_SCHEMA
    AND s.TABLE_NAME = kcu.TABLE_NAME
    AND s.COLUMN_NAME = kcu.COLUMN_NAME
WHERE kcu.TABLE_SCHEMA = DATABASE()
    AND kcu.REFERENCED_TABLE_NAME IS NOT NULL
    AND s.INDEX_NAME IS NULL;

-- Find slow queries (if slow_query_log is enabled)
-- Look for queries with no index usage
SELECT * FROM mysql.slow_log
WHERE query_time > '00:00:01'
ORDER BY start_time DESC LIMIT 20;
```


path: references/reference-schemas.md
content: # Reference Schemas

This file contains complete, ready-to-use CRM database schemas for common scenarios. Copy and adapt as needed.

## Schema A: Minimal CRM (Contacts + Deals)

For small teams that need basic contact and deal tracking. Approximately 8 tables.

```sql
-- ============================================================
-- Minimal CRM Schema
-- MySQL 8.0+
-- ============================================================

SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;

-- Users
CREATE TABLE IF NOT EXISTS `users` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `email` VARCHAR(255) NOT NULL,
    `password_hash` VARCHAR(255) NOT NULL,
    `first_name` VARCHAR(100) NOT NULL,
    `last_name` VARCHAR(100) NOT NULL,
    `role` ENUM('admin','manager','sales_rep','read_only') NOT NULL DEFAULT 'sales_rep',
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_users_email` (`email`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Accounts
CREATE TABLE IF NOT EXISTS `accounts` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(255) NOT NULL,
    `domain` VARCHAR(255) NULL,
    `industry` VARCHAR(100) NULL,
    `type` ENUM('prospect','customer','partner','other') NOT NULL DEFAULT 'prospect',
    `phone` VARCHAR(30) NULL,
    `owner_id` BIGINT UNSIGNED NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    PRIMARY KEY (`id`),
    INDEX `idx_accounts_owner` (`owner_id`),
    INDEX `idx_accounts_deleted_at` (`deleted_at`),
    CONSTRAINT `fk_accounts_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Contacts
CREATE TABLE IF NOT EXISTS `contacts` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `account_id` BIGINT UNSIGNED NULL,
    `first_name` VARCHAR(100) NOT NULL,
    `last_name` VARCHAR(100) NOT NULL,
    `email` VARCHAR(255) NULL,
    `phone` VARCHAR(30) NULL,
    `job_title` VARCHAR(150) NULL,
    `owner_id` BIGINT UNSIGNED NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    PRIMARY KEY (`id`),
    INDEX `idx_contacts_account` (`account_id`),
    INDEX `idx_contacts_email` (`email`),
    INDEX `idx_contacts_name` (`last_name`, `first_name`),
    INDEX `idx_contacts_owner` (`owner_id`),
    INDEX `idx_contacts_deleted_at` (`deleted_at`),
    CONSTRAINT `fk_contacts_account` FOREIGN KEY (`account_id`) REFERENCES `accounts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_contacts_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Pipeline Stages
CREATE TABLE IF NOT EXISTS `pipeline_stages` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL,
    `display_order` INT UNSIGNED NOT NULL DEFAULT 0,
    `probability` DECIMAL(5,2) NULL,
    `is_won` TINYINT(1) NOT NULL DEFAULT 0,
    `is_lost` TINYINT(1) NOT NULL DEFAULT 0,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Opportunities
CREATE TABLE IF NOT EXISTS `opportunities` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(255) NOT NULL,
    `account_id` BIGINT UNSIGNED NOT NULL,
    `stage_id` BIGINT UNSIGNED NOT NULL,
    `amount` DECIMAL(15,2) NULL,
    `expected_close_date` DATE NULL,
    `owner_id` BIGINT UNSIGNED NULL,
    `primary_contact_id` BIGINT UNSIGNED NULL,
    `description` TEXT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    PRIMARY KEY (`id`),
    INDEX `idx_opp_account` (`account_id`),
    INDEX `idx_opp_stage` (`stage_id`),
    INDEX `idx_opp_owner` (`owner_id`),
    INDEX `idx_opp_close_date` (`expected_close_date`),
    INDEX `idx_opp_deleted_at` (`deleted_at`),
    CONSTRAINT `fk_opp_account` FOREIGN KEY (`account_id`) REFERENCES `accounts` (`id`) ON DELETE RESTRICT,
    CONSTRAINT `fk_opp_stage` FOREIGN KEY (`stage_id`) REFERENCES `pipeline_stages` (`id`) ON DELETE RESTRICT,
    CONSTRAINT `fk_opp_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_opp_contact` FOREIGN KEY (`primary_contact_id`) REFERENCES `contacts` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Activities (simplified)
CREATE TABLE IF NOT EXISTS `activities` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `activity_type` ENUM('call','email','meeting','note','task') NOT NULL,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `subject` VARCHAR(500) NULL,
    `body` TEXT NULL,
    `activity_date` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `user_id` BIGINT UNSIGNED NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    PRIMARY KEY (`id`),
    INDEX `idx_activities_entity` (`entity_type`, `entity_id`, `activity_date` DESC),
    INDEX `idx_activities_user` (`user_id`),
    CONSTRAINT `fk_activities_user` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Tags
CREATE TABLE IF NOT EXISTS `tags` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL,
    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_tags_name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE IF NOT EXISTS `taggables` (
    `tag_id` BIGINT UNSIGNED NOT NULL,
    `taggable_type` VARCHAR(50) NOT NULL,
    `taggable_id` BIGINT UNSIGNED NOT NULL,
    PRIMARY KEY (`tag_id`, `taggable_type`, `taggable_id`),
    INDEX `idx_taggables_target` (`taggable_type`, `taggable_id`),
    CONSTRAINT `fk_taggables_tag` FOREIGN KEY (`tag_id`) REFERENCES `tags` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

SET FOREIGN_KEY_CHECKS = 1;

-- Default pipeline stages
INSERT INTO `pipeline_stages` (`name`, `display_order`, `probability`, `is_won`, `is_lost`) VALUES
('Qualification', 1, 10.00, 0, 0),
('Discovery', 2, 20.00, 0, 0),
('Proposal', 3, 50.00, 0, 0),
('Negotiation', 4, 75.00, 0, 0),
('Closed Won', 5, 100.00, 1, 0),
('Closed Lost', 6, 0.00, 0, 1);
```

## Schema B: Full-Featured CRM

For teams that need the complete CRM stack: contacts, accounts, leads, multi-pipeline deals, products, activities, tasks, notes, audit trail, custom fields, RBAC, and tags. This builds on Schema A and adds all the enterprise features.

Tables included (in creation order):
1. `users`
2. `roles`, `permissions`, `role_permissions`, `user_roles`
3. `teams`, `team_members`
4. `accounts`
5. `contacts`
6. `leads`
7. `pipelines`, `pipeline_stages`
8. `products`
9. `opportunities`
10. `opportunity_line_items`
11. `opportunity_contacts`
12. `activities`
13. `activity_associations`
14. `notes`
15. `tasks`
16. `attachments`
17. `tags`, `taggables`
18. `custom_field_definitions`, `custom_field_values`
19. `audit_logs`

See the individual reference files for the complete CREATE TABLE statements for each. Combine them in the order listed above to produce a full migration script.

## Schema C: Multi-Tenant SaaS CRM

Extends Schema B with a `tenant_id` column on every table. Key differences:

```sql
-- Add to every CRM table:
ALTER TABLE `accounts` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;
ALTER TABLE `contacts` ADD COLUMN `tenant_id` BIGINT UNSIGNED NOT NULL AFTER `id`;
-- ... repeat for all tables

-- Modify every index to be tenant-scoped:
ALTER TABLE `contacts` DROP INDEX `idx_contacts_email`;
ALTER TABLE `contacts` ADD INDEX `idx_contacts_tenant_email` (`tenant_id`, `email`);

-- Modify unique constraints to be tenant-scoped:
ALTER TABLE `pipelines` ADD UNIQUE INDEX `uq_pipelines_tenant_name` (`tenant_id`, `name`);
```

See `security-and-multitenancy.md` for the complete multi-tenancy implementation guide.

## Common CRM Queries Reference

These are the most commonly run queries in a CRM, useful for verifying that your schema and indexes support them efficiently:

```sql
-- 1. Pipeline board (kanban view)
SELECT o.*, ps.name AS stage_name, a.name AS account_name
FROM opportunities o
INNER JOIN pipeline_stages ps ON ps.id = o.stage_id
INNER JOIN accounts a ON a.id = o.account_id
WHERE o.pipeline_id = :pipeline_id AND o.deleted_at IS NULL
ORDER BY ps.display_order, o.updated_at DESC;

-- 2. My open deals
SELECT o.*, a.name AS account_name
FROM opportunities o
INNER JOIN accounts a ON a.id = o.account_id
WHERE o.owner_id = :user_id
    AND o.deleted_at IS NULL
    AND o.stage_id NOT IN (SELECT id FROM pipeline_stages WHERE is_won = 1 OR is_lost = 1)
ORDER BY o.expected_close_date;

-- 3. Account 360 view
SELECT a.*,
    (SELECT COUNT(*) FROM contacts c WHERE c.account_id = a.id AND c.deleted_at IS NULL) AS contact_count,
    (SELECT COUNT(*) FROM opportunities o WHERE o.account_id = a.id AND o.deleted_at IS NULL) AS deal_count,
    (SELECT SUM(o.amount) FROM opportunities o
     INNER JOIN pipeline_stages ps ON ps.id = o.stage_id
     WHERE o.account_id = a.id AND ps.is_won = 1 AND o.deleted_at IS NULL) AS total_won_revenue
FROM accounts a
WHERE a.id = :account_id;

-- 4. Lead conversion funnel
SELECT
    source,
    COUNT(*) AS total_leads,
    SUM(CASE WHEN status = 'converted' THEN 1 ELSE 0 END) AS converted,
    ROUND(SUM(CASE WHEN status = 'converted' THEN 1 ELSE 0 END) / COUNT(*) * 100, 1) AS conversion_rate
FROM leads
WHERE deleted_at IS NULL AND created_at >= :start_date
GROUP BY source
ORDER BY total_leads DESC;

-- 5. Sales forecast
SELECT
    ps.name AS stage_name,
    COUNT(*) AS deal_count,
    SUM(o.amount) AS total_value,
    SUM(o.amount * COALESCE(o.probability, ps.probability) / 100) AS weighted_value
FROM opportunities o
INNER JOIN pipeline_stages ps ON ps.id = o.stage_id
WHERE o.pipeline_id = :pipeline_id
    AND o.deleted_at IS NULL
    AND ps.is_won = 0 AND ps.is_lost = 0
    AND o.expected_close_date BETWEEN :start_date AND :end_date
GROUP BY ps.id, ps.name, ps.display_order
ORDER BY ps.display_order;

-- 6. Activity leaderboard (rep productivity)
SELECT
    u.first_name, u.last_name,
    COUNT(*) AS total_activities,
    SUM(CASE WHEN a.activity_type = 'call' THEN 1 ELSE 0 END) AS calls,
    SUM(CASE WHEN a.activity_type = 'email' THEN 1 ELSE 0 END) AS emails,
    SUM(CASE WHEN a.activity_type = 'meeting' THEN 1 ELSE 0 END) AS meetings
FROM activities a
INNER JOIN users u ON u.id = a.user_id
WHERE a.activity_date BETWEEN :start_date AND :end_date
    AND a.deleted_at IS NULL
GROUP BY u.id, u.first_name, u.last_name
ORDER BY total_activities DESC;

-- 7. Contacts due for follow-up (no activity in 30 days)
SELECT c.*, a.name AS account_name,
    MAX(act.activity_date) AS last_activity_date
FROM contacts c
LEFT JOIN accounts a ON a.id = c.account_id
LEFT JOIN activities act ON act.contact_id = c.id AND act.deleted_at IS NULL
WHERE c.deleted_at IS NULL
    AND c.lifecycle_stage IN ('lead', 'mql', 'sql', 'opportunity')
GROUP BY c.id
HAVING last_activity_date < DATE_SUB(NOW(), INTERVAL 30 DAY)
    OR last_activity_date IS NULL
ORDER BY last_activity_date ASC
LIMIT 50;

-- 8. Overdue tasks
SELECT t.*, u.first_name, u.last_name
FROM tasks t
LEFT JOIN users u ON u.id = t.assignee_id
WHERE t.status IN ('open', 'in_progress')
    AND t.due_date < NOW()
    AND t.deleted_at IS NULL
ORDER BY t.due_date ASC;
```

Run `EXPLAIN` on each of these queries against your schema to verify index usage. If any show `type: ALL` (full table scan) or missing indexes, revisit `indexing-and-performance.md`.


path: references/core-entities.md
content: # Core CRM Entities

This reference covers the foundational tables that every CRM system needs. These entities form the backbone of customer relationship management: tracking who your customers are, how you acquire them, and what business you do with them.

## Entity Overview

A well-designed CRM revolves around these core entities and their relationships:

- **Accounts** (companies/organizations) — the central entity everything connects to
- **Contacts** (people) — individuals associated with accounts
- **Leads** (unqualified prospects) — potential customers not yet converted
- **Opportunities** (deals) — revenue-bearing objects tied to accounts
- **Products** — what you sell, referenced by opportunities
- **Pipeline Stages** — the stages a deal moves through
- **Users** (CRM users/sales reps) — the people who use the CRM system

## Table Designs

### Users (CRM System Users)

The users table represents sales reps, managers, admins — anyone who logs into the CRM. This is not the same as contacts or customers.

```sql
CREATE TABLE `users` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `email` VARCHAR(255) NOT NULL,
    `password_hash` VARCHAR(255) NOT NULL,
    `first_name` VARCHAR(100) NOT NULL,
    `last_name` VARCHAR(100) NOT NULL,
    `phone` VARCHAR(30) NULL DEFAULT NULL,
    `avatar_url` VARCHAR(500) NULL DEFAULT NULL,
    `role` ENUM('admin', 'manager', 'sales_rep', 'support_agent', 'read_only') NOT NULL DEFAULT 'sales_rep',
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `timezone` VARCHAR(50) NOT NULL DEFAULT 'UTC',
    `last_login_at` DATETIME NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_users_email` (`email`),
    INDEX `idx_users_role` (`role`),
    INDEX `idx_users_is_active` (`is_active`),
    INDEX `idx_users_deleted_at` (`deleted_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- `email` is unique and serves as the login identifier. Never use email as a primary key — use a surrogate BIGINT.
- `password_hash` stores bcrypt/argon2 output, never plaintext.
- `role` uses ENUM for a fixed set of roles. If you need more flexible RBAC, see `security-and-multitenancy.md`.
- `is_active` is separate from `deleted_at`. A user can be deactivated but still exist for historical reference.

### Accounts (Companies / Organizations)

Accounts are the primary entity in B2B CRM. In B2C, this may be optional or represent household groupings.

```sql
CREATE TABLE `accounts` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(255) NOT NULL,
    `domain` VARCHAR(255) NULL DEFAULT NULL,
    `industry` VARCHAR(100) NULL DEFAULT NULL,
    `company_size` ENUM('1-10', '11-50', '51-200', '201-500', '501-1000', '1001-5000', '5000+') NULL DEFAULT NULL,
    `annual_revenue` DECIMAL(15, 2) NULL DEFAULT NULL,
    `type` ENUM('prospect', 'customer', 'partner', 'vendor', 'competitor', 'other') NOT NULL DEFAULT 'prospect',
    `status` ENUM('active', 'inactive', 'churned') NOT NULL DEFAULT 'active',
    `phone` VARCHAR(30) NULL DEFAULT NULL,
    `website` VARCHAR(500) NULL DEFAULT NULL,
    `description` TEXT NULL DEFAULT NULL,

    -- Address fields (consider a separate addresses table for multi-address support)
    `billing_address_line1` VARCHAR(255) NULL DEFAULT NULL,
    `billing_address_line2` VARCHAR(255) NULL DEFAULT NULL,
    `billing_city` VARCHAR(100) NULL DEFAULT NULL,
    `billing_state` VARCHAR(100) NULL DEFAULT NULL,
    `billing_postal_code` VARCHAR(20) NULL DEFAULT NULL,
    `billing_country` VARCHAR(2) NULL DEFAULT NULL COMMENT 'ISO 3166-1 alpha-2',

    -- Ownership and hierarchy
    `owner_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'Sales rep who owns this account',
    `parent_account_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'For subsidiaries/divisions',

    -- Integration
    `external_id` VARCHAR(255) NULL DEFAULT NULL COMMENT 'ID in external system (ERP, billing, etc.)',
    `external_source` VARCHAR(100) NULL DEFAULT NULL COMMENT 'Name of external system',

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,
    `updated_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_accounts_name` (`name`),
    INDEX `idx_accounts_domain` (`domain`),
    INDEX `idx_accounts_type` (`type`),
    INDEX `idx_accounts_status` (`status`),
    INDEX `idx_accounts_owner_id` (`owner_id`),
    INDEX `idx_accounts_parent_account_id` (`parent_account_id`),
    INDEX `idx_accounts_industry` (`industry`),
    INDEX `idx_accounts_deleted_at` (`deleted_at`),
    INDEX `idx_accounts_external` (`external_source`, `external_id`),

    CONSTRAINT `fk_accounts_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_accounts_parent` FOREIGN KEY (`parent_account_id`) REFERENCES `accounts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_accounts_created_by` FOREIGN KEY (`created_by`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_accounts_updated_by` FOREIGN KEY (`updated_by`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- `parent_account_id` enables account hierarchies (parent company → subsidiaries). Query these with MySQL 8 recursive CTEs.
- `billing_country` uses ISO 3166-1 alpha-2 codes (US, GB, DE). Consider a lookup table for countries if you need names and metadata.
- `owner_id` establishes the primary sales rep responsible. Many CRM systems also use a team/territory assignment table.
- `external_id` + `external_source` together form a composite key for integration deduplication.

### Contacts (People)

Contacts are individual people, typically associated with an account.

```sql
CREATE TABLE `contacts` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `account_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `first_name` VARCHAR(100) NOT NULL,
    `last_name` VARCHAR(100) NOT NULL,
    `email` VARCHAR(255) NULL DEFAULT NULL,
    `secondary_email` VARCHAR(255) NULL DEFAULT NULL,
    `phone` VARCHAR(30) NULL DEFAULT NULL,
    `mobile_phone` VARCHAR(30) NULL DEFAULT NULL,
    `job_title` VARCHAR(150) NULL DEFAULT NULL,
    `department` VARCHAR(100) NULL DEFAULT NULL,
    `linkedin_url` VARCHAR(500) NULL DEFAULT NULL,
    `is_primary` TINYINT(1) NOT NULL DEFAULT 0 COMMENT 'Primary contact for the account',
    `lifecycle_stage` ENUM('subscriber', 'lead', 'mql', 'sql', 'opportunity', 'customer', 'evangelist', 'other') NOT NULL DEFAULT 'subscriber',
    `lead_source` VARCHAR(100) NULL DEFAULT NULL,
    `status` ENUM('active', 'inactive', 'bounced', 'unsubscribed', 'do_not_contact') NOT NULL DEFAULT 'active',

    -- Communication preferences
    `email_opt_in` TINYINT(1) NOT NULL DEFAULT 0,
    `sms_opt_in` TINYINT(1) NOT NULL DEFAULT 0,
    `preferred_language` VARCHAR(10) NULL DEFAULT NULL COMMENT 'IETF language tag, e.g. en-US',

    -- Mailing address
    `mailing_address_line1` VARCHAR(255) NULL DEFAULT NULL,
    `mailing_address_line2` VARCHAR(255) NULL DEFAULT NULL,
    `mailing_city` VARCHAR(100) NULL DEFAULT NULL,
    `mailing_state` VARCHAR(100) NULL DEFAULT NULL,
    `mailing_postal_code` VARCHAR(20) NULL DEFAULT NULL,
    `mailing_country` VARCHAR(2) NULL DEFAULT NULL,

    -- Ownership
    `owner_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- Integration
    `external_id` VARCHAR(255) NULL DEFAULT NULL,
    `external_source` VARCHAR(100) NULL DEFAULT NULL,

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,
    `updated_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_contacts_account_id` (`account_id`),
    INDEX `idx_contacts_email` (`email`),
    INDEX `idx_contacts_name` (`last_name`, `first_name`),
    INDEX `idx_contacts_owner_id` (`owner_id`),
    INDEX `idx_contacts_lifecycle_stage` (`lifecycle_stage`),
    INDEX `idx_contacts_status` (`status`),
    INDEX `idx_contacts_lead_source` (`lead_source`),
    INDEX `idx_contacts_deleted_at` (`deleted_at`),
    INDEX `idx_contacts_external` (`external_source`, `external_id`),

    CONSTRAINT `fk_contacts_account` FOREIGN KEY (`account_id`) REFERENCES `accounts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_contacts_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_contacts_created_by` FOREIGN KEY (`created_by`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_contacts_updated_by` FOREIGN KEY (`updated_by`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- `account_id` is nullable because some contacts (individuals, freelancers) may not belong to an account.
- `email` is not UNIQUE because a person might appear in the system via different sources. Deduplication is an application-level concern. If you want uniqueness, add a unique index.
- `lifecycle_stage` tracks where this contact is in the marketing/sales funnel.
- `is_primary` marks the main contact at an account. Enforce only-one-primary per account in application logic or with a filtered unique index.

### Leads (Unqualified Prospects)

Leads are potential contacts that have not yet been qualified. Some CRM systems merge leads into the contacts table with a status flag; others keep them separate. A separate table is cleaner when lead qualification is a distinct workflow.

```sql
CREATE TABLE `leads` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `first_name` VARCHAR(100) NOT NULL,
    `last_name` VARCHAR(100) NOT NULL,
    `email` VARCHAR(255) NULL DEFAULT NULL,
    `phone` VARCHAR(30) NULL DEFAULT NULL,
    `company_name` VARCHAR(255) NULL DEFAULT NULL,
    `job_title` VARCHAR(150) NULL DEFAULT NULL,
    `website` VARCHAR(500) NULL DEFAULT NULL,

    -- Lead qualification
    `source` VARCHAR(100) NULL DEFAULT NULL COMMENT 'e.g., website, trade_show, referral, cold_call, linkedin, advertisement',
    `source_detail` VARCHAR(255) NULL DEFAULT NULL COMMENT 'Specific campaign, event name, or referrer',
    `status` ENUM('new', 'contacted', 'qualified', 'unqualified', 'converted', 'lost') NOT NULL DEFAULT 'new',
    `rating` ENUM('hot', 'warm', 'cold') NULL DEFAULT NULL,
    `score` INT UNSIGNED NULL DEFAULT NULL COMMENT 'Numeric lead score from 0-100',

    -- Conversion tracking
    `converted_at` DATETIME NULL DEFAULT NULL,
    `converted_contact_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `converted_account_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `converted_opportunity_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- Ownership
    `owner_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- Integration
    `external_id` VARCHAR(255) NULL DEFAULT NULL,
    `external_source` VARCHAR(100) NULL DEFAULT NULL,

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,
    `updated_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_leads_email` (`email`),
    INDEX `idx_leads_status` (`status`),
    INDEX `idx_leads_source` (`source`),
    INDEX `idx_leads_owner_id` (`owner_id`),
    INDEX `idx_leads_rating` (`rating`),
    INDEX `idx_leads_score` (`score`),
    INDEX `idx_leads_converted_at` (`converted_at`),
    INDEX `idx_leads_deleted_at` (`deleted_at`),
    INDEX `idx_leads_created_at` (`created_at`),
    INDEX `idx_leads_external` (`external_source`, `external_id`),

    CONSTRAINT `fk_leads_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_leads_converted_contact` FOREIGN KEY (`converted_contact_id`) REFERENCES `contacts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_leads_converted_account` FOREIGN KEY (`converted_account_id`) REFERENCES `accounts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_leads_created_by` FOREIGN KEY (`created_by`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_leads_updated_by` FOREIGN KEY (`updated_by`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- When a lead is converted, set `status = 'converted'`, record `converted_at`, and link to the newly created contact, account, and/or opportunity.
- `score` supports numeric lead scoring (0-100). The scoring rules live in application logic; the database just stores the result.
- `source` + `source_detail` together tell you where the lead came from and the specific campaign or event.

### Pipelines and Pipeline Stages

Most CRMs support multiple sales pipelines (e.g., "New Business", "Renewals", "Upsell"), each with their own stages.

```sql
CREATE TABLE `pipelines` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL,
    `description` TEXT NULL DEFAULT NULL,
    `is_default` TINYINT(1) NOT NULL DEFAULT 0,
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_pipelines_is_active` (`is_active`),
    INDEX `idx_pipelines_deleted_at` (`deleted_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `pipeline_stages` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `pipeline_id` BIGINT UNSIGNED NOT NULL,
    `name` VARCHAR(100) NOT NULL,
    `display_order` INT UNSIGNED NOT NULL DEFAULT 0,
    `probability` DECIMAL(5, 2) NULL DEFAULT NULL COMMENT 'Win probability percentage, e.g. 75.00',
    `is_won` TINYINT(1) NOT NULL DEFAULT 0 COMMENT 'True for closed-won stage',
    `is_lost` TINYINT(1) NOT NULL DEFAULT 0 COMMENT 'True for closed-lost stage',
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    INDEX `idx_pipeline_stages_pipeline_id` (`pipeline_id`),
    INDEX `idx_pipeline_stages_display_order` (`pipeline_id`, `display_order`),

    CONSTRAINT `fk_pipeline_stages_pipeline` FOREIGN KEY (`pipeline_id`) REFERENCES `pipelines` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- `display_order` controls the visual left-to-right order of stages in a kanban view.
- `probability` is used to calculate weighted pipeline value (opportunity amount × stage probability).
- `is_won` and `is_lost` are mutually exclusive flags that mark terminal stages. CHECK constraints can enforce this.

### Opportunities (Deals)

Opportunities represent potential revenue. They are the lifeblood of CRM reporting.

```sql
CREATE TABLE `opportunities` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(255) NOT NULL,
    `account_id` BIGINT UNSIGNED NOT NULL,
    `pipeline_id` BIGINT UNSIGNED NOT NULL,
    `stage_id` BIGINT UNSIGNED NOT NULL,
    `amount` DECIMAL(15, 2) NULL DEFAULT NULL COMMENT 'Deal value in base currency',
    `currency` VARCHAR(3) NOT NULL DEFAULT 'USD' COMMENT 'ISO 4217 currency code',
    `probability` DECIMAL(5, 2) NULL DEFAULT NULL COMMENT 'Override of stage probability if set',
    `expected_close_date` DATE NULL DEFAULT NULL,
    `actual_close_date` DATE NULL DEFAULT NULL,
    `type` ENUM('new_business', 'renewal', 'upsell', 'cross_sell', 'other') NULL DEFAULT NULL,
    `priority` ENUM('low', 'medium', 'high', 'critical') NOT NULL DEFAULT 'medium',
    `loss_reason` VARCHAR(255) NULL DEFAULT NULL,
    `description` TEXT NULL DEFAULT NULL,
    `next_step` VARCHAR(500) NULL DEFAULT NULL,

    -- Ownership
    `owner_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- Primary contact on the deal
    `primary_contact_id` BIGINT UNSIGNED NULL DEFAULT NULL,

    -- Integration
    `external_id` VARCHAR(255) NULL DEFAULT NULL,
    `external_source` VARCHAR(100) NULL DEFAULT NULL,

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,
    `created_by` BIGINT UNSIGNED NULL DEFAULT NULL,
    `updated_by` BIGINT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_opportunities_account_id` (`account_id`),
    INDEX `idx_opportunities_pipeline_id` (`pipeline_id`),
    INDEX `idx_opportunities_stage_id` (`stage_id`),
    INDEX `idx_opportunities_owner_id` (`owner_id`),
    INDEX `idx_opportunities_expected_close` (`expected_close_date`),
    INDEX `idx_opportunities_amount` (`amount`),
    INDEX `idx_opportunities_type` (`type`),
    INDEX `idx_opportunities_deleted_at` (`deleted_at`),
    INDEX `idx_opportunities_created_at` (`created_at`),
    INDEX `idx_opportunities_external` (`external_source`, `external_id`),
    -- Composite index for pipeline reporting
    INDEX `idx_opportunities_pipeline_stage` (`pipeline_id`, `stage_id`, `deleted_at`),

    CONSTRAINT `fk_opportunities_account` FOREIGN KEY (`account_id`) REFERENCES `accounts` (`id`) ON DELETE RESTRICT,
    CONSTRAINT `fk_opportunities_pipeline` FOREIGN KEY (`pipeline_id`) REFERENCES `pipelines` (`id`) ON DELETE RESTRICT,
    CONSTRAINT `fk_opportunities_stage` FOREIGN KEY (`stage_id`) REFERENCES `pipeline_stages` (`id`) ON DELETE RESTRICT,
    CONSTRAINT `fk_opportunities_owner` FOREIGN KEY (`owner_id`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_opportunities_primary_contact` FOREIGN KEY (`primary_contact_id`) REFERENCES `contacts` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_opportunities_created_by` FOREIGN KEY (`created_by`) REFERENCES `users` (`id`) ON DELETE SET NULL,
    CONSTRAINT `fk_opportunities_updated_by` FOREIGN KEY (`updated_by`) REFERENCES `users` (`id`) ON DELETE SET NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- `account_id` is NOT NULL and uses ON DELETE RESTRICT. Opportunities should never exist without an account, and deleting an account with active deals should be blocked.
- `amount` uses DECIMAL(15,2) for precision. Never use FLOAT or DOUBLE for monetary values.
- `currency` stores the ISO 4217 code. Multi-currency CRMs need a currency conversion table or integrate with a rate API.
- The composite index on `(pipeline_id, stage_id, deleted_at)` optimizes the most common CRM query: "show me all active deals by pipeline stage."

### Opportunity Contact Roles (Junction Table)

Deals involve multiple contacts playing different roles (decision maker, influencer, end user, etc.).

```sql
CREATE TABLE `opportunity_contacts` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `opportunity_id` BIGINT UNSIGNED NOT NULL,
    `contact_id` BIGINT UNSIGNED NOT NULL,
    `role` VARCHAR(100) NULL DEFAULT NULL COMMENT 'e.g., decision_maker, influencer, champion, end_user, evaluator',
    `is_primary` TINYINT(1) NOT NULL DEFAULT 0,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_opportunity_contacts` (`opportunity_id`, `contact_id`),
    INDEX `idx_opportunity_contacts_contact_id` (`contact_id`),

    CONSTRAINT `fk_opp_contacts_opportunity` FOREIGN KEY (`opportunity_id`) REFERENCES `opportunities` (`id`) ON DELETE CASCADE,
    CONSTRAINT `fk_opp_contacts_contact` FOREIGN KEY (`contact_id`) REFERENCES `contacts` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

### Products and Opportunity Line Items

```sql
CREATE TABLE `products` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(255) NOT NULL,
    `sku` VARCHAR(100) NULL DEFAULT NULL,
    `description` TEXT NULL DEFAULT NULL,
    `unit_price` DECIMAL(15, 2) NOT NULL DEFAULT 0.00,
    `currency` VARCHAR(3) NOT NULL DEFAULT 'USD',
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `product_category` VARCHAR(100) NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_products_sku` (`sku`),
    INDEX `idx_products_is_active` (`is_active`),
    INDEX `idx_products_category` (`product_category`),
    INDEX `idx_products_deleted_at` (`deleted_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `opportunity_line_items` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `opportunity_id` BIGINT UNSIGNED NOT NULL,
    `product_id` BIGINT UNSIGNED NOT NULL,
    `quantity` DECIMAL(10, 2) NOT NULL DEFAULT 1.00,
    `unit_price` DECIMAL(15, 2) NOT NULL COMMENT 'Price at time of quote, may differ from product list price',
    `discount_percent` DECIMAL(5, 2) NOT NULL DEFAULT 0.00,
    `total_price` DECIMAL(15, 2) GENERATED ALWAYS AS (
        `quantity` * `unit_price` * (1 - `discount_percent` / 100)
    ) STORED,
    `description` VARCHAR(500) NULL DEFAULT NULL,
    `sort_order` INT UNSIGNED NOT NULL DEFAULT 0,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    INDEX `idx_opp_line_items_opportunity_id` (`opportunity_id`),
    INDEX `idx_opp_line_items_product_id` (`product_id`),

    CONSTRAINT `fk_opp_line_items_opportunity` FOREIGN KEY (`opportunity_id`) REFERENCES `opportunities` (`id`) ON DELETE CASCADE,
    CONSTRAINT `fk_opp_line_items_product` FOREIGN KEY (`product_id`) REFERENCES `products` (`id`) ON DELETE RESTRICT
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- `unit_price` on line items is separate from the product's `unit_price` because the quoted price may differ from the list price (discounts, negotiated rates, pricing at time of quote).
- `total_price` is a STORED generated column — MySQL computes it automatically and it can be indexed if needed.
- `discount_percent` stores the percentage discount. Some systems prefer storing `discount_amount` instead. Choose one convention and be consistent.

### Tags (Flexible Categorization)

Tags provide lightweight, user-defined categorization across multiple entity types.

```sql
CREATE TABLE `tags` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `name` VARCHAR(100) NOT NULL,
    `color` VARCHAR(7) NULL DEFAULT NULL COMMENT 'Hex color code, e.g. #FF5733',
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_tags_name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

CREATE TABLE `taggables` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `tag_id` BIGINT UNSIGNED NOT NULL,
    `taggable_type` VARCHAR(50) NOT NULL COMMENT 'e.g., account, contact, lead, opportunity',
    `taggable_id` BIGINT UNSIGNED NOT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_taggables` (`tag_id`, `taggable_type`, `taggable_id`),
    INDEX `idx_taggables_target` (`taggable_type`, `taggable_id`),

    CONSTRAINT `fk_taggables_tag` FOREIGN KEY (`tag_id`) REFERENCES `tags` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Design notes:
- The `taggables` table uses a polymorphic pattern — `taggable_type` + `taggable_id` point to any entity. This avoids creating a separate junction table for every entity type.
- The tradeoff is that MySQL cannot enforce a foreign key on the polymorphic reference. Application logic must validate that `taggable_id` exists in the referenced table.
- The composite unique index prevents duplicate tag assignments on the same entity.


path: references/relationships-and-normalization.md
content: # Relationships and Normalization

This reference covers how to properly establish relationships between CRM entities and ensure the schema follows sound normalization principles.

## Relationship Types in CRM

### One-to-Many (1:N) — The Most Common Pattern

Most CRM relationships are one-to-many. The "many" side holds a foreign key to the "one" side.

| Relationship | Implementation |
|---|---|
| Account → Contacts | `contacts.account_id` → `accounts.id` |
| Account → Opportunities | `opportunities.account_id` → `accounts.id` |
| User → Owned Accounts | `accounts.owner_id` → `users.id` |
| Pipeline → Stages | `pipeline_stages.pipeline_id` → `pipelines.id` |
| Opportunity → Line Items | `opportunity_line_items.opportunity_id` → `opportunities.id` |

Rules for 1:N foreign keys:
- Always create an index on the foreign key column. MySQL does NOT automatically index foreign keys (unlike some other databases).
- Choose the correct ON DELETE behavior:
  - `CASCADE` — child rows deleted when parent is deleted (use for true child entities like line items, stage history)
  - `SET NULL` — FK set to NULL when parent is deleted (use for optional references like owner_id)
  - `RESTRICT` — block parent deletion if children exist (use for critical dependencies like account on an opportunity)

### Many-to-Many (M:N) — Junction Tables

When two entities relate to each other in both directions, use a junction table. Never store comma-separated IDs in a single column.

Common CRM M:N relationships:

```sql
-- Opportunities can involve multiple contacts; contacts can appear in multiple deals
CREATE TABLE `opportunity_contacts` (
    `opportunity_id` BIGINT UNSIGNED NOT NULL,
    `contact_id` BIGINT UNSIGNED NOT NULL,
    `role` VARCHAR(100) NULL DEFAULT NULL,
    PRIMARY KEY (`opportunity_id`, `contact_id`)
);

-- Contacts can belong to multiple accounts (consultants, board members)
CREATE TABLE `account_contacts` (
    `account_id` BIGINT UNSIGNED NOT NULL,
    `contact_id` BIGINT UNSIGNED NOT NULL,
    `relationship_type` VARCHAR(50) NULL DEFAULT NULL,
    PRIMARY KEY (`account_id`, `contact_id`)
);

-- Campaigns target multiple contacts; contacts receive multiple campaigns
CREATE TABLE `campaign_contacts` (
    `campaign_id` BIGINT UNSIGNED NOT NULL,
    `contact_id` BIGINT UNSIGNED NOT NULL,
    `status` ENUM('sent', 'opened', 'clicked', 'responded', 'bounced', 'unsubscribed') NOT NULL DEFAULT 'sent',
    `sent_at` DATETIME NULL DEFAULT NULL,
    PRIMARY KEY (`campaign_id`, `contact_id`)
);
```

Junction table best practices:
- Use a composite primary key on the two FK columns for simple associations.
- Add a surrogate `id` column if you need to reference the junction row from other tables (e.g., activity tracking on a campaign-contact pair).
- Store relationship metadata on the junction table (role, status, timestamps) — not on either parent.
- Always add a reverse index. If the PK is `(opportunity_id, contact_id)`, add an index on `(contact_id)` so lookups from the contact side are fast too.

### Self-Referencing Relationships

CRM uses self-references for hierarchies:

```sql
-- Account hierarchy (parent company → subsidiaries)
ALTER TABLE `accounts`
    ADD COLUMN `parent_account_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    ADD CONSTRAINT `fk_accounts_parent`
        FOREIGN KEY (`parent_account_id`) REFERENCES `accounts` (`id`) ON DELETE SET NULL;

-- User reporting structure (manager → direct reports)
ALTER TABLE `users`
    ADD COLUMN `manager_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    ADD CONSTRAINT `fk_users_manager`
        FOREIGN KEY (`manager_id`) REFERENCES `users` (`id`) ON DELETE SET NULL;
```

Querying hierarchies with MySQL 8 recursive CTEs:

```sql
-- Get full account hierarchy from a given parent
WITH RECURSIVE account_tree AS (
    SELECT id, name, parent_account_id, 0 AS depth
    FROM accounts
    WHERE id = :root_account_id

    UNION ALL

    SELECT a.id, a.name, a.parent_account_id, t.depth + 1
    FROM accounts a
    INNER JOIN account_tree t ON a.parent_account_id = t.id
    WHERE t.depth < 10  -- safety limit to prevent infinite loops
)
SELECT * FROM account_tree;
```

### Polymorphic Relationships

Some CRM data relates to multiple entity types. Tags, notes, activities, and attachments commonly use this pattern.

```
taggable_type = 'contact'  + taggable_id = 42  → contacts row with id=42
taggable_type = 'account'  + taggable_id = 7   → accounts row with id=7
taggable_type = 'opportunity' + taggable_id = 15 → opportunities row with id=15
```

Tradeoffs:
- **Pro:** One table handles all entity types. No proliferation of junction tables.
- **Con:** MySQL cannot enforce foreign keys on polymorphic references. Referential integrity relies on application code.
- **Mitigation:** Use a CHECK constraint to restrict `taggable_type` to known values. Add application-level validation. Consider periodic integrity checks via scheduled queries.

## Normalization Guide for CRM

### First Normal Form (1NF)

Every column stores a single atomic value. No comma-separated lists, no JSON arrays in place of proper relationships.

**Violation:** Storing multiple phone numbers as `"555-1234, 555-5678"` in a single column.
**Fix:** Use a separate `contact_phones` table or dedicated columns (`phone`, `mobile_phone`, `work_phone`).

**Violation:** Storing tag names as `"enterprise, hot-lead, east-coast"` in a tags column.
**Fix:** Use a `tags` table and `taggables` junction table.

### Second Normal Form (2NF)

Every non-key column depends on the entire primary key (relevant for composite keys).

**Violation:** In an `opportunity_contacts` junction table with PK (`opportunity_id`, `contact_id`), storing `contact_email` — this depends only on `contact_id`, not the full key.
**Fix:** Remove `contact_email`; look it up via JOIN to `contacts`.

### Third Normal Form (3NF)

No transitive dependencies. Every non-key column depends directly on the primary key, not on another non-key column.

**Violation:** Storing both `stage_id` and `stage_name` on the `opportunities` table. `stage_name` depends on `stage_id`, not on `opportunity.id`.
**Fix:** Store only `stage_id` on `opportunities`. Look up `stage_name` via JOIN to `pipeline_stages`.

**Acceptable denormalization:** Storing `total_price` as a generated column on `opportunity_line_items`. This is technically a 3NF violation (it depends on `quantity`, `unit_price`, `discount_percent`), but as a generated column, MySQL keeps it consistent automatically.

### When to Deliberately Denormalize

Denormalize only when you have measured evidence of a performance problem, and document the decision:

1. **Cached aggregates.** Store `deal_count` and `total_revenue` on `accounts` if you frequently display account summaries. Update via triggers or application events.

2. **Snapshot values.** Store `unit_price` on `opportunity_line_items` separately from `products.unit_price` because the price at the time of the quote matters, not the current list price.

3. **Reporting tables.** Create materialized summary tables updated by scheduled jobs for dashboard queries. Keep these separate from operational tables.

## Foreign Key Strategy

### Rules of Thumb

1. Every foreign key column gets its own index.
2. Use `ON DELETE RESTRICT` as the default. Only use CASCADE for true child/dependent entities.
3. Use `ON DELETE SET NULL` for optional relationships (owner_id, assigned_to).
4. Never use `ON DELETE CASCADE` on core business entities (accounts, contacts, opportunities). Soft delete instead.
5. Use `ON UPDATE CASCADE` sparingly — if you're updating primary keys, reconsider your schema design.

### FK Index Pattern

```sql
-- For every FK column, create a dedicated index
ALTER TABLE `contacts` ADD INDEX `idx_contacts_account_id` (`account_id`);
ALTER TABLE `contacts` ADD INDEX `idx_contacts_owner_id` (`owner_id`);
ALTER TABLE `opportunities` ADD INDEX `idx_opportunities_account_id` (`account_id`);
ALTER TABLE `opportunities` ADD INDEX `idx_opportunities_stage_id` (`stage_id`);
```

Without these indexes, DELETE operations on parent tables will cause full table scans on child tables to check for references — this can lock the table and cause severe performance issues.


path: references/migrations-and-seeding.md
content: # Migrations and Seeding

This reference covers schema versioning, writing safe migration scripts, and generating realistic CRM test data.

## Migration Best Practices

### File Naming Convention

Use sequential, timestamped filenames:

```
migrations/
├── 001_20260101_000000_create_users_table.sql
├── 002_20260101_000001_create_accounts_table.sql
├── 003_20260101_000002_create_contacts_table.sql
├── 004_20260101_000003_create_leads_table.sql
├── 005_20260101_000004_create_pipelines_and_stages.sql
├── 006_20260101_000005_create_opportunities_table.sql
├── 007_20260101_000006_create_products_and_line_items.sql
├── 008_20260101_000007_create_activities_table.sql
├── 009_20260101_000008_create_notes_and_tasks.sql
├── 010_20260101_000009_create_audit_logs_table.sql
├── 011_20260101_000010_create_tags_table.sql
├── 012_20260101_000011_create_custom_fields.sql
├── 013_20260101_000012_create_attachments_table.sql
└── 014_20260101_000013_add_fulltext_indexes.sql
```

### Migration Tracking Table

Track which migrations have been applied:

```sql
CREATE TABLE `schema_migrations` (
    `id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
    `migration` VARCHAR(255) NOT NULL,
    `applied_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `checksum` VARCHAR(64) NULL DEFAULT NULL COMMENT 'SHA-256 of migration content',
    `execution_time_ms` INT UNSIGNED NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_migrations_name` (`migration`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

### Writing Safe Migrations

**Always use IF NOT EXISTS / IF EXISTS:**

```sql
-- Safe table creation
CREATE TABLE IF NOT EXISTS `contacts` (
    -- ...
);

-- Safe column addition
-- MySQL does not support IF NOT EXISTS for columns, so check first:
SET @col_exists = (
    SELECT COUNT(*) FROM information_schema.COLUMNS
    WHERE TABLE_SCHEMA = DATABASE()
        AND TABLE_NAME = 'contacts'
        AND COLUMN_NAME = 'linkedin_url'
);
SET @sql = IF(@col_exists = 0,
    'ALTER TABLE `contacts` ADD COLUMN `linkedin_url` VARCHAR(500) NULL DEFAULT NULL',
    'SELECT 1');
PREPARE stmt FROM @sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
```

**Include both up and down migrations:**

```sql
-- ============================================================
-- Migration: 015_20260415_add_priority_to_opportunities
-- ============================================================

-- UP
ALTER TABLE `opportunities`
    ADD COLUMN `priority` ENUM('low', 'medium', 'high', 'critical')
    NOT NULL DEFAULT 'medium' AFTER `type`;

ALTER TABLE `opportunities`
    ADD INDEX `idx_opportunities_priority` (`priority`);

-- DOWN (rollback)
-- ALTER TABLE `opportunities` DROP INDEX `idx_opportunities_priority`;
-- ALTER TABLE `opportunities` DROP COLUMN `priority`;
```

**Handle large tables carefully:**

For tables with millions of rows, direct ALTER TABLE can lock the table for a long time. Strategies:

1. Use `ALGORITHM=INPLACE, LOCK=NONE` when MySQL supports it:
```sql
ALTER TABLE `activities`
    ADD COLUMN `sentiment` ENUM('positive', 'neutral', 'negative') NULL DEFAULT NULL,
    ALGORITHM=INPLACE, LOCK=NONE;
```

2. For changes that require table rebuild, use pt-online-schema-change or gh-ost:
```bash
# Using Percona pt-online-schema-change
pt-online-schema-change --alter "ADD COLUMN sentiment ENUM('positive','neutral','negative') NULL DEFAULT NULL" \
    D=crm,t=activities --execute
```

3. Create a new table, copy data, and swap (manual online DDL):
```sql
CREATE TABLE `activities_new` LIKE `activities`;
ALTER TABLE `activities_new` ADD COLUMN `sentiment` ENUM('positive','neutral','negative') NULL DEFAULT NULL;
-- Copy data in batches...
RENAME TABLE `activities` TO `activities_old`, `activities_new` TO `activities`;
```

### Execution Order

Migrations must execute in dependency order. The correct order for a CRM schema:

1. `users` — no dependencies
2. `tenants` — no dependencies (if multi-tenant)
3. `teams`, `team_members` — depends on `users`
4. `roles`, `permissions`, `role_permissions`, `user_roles` — depends on `users`
5. `accounts` — depends on `users` (for owner_id)
6. `contacts` — depends on `accounts`, `users`
7. `leads` — depends on `users`, `contacts`, `accounts`
8. `pipelines`, `pipeline_stages` — depends on nothing or `tenants`
9. `products` — no dependencies
10. `opportunities` — depends on `accounts`, `pipelines`, `pipeline_stages`, `users`, `contacts`
11. `opportunity_line_items` — depends on `opportunities`, `products`
12. `opportunity_contacts` — depends on `opportunities`, `contacts`
13. `activities` — depends on `users`, `accounts`, `contacts`, `opportunities`
14. `notes`, `tasks` — depends on `users`
15. `tags`, `taggables` — depends on nothing (polymorphic)
16. `attachments` — depends on `users`
17. `audit_logs` — no FK dependencies (polymorphic)
18. `custom_field_definitions`, `custom_field_values` — no FK dependencies (polymorphic)

## Seed Data

### Generating Realistic CRM Test Data

Seed data should be realistic enough to test query patterns, UI layouts, and reporting.

```sql
-- ============================================================
-- Seed: Users (sales team)
-- ============================================================
INSERT INTO `users` (`email`, `password_hash`, `first_name`, `last_name`, `role`, `timezone`) VALUES
('[REDACTED_SECRET]', '$2b$12$placeholder_hash', 'Admin', 'User', 'admin', 'America/Chicago'),
('[REDACTED_SECRET]', '$2b$12$placeholder_hash', 'Sarah', 'Chen', 'manager', 'America/New_York'),
('[REDACTED_SECRET]', '$2b$12$placeholder_hash', 'James', 'Wilson', 'sales_rep', 'America/Chicago'),
('[REDACTED_SECRET]', '$2b$12$placeholder_hash', 'Maria', 'Garcia', 'sales_rep', 'America/Los_Angeles'),
('[REDACTED_SECRET]', '$2b$12$placeholder_hash', 'Alex', 'Johnson', 'sales_rep', 'America/New_York'),
('[REDACTED_SECRET]', '$2b$12$placeholder_hash', 'Priya', 'Patel', 'support_agent', 'America/Chicago');

-- ============================================================
-- Seed: Accounts
-- ============================================================
INSERT INTO `accounts` (`name`, `domain`, `industry`, `company_size`, `type`, `status`, `owner_id`, `billing_country`) VALUES
('Acme Corporation', 'acme.com', 'Manufacturing', '201-500', 'customer', 'active', 3, 'US'),
('TechStart Inc', 'techstart.io', 'Technology', '11-50', 'prospect', 'active', 4, 'US'),
('Global Logistics Ltd', 'globallogistics.co.uk', 'Transportation', '501-1000', 'customer', 'active', 3, 'GB'),
('Sunrise Healthcare', 'sunrisehealth.org', 'Healthcare', '1001-5000', 'prospect', 'active', 5, 'US'),
('Nordic Design Studio', 'nordicdesign.se', 'Design', '1-10', 'partner', 'active', 4, 'SE'),
('Pacific Financial Group', 'pacificfin.com', 'Finance', '201-500', 'customer', 'active', 5, 'US'),
('RedBridge Consulting', 'redbridge.com.au', 'Consulting', '51-200', 'prospect', 'active', 3, 'AU'),
('MegaRetail Corp', 'megaretail.com', 'Retail', '5000+', 'customer', 'active', 2, 'US');

-- ============================================================
-- Seed: Contacts
-- ============================================================
INSERT INTO `contacts` (`account_id`, `first_name`, `last_name`, `email`, `phone`, `job_title`, `lifecycle_stage`, `owner_id`, `is_primary`) VALUES
(1, 'Robert', 'Smith', '[REDACTED_SECRET]', '+1-555-0101', 'VP of Operations', 'customer', 3, 1),
(1, 'Lisa', 'Wang', '[REDACTED_SECRET]', '+1-555-0102', 'Procurement Manager', 'customer', 3, 0),
(2, 'David', 'Kim', '[REDACTED_SECRET]', '+1-555-0201', 'CEO', 'sql', 4, 1),
(2, 'Emily', 'Brown', '[REDACTED_SECRET]', '+1-555-0202', 'CTO', 'sql', 4, 0),
(3, 'Oliver', 'Hughes', '[REDACTED_SECRET]', '+44-20-7123-0001', 'Head of IT', 'customer', 3, 1),
(4, 'Jennifer', 'Martinez', '[REDACTED_SECRET]', '+1-555-0401', 'CFO', 'mql', 5, 1),
(5, 'Erik', 'Lindqvist', '[REDACTED_SECRET]', '+46-8-123-4567', 'Founder', 'customer', 4, 1),
(6, 'Patricia', 'Adams', '[REDACTED_SECRET]', '+1-555-0601', 'Director of Technology', 'customer', 5, 1),
(7, 'Michael', 'Thompson', '[REDACTED_SECRET]', '+61-2-8765-4321', 'Managing Partner', 'lead', 3, 1),
(8, 'Catherine', 'Lee', '[REDACTED_SECRET]', '+1-555-0801', 'SVP of Supply Chain', 'customer', 2, 1);

-- ============================================================
-- Seed: Pipelines and Stages
-- ============================================================
INSERT INTO `pipelines` (`name`, `description`, `is_default`) VALUES
('New Business', 'Pipeline for new customer acquisition', 1),
('Renewals', 'Pipeline for contract renewals', 0),
('Upsell', 'Pipeline for upsell and expansion deals', 0);

INSERT INTO `pipeline_stages` (`pipeline_id`, `name`, `display_order`, `probability`, `is_won`, `is_lost`) VALUES
(1, 'Qualification', 1, 10.00, 0, 0),
(1, 'Discovery', 2, 20.00, 0, 0),
(1, 'Proposal', 3, 50.00, 0, 0),
(1, 'Negotiation', 4, 75.00, 0, 0),
(1, 'Closed Won', 5, 100.00, 1, 0),
(1, 'Closed Lost', 6, 0.00, 0, 1),
(2, 'Renewal Pending', 1, 70.00, 0, 0),
(2, 'In Negotiation', 2, 85.00, 0, 0),
(2, 'Renewed', 3, 100.00, 1, 0),
(2, 'Churned', 4, 0.00, 0, 1),
(3, 'Identified', 1, 20.00, 0, 0),
(3, 'Proposed', 2, 50.00, 0, 0),
(3, 'Committed', 3, 90.00, 0, 0),
(3, 'Won', 4, 100.00, 1, 0),
(3, 'Lost', 5, 0.00, 0, 1);

-- ============================================================
-- Seed: Opportunities
-- ============================================================
INSERT INTO `opportunities` (`name`, `account_id`, `pipeline_id`, `stage_id`, `amount`, `expected_close_date`, `type`, `owner_id`, `primary_contact_id`) VALUES
('Acme - Annual License Renewal', 1, 2, 7, 45000.00, '2026-06-30', 'renewal', 3, 1),
('TechStart - Platform Subscription', 2, 1, 3, 120000.00, '2026-05-15', 'new_business', 4, 3),
('Global Logistics - Expansion', 3, 3, 11, 85000.00, '2026-07-01', 'upsell', 3, 5),
('Sunrise Healthcare - Enterprise License', 4, 1, 2, 250000.00, '2026-08-30', 'new_business', 5, 6),
('Pacific Financial - Add-on Modules', 6, 3, 12, 35000.00, '2026-05-30', 'cross_sell', 5, 8),
('RedBridge - Initial Contract', 7, 1, 1, 60000.00, '2026-09-15', 'new_business', 3, 9);

-- ============================================================
-- Seed: Leads
-- ============================================================
INSERT INTO `leads` (`first_name`, `last_name`, `email`, `company_name`, `job_title`, `source`, `status`, `rating`, `score`, `owner_id`) VALUES
('Andrew', 'Taylor', '[REDACTED_SECRET]', 'BigCorp Industries', 'IT Director', 'website', 'new', 'hot', 85, 3),
('Sophie', 'Dubois', '[REDACTED_SECRET]', 'ParisTech Solutions', 'COO', 'trade_show', 'contacted', 'warm', 62, 4),
('Raj', 'Sharma', '[REDACTED_SECRET]', 'Mumbai Software Labs', 'VP Engineering', 'referral', 'qualified', 'hot', 90, 5),
('Hannah', 'Miller', '[REDACTED_SECRET]', 'GreenLeaf Sustainability', 'Founder', 'linkedin', 'new', 'cold', 30, NULL),
('Carlos', 'Reyes', '[REDACTED_SECRET]', 'LatAm Logix', 'Head of Procurement', 'advertisement', 'contacted', 'warm', 55, 4);

-- ============================================================
-- Seed: Activities
-- ============================================================
INSERT INTO `activities` (`activity_type`, `entity_type`, `entity_id`, `activity_date`, `user_id`, `subject`, `body`, `duration_minutes`, `account_id`, `contact_id`, `metadata`) VALUES
('call', 'contact', 1, '2026-03-28 10:00:00', 3, 'Renewal discussion call', 'Discussed renewal terms. Robert is happy with service but wants to negotiate pricing.', 25, 1, 1, '{"direction":"outbound","outcome":"connected"}'),
('email', 'contact', 3, '2026-03-29 14:30:00', 4, 'Proposal follow-up', 'Sent updated proposal with revised pricing tier.', NULL, 2, 3, '{"direction":"outbound","has_attachments":true}'),
('meeting', 'opportunity', 4, '2026-04-01 09:00:00', 5, 'Discovery meeting with Sunrise Healthcare', 'Deep dive into their requirements. They need HIPAA compliance features.', 60, 4, 6, '{"location":"Zoom","attendees":2,"outcome":"completed"}'),
('note', 'account', 8, '2026-04-02 11:00:00', 2, 'Quarterly business review notes', 'MegaRetail is very satisfied. Potential for 30% expansion this year.', NULL, 8, 10, NULL),
('task', 'lead', 1, '2026-04-03 08:00:00', 3, 'Follow up with Andrew Taylor at BigCorp', NULL, NULL, NULL, NULL, NULL);
```

### Volume Testing

For performance testing, generate large volumes of data. Use a stored procedure or an external script:

```sql
-- Generate 10,000 contacts across existing accounts
DELIMITER //
CREATE PROCEDURE seed_contacts(IN num_contacts INT)
BEGIN
    DECLARE i INT DEFAULT 0;
    DECLARE v_account_id BIGINT;
    DECLARE v_owner_id BIGINT;

    WHILE i < num_contacts DO
        SET v_account_id = FLOOR(1 + RAND() * 8);
        SET v_owner_id = FLOOR(2 + RAND() * 4);

        INSERT INTO contacts (account_id, first_name, last_name, email, lifecycle_stage, owner_id)
        VALUES (
            v_account_id,
            ELT(FLOOR(1 + RAND() * 10), 'James','Mary','John','Patricia','Robert','Jennifer','Michael','Linda','David','Elizabeth'),
            ELT(FLOOR(1 + RAND() * 10), 'Smith','Johnson','Williams','Brown','Jones','Garcia','Miller','Davis','Rodriguez','Martinez'),
            CONCAT('contact_', i, '_', FLOOR(RAND() * 10000), '@example.com'),
            ELT(FLOOR(1 + RAND() * 5), 'subscriber','lead','mql','sql','customer'),
            v_owner_id
        );

        SET i = i + 1;
    END WHILE;
END //
DELIMITER ;

-- Run it
CALL seed_contacts(10000);

-- Clean up
DROP PROCEDURE IF EXISTS seed_contacts;
```

### Seed Data Principles

1. Use realistic names, emails, and company names — not "test1", "test2".
2. Create data that exercises all relationship types (accounts with multiple contacts, contacts with multiple opportunities, etc.).
3. Include data in various statuses (active, inactive, converted, closed-won, closed-lost).
4. Include edge cases: contacts without accounts, leads that have been converted, accounts with hierarchies.
5. Use consistent, documented passwords (like `$2b$12$placeholder_hash`) and note that they are not real hashes.
6. Timestamp data should span multiple months/quarters to test reporting.


path: references/custom-fields-and-flexibility.md
content: # Custom Fields and Flexibility

CRM systems must accommodate user-defined fields. Sales teams want to track industry-specific data, marketing wants custom segmentation attributes, and every client has unique requirements. This reference covers the three main approaches to custom fields in MySQL 8 and when to use each.

## Approach 1: Entity-Attribute-Value (EAV)

The EAV pattern stores custom field definitions and values in separate tables rather than adding columns to the entity table.

### Schema

```sql
-- Define available custom fields
CREATE TABLE `custom_field_definitions` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL COMMENT 'e.g., contact, account, opportunity, lead',
    `field_name` VARCHAR(100) NOT NULL,
    `field_label` VARCHAR(200) NOT NULL,
    `field_type` ENUM('text', 'number', 'decimal', 'boolean', 'date', 'datetime',
                       'select', 'multi_select', 'email', 'url', 'phone') NOT NULL,
    `is_required` TINYINT(1) NOT NULL DEFAULT 0,
    `default_value` VARCHAR(500) NULL DEFAULT NULL,
    `options` JSON NULL DEFAULT NULL COMMENT 'For select/multi_select: ["opt1","opt2","opt3"]',
    `validation_rules` JSON NULL DEFAULT NULL COMMENT '{"min_length":1,"max_length":500,"regex":"..."}',
    `display_order` INT UNSIGNED NOT NULL DEFAULT 0,
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_cfd_entity_field` (`entity_type`, `field_name`),
    INDEX `idx_cfd_entity_type` (`entity_type`, `is_active`, `display_order`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Store custom field values (one row per entity-field pair)
CREATE TABLE `custom_field_values` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `field_definition_id` BIGINT UNSIGNED NOT NULL,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `value_text` TEXT NULL DEFAULT NULL,
    `value_number` DECIMAL(20, 6) NULL DEFAULT NULL,
    `value_boolean` TINYINT(1) NULL DEFAULT NULL,
    `value_date` DATE NULL DEFAULT NULL,
    `value_datetime` DATETIME(3) NULL DEFAULT NULL,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_cfv_entity_field` (`field_definition_id`, `entity_type`, `entity_id`),
    INDEX `idx_cfv_entity` (`entity_type`, `entity_id`),
    INDEX `idx_cfv_value_text` (`field_definition_id`, `value_text`(100)),
    INDEX `idx_cfv_value_number` (`field_definition_id`, `value_number`),
    INDEX `idx_cfv_value_date` (`field_definition_id`, `value_date`),

    CONSTRAINT `fk_cfv_definition` FOREIGN KEY (`field_definition_id`)
        REFERENCES `custom_field_definitions` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

### Querying EAV Data

Pivoting EAV data into columns for display:

```sql
-- Get all custom fields for a specific contact
SELECT
    cfd.field_name,
    cfd.field_label,
    cfd.field_type,
    COALESCE(cfv.value_text, cfv.value_number, cfv.value_boolean,
             cfv.value_date, cfv.value_datetime) AS value
FROM custom_field_definitions cfd
LEFT JOIN custom_field_values cfv
    ON cfv.field_definition_id = cfd.id
    AND cfv.entity_type = 'contact'
    AND cfv.entity_id = :contact_id
WHERE cfd.entity_type = 'contact'
    AND cfd.is_active = 1
ORDER BY cfd.display_order;

-- Filter contacts by a custom field value
SELECT c.*
FROM contacts c
INNER JOIN custom_field_values cfv
    ON cfv.entity_type = 'contact' AND cfv.entity_id = c.id
INNER JOIN custom_field_definitions cfd
    ON cfd.id = cfv.field_definition_id
WHERE cfd.field_name = 'preferred_language'
    AND cfv.value_text = 'Spanish'
    AND c.deleted_at IS NULL;
```

### EAV Pros and Cons

**Pros:**
- Unlimited custom fields without schema changes
- Users can create fields at runtime
- No ALTER TABLE needed (no table locks, no downtime)
- Clean separation between core schema and customizations

**Cons:**
- Complex queries — pivoting and filtering require JOINs
- Harder to enforce data types at the database level
- Reporting becomes more complex
- Cannot use standard SQL constraints (NOT NULL, CHECK, UNIQUE) on values
- Performance degrades with heavy filtering across multiple custom fields

**Best for:** SaaS CRM where tenants define their own fields, systems with 50+ custom fields, fields that change frequently.

## Approach 2: JSON Columns

Store custom fields as a JSON document directly on the entity table.

### Schema

```sql
ALTER TABLE `contacts`
    ADD COLUMN `custom_fields` JSON NULL DEFAULT NULL,
    ADD CONSTRAINT `chk_contacts_custom_fields` CHECK (
        `custom_fields` IS NULL OR JSON_VALID(`custom_fields`)
    );
```

### Example Data

```json
{
    "preferred_language": "Spanish",
    "shirt_size": "L",
    "nps_score": 8,
    "renewal_date": "2026-03-15",
    "interests": ["AI", "cloud", "security"],
    "custom_notes": "Prefers morning calls"
}
```

### Indexing JSON for Performance

```sql
-- Generated column approach (most reliable)
ALTER TABLE `contacts`
    ADD COLUMN `cf_preferred_language` VARCHAR(50)
        GENERATED ALWAYS AS (`custom_fields` ->> '$.preferred_language') STORED,
    ADD INDEX `idx_contacts_cf_language` (`cf_preferred_language`);

-- Functional index approach (MySQL 8.0.13+)
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_cf_nps` ((
        CAST(`custom_fields` ->> '$.nps_score' AS SIGNED)
    ));

-- Multi-valued index for arrays (MySQL 8.0.17+)
ALTER TABLE `contacts`
    ADD INDEX `idx_contacts_cf_interests` ((
        CAST(`custom_fields` -> '$.interests' AS CHAR(100) ARRAY)
    ));
```

### Querying JSON Custom Fields

```sql
-- Simple filter
SELECT * FROM contacts
WHERE custom_fields ->> '$.preferred_language' = 'Spanish'
    AND deleted_at IS NULL;

-- Numeric comparison
SELECT * FROM contacts
WHERE CAST(custom_fields ->> '$.nps_score' AS SIGNED) >= 8
    AND deleted_at IS NULL;

-- Array membership
SELECT * FROM contacts
WHERE 'AI' MEMBER OF (custom_fields -> '$.interests')
    AND deleted_at IS NULL;

-- Check if a field exists
SELECT * FROM contacts
WHERE JSON_CONTAINS_PATH(custom_fields, 'one', '$.renewal_date')
    AND deleted_at IS NULL;

-- Update a single field without replacing the entire document
UPDATE contacts
SET custom_fields = JSON_SET(
    COALESCE(custom_fields, '{}'),
    '$.nps_score', 9,
    '$.last_survey_date', '2026-04-01'
)
WHERE id = :contact_id;
```

### JSON Pros and Cons

**Pros:**
- Simple to implement — just one column
- Read performance is good for fetching a single record's custom data
- MySQL 8 partial update optimization for JSON_SET (no full document rewrite)
- Natural fit for application frameworks with JSON serialization
- Flexible schema — any structure without ALTER TABLE

**Cons:**
- Cannot enforce field-level constraints (required fields, data types, allowed values)
- Indexing requires explicit generated columns or functional indexes
- Complex cross-record filtering on JSON fields is slower than regular columns
- No built-in field definitions or validation at database level
- JSON documents should stay under a few KB — avoid storing large blobs

**Best for:** Small to moderate number of custom fields (under 20), systems where custom fields are rarely filtered or sorted, metadata and preferences storage.

## Approach 3: Hybrid (Recommended for Most CRMs)

Combine fixed columns for well-known fields with JSON for truly custom/flexible data.

### Schema Pattern

```sql
CREATE TABLE `contacts` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,

    -- Fixed, well-known columns (indexed, typed, constrained)
    `first_name` VARCHAR(100) NOT NULL,
    `last_name` VARCHAR(100) NOT NULL,
    `email` VARCHAR(255) NULL DEFAULT NULL,
    `phone` VARCHAR(30) NULL DEFAULT NULL,
    `account_id` BIGINT UNSIGNED NULL DEFAULT NULL,
    `lifecycle_stage` ENUM('subscriber','lead','mql','sql','opportunity','customer','evangelist','other') NOT NULL DEFAULT 'subscriber',

    -- JSON for custom/flexible fields
    `custom_fields` JSON NULL DEFAULT NULL,

    -- Generated columns to index frequently queried custom fields
    `cf_preferred_language` VARCHAR(50)
        GENERATED ALWAYS AS (`custom_fields` ->> '$.preferred_language') STORED,

    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    `deleted_at` TIMESTAMP NULL DEFAULT NULL,

    PRIMARY KEY (`id`),
    INDEX `idx_contacts_cf_language` (`cf_preferred_language`),
    -- ... other indexes
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

### When to Promote a JSON Field to a Column

A custom field should graduate from JSON to a dedicated column when:
- It is queried in WHERE or ORDER BY clauses frequently
- It needs database-level constraints (NOT NULL, UNIQUE, CHECK, FK)
- It is used in JOINs
- It appears in reports or dashboards
- It is present on more than 80% of records
- It has a fixed, well-defined data type

### Decision Matrix

| Factor | Use Fixed Column | Use JSON | Use EAV |
|--------|-----------------|----------|---------|
| Known at design time | Yes | — | — |
| Used in WHERE/JOIN | Yes | Maybe (with gen col) | Possible but slow |
| Needs FK constraint | Yes | No | No |
| User-defined at runtime | — | Yes (small count) | Yes (large count) |
| Per-tenant schema | — | Yes | Yes |
| 50+ custom fields | — | — | Yes |
| Reporting/analytics | Yes | With gen columns | Complex |

### Metadata for JSON Custom Fields

Even with the JSON approach, maintain a field definitions table so the UI knows what fields exist and how to render them:

```sql
CREATE TABLE `custom_field_metadata` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL,
    `json_path` VARCHAR(200) NOT NULL COMMENT 'Path in JSON document, e.g. $.preferred_language',
    `field_label` VARCHAR(200) NOT NULL,
    `field_type` ENUM('text','number','decimal','boolean','date','datetime',
                       'select','multi_select','email','url','phone') NOT NULL,
    `options` JSON NULL DEFAULT NULL,
    `is_required` TINYINT(1) NOT NULL DEFAULT 0,
    `display_order` INT UNSIGNED NOT NULL DEFAULT 0,
    `is_active` TINYINT(1) NOT NULL DEFAULT 1,
    `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

    PRIMARY KEY (`id`),
    UNIQUE INDEX `uq_cfm_entity_path` (`entity_type`, `json_path`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

This gives you the flexibility of JSON storage with the discoverability of EAV field definitions.


path: references/audit-and-soft-deletes.md
content: # Audit Trails and Soft Deletes

CRM data has legal, compliance, and business intelligence value. Deleting records permanently destroys historical context. This reference covers patterns for preserving data integrity, tracking changes, and meeting compliance requirements.

## Soft Deletes

### The Pattern

Instead of `DELETE FROM contacts WHERE id = 42`, set a deletion timestamp:

```sql
UPDATE contacts SET deleted_at = NOW(), updated_by = :user_id WHERE id = 42;
```

Every CRM entity table should include:

```sql
`deleted_at` TIMESTAMP NULL DEFAULT NULL,
```

And an index to support filtering:

```sql
INDEX `idx_tablename_deleted_at` (`deleted_at`)
```

### Query Discipline

Every query that fetches active records must include the soft delete filter:

```sql
-- Always filter for active records
SELECT * FROM contacts WHERE account_id = 42 AND deleted_at IS NULL;

-- For admin/compliance views that need to see deleted records
SELECT * FROM contacts WHERE account_id = 42;

-- To see only deleted records
SELECT * FROM contacts WHERE account_id = 42 AND deleted_at IS NOT NULL;
```

Consider creating views for active records to avoid forgetting the filter:

```sql
CREATE VIEW `active_contacts` AS
SELECT * FROM `contacts` WHERE `deleted_at` IS NULL;

CREATE VIEW `active_accounts` AS
SELECT * FROM `accounts` WHERE `deleted_at` IS NULL;

CREATE VIEW `active_opportunities` AS
SELECT * FROM `opportunities` WHERE `deleted_at` IS NULL;
```

### Handling Unique Constraints with Soft Deletes

Problem: if `email` has a UNIQUE index and a contact is soft-deleted, no new contact can be created with the same email.

Solutions:

**Option A: Partial unique index (not natively supported in MySQL)**
MySQL does not support partial/filtered unique indexes. Use Option B or C.

**Option B: Unique on (email, deleted_at) with a sentinel value**
```sql
-- Use a fixed date for active records, actual deletion time for deleted ones
-- This allows the same email to exist once as active plus multiple times as deleted
ALTER TABLE `contacts`
    ADD UNIQUE INDEX `uq_contacts_email_active` (`email`, `deleted_at`);
```
This works because `deleted_at = NULL` is treated as unique by MySQL (each NULL is distinct). However, it means two active records with the same email are both allowed since `(email, NULL)` and `(email, NULL)` are not considered duplicates. This approach has limitations.

**Option C: Application-level uniqueness check (most practical)**
```sql
-- Add a non-unique index for performance
ALTER TABLE `contacts` ADD INDEX `idx_contacts_email` (`email`);
```
Check uniqueness in application code: `SELECT id FROM contacts WHERE email = :email AND deleted_at IS NULL LIMIT 1`. This is the most common approach in production CRM systems.

**Option D: Generated column approach**
```sql
ALTER TABLE `contacts`
    ADD COLUMN `email_unique_check` VARCHAR(255)
        GENERATED ALWAYS AS (IF(`deleted_at` IS NULL, `email`, NULL)) STORED,
    ADD UNIQUE INDEX `uq_contacts_email_active` (`email_unique_check`);
```
This creates a generated column that is non-NULL only for active records, allowing a unique index that only applies to active rows. Multiple NULL values in a unique index are allowed.

### Restore (Undelete)

```sql
UPDATE contacts SET deleted_at = NULL, updated_by = :user_id WHERE id = 42;
```

### Cascading Soft Deletes

When an account is soft-deleted, should its contacts and opportunities also be soft-deleted? This depends on business rules. Common approaches:
- Soft-delete only the parent, and filter children by joining to the parent's `deleted_at`
- Cascade the soft delete in application code or a trigger
- Leave children active but orphaned, and let a cleanup job handle them

## Audit Trail

### Audit Log Table

A central audit log records every significant change to CRM data.

```sql
CREATE TABLE `audit_logs` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL COMMENT 'Table name: contact, account, opportunity, etc.',
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `action` ENUM('create', 'update', 'delete', 'restore', 'merge', 'convert', 'assign') NOT NULL,
    `user_id` BIGINT UNSIGNED NULL DEFAULT NULL COMMENT 'User who made the change, NULL for system actions',
    `changes` JSON NULL DEFAULT NULL COMMENT 'Field-level diff: {"field": {"old": "x", "new": "y"}}',
    `metadata` JSON NULL DEFAULT NULL COMMENT 'Additional context: IP address, source, request_id',
    `created_at` TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),

    PRIMARY KEY (`id`),
    INDEX `idx_audit_entity` (`entity_type`, `entity_id`, `created_at`),
    INDEX `idx_audit_user` (`user_id`, `created_at`),
    INDEX `idx_audit_action` (`action`, `created_at`),
    INDEX `idx_audit_created_at` (`created_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
```

Use `TIMESTAMP(3)` for millisecond precision — important when multiple changes happen in rapid succession.

### What to Log

Log all changes to core CRM entities. The `changes` JSON should capture:

```json
{
    "first_name": {"old": "Jon", "new": "John"},
    "email": {"old": "[REDACTED_SECRET]", "new": "[REDACTED_SECRET]"},
    "lifecycle_stage": {"old": "lead", "new": "mql"}
}
```

For create actions, log the initial values:
```json
{
    "first_name": {"old": null, "new": "John"},
    "email": {"old": null, "new": "[REDACTED_SECRET]"}
}
```

For delete actions, log the key identifying fields:
```json
{
    "first_name": {"old": "John", "new": null},
    "email": {"old": "[REDACTED_SECRET]", "new": null}
}
```

### Implementation: Application Level vs. Triggers

**Application-level logging (recommended):**
- Captured in the service layer before/after save
- Has access to the current user, request context, and business logic
- Can batch audit entries and write asynchronously
- Easier to test and maintain

**Trigger-based logging:**
- Catches all changes including direct SQL and admin operations
- Does not know who made the change (no user context in MySQL triggers)
- Adds overhead to every write operation
- Harder to debug and maintain

Recommendation: Use application-level logging as the primary mechanism. Add database triggers only if you need to catch out-of-band changes (direct SQL access, migration scripts, etc.).

### Example Trigger (for reference)

```sql
DELIMITER //

CREATE TRIGGER `trg_contacts_after_update`
AFTER UPDATE ON `contacts`
FOR EACH ROW
BEGIN
    DECLARE v_changes JSON DEFAULT JSON_OBJECT();

    IF OLD.first_name != NEW.first_name OR (OLD.first_name IS NULL) != (NEW.first_name IS NULL) THEN
        SET v_changes = JSON_SET(v_changes, '$.first_name',
            JSON_OBJECT('old', OLD.first_name, 'new', NEW.first_name));
    END IF;

    IF OLD.email != NEW.email OR (OLD.email IS NULL) != (NEW.email IS NULL) THEN
        SET v_changes = JSON_SET(v_changes, '$.email',
            JSON_OBJECT('old', OLD.email, 'new', NEW.email));
    END IF;

    IF OLD.lifecycle_stage != NEW.lifecycle_stage THEN
        SET v_changes = JSON_SET(v_changes, '$.lifecycle_stage',
            JSON_OBJECT('old', OLD.lifecycle_stage, 'new', NEW.lifecycle_stage));
    END IF;

    -- Only insert if something actually changed
    IF JSON_LENGTH(v_changes) > 0 THEN
        INSERT INTO audit_logs (entity_type, entity_id, action, user_id, changes)
        VALUES ('contact', NEW.id, 'update', NEW.updated_by, v_changes);
    END IF;
END //

DELIMITER ;
```

### Partitioning Audit Logs

Audit logs grow unboundedly. Partition by time period:

```sql
CREATE TABLE `audit_logs` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type` VARCHAR(50) NOT NULL,
    `entity_id` BIGINT UNSIGNED NOT NULL,
    `action` ENUM('create','update','delete','restore','merge','convert','assign') NOT NULL,
    `user_id` BIGINT UNSIGNED NULL,
    `changes` JSON NULL,
    `metadata` JSON NULL,
    `created_at` TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),

    PRIMARY KEY (`id`, `created_at`),
    INDEX `idx_audit_entity` (`entity_type`, `entity_id`, `created_at`),
    INDEX `idx_audit_user` (`user_id`, `created_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
PARTITION BY RANGE (UNIX_TIMESTAMP(`created_at`)) (
    PARTITION p2025_q1 VALUES LESS THAN (UNIX_TIMESTAMP('2025-04-01')),
    PARTITION p2025_q2 VALUES LESS THAN (UNIX_TIMESTAMP('2025-07-01')),
    PARTITION p2025_q3 VALUES LESS THAN (UNIX_TIMESTAMP('2025-10-01')),
    PARTITION p2025_q4 VALUES LESS THAN (UNIX_TIMESTAMP('2026-01-01')),
    PARTITION p2026_q1 VALUES LESS THAN (UNIX_TIMESTAMP('2026-04-01')),
    PARTITION p2026_q2 VALUES LESS THAN (UNIX_TIMESTAMP('2026-07-01')),
    PARTITION p_future VALUES LESS THAN MAXVALUE
);
```

Benefits:
- Old partitions can be archived or dropped without affecting active data
- Queries that filter by date range automatically prune irrelevant partitions
- Maintenance operations (OPTIMIZE, ANALYZE) can target individual partitions

## GDPR and Data Privacy Compliance

### Right to Erasure

GDPR requires the ability to permanently delete personal data on request. Soft deletes alone are not sufficient.

Implement a "hard purge" process:

```sql
-- Step 1: Anonymize the contact record
UPDATE contacts SET
    first_name = 'REDACTED',
    last_name = 'REDACTED',
    email = CONCAT('redacted_', id, '@deleted.invalid'),
    phone = NULL,
    mobile_phone = NULL,
    mailing_address_line1 = NULL,
    mailing_address_line2 = NULL,
    mailing_city = NULL,
    mailing_state = NULL,
    mailing_postal_code = NULL,
    mailing_country = NULL,
    linkedin_url = NULL,
    custom_fields = NULL,
    deleted_at = NOW()
WHERE id = :contact_id;

-- Step 2: Anonymize related audit log entries
UPDATE audit_logs SET
    changes = JSON_OBJECT('redacted', true, 'reason', 'gdpr_erasure_request')
WHERE entity_type = 'contact' AND entity_id = :contact_id;

-- Step 3: Log the erasure event itself
INSERT INTO audit_logs (entity_type, entity_id, action, user_id, metadata)
VALUES ('contact', :contact_id, 'delete', :admin_user_id,
    JSON_OBJECT('reason', 'gdpr_erasure_request', 'request_date', NOW()));
```

Key points:
- Anonymize rather than DELETE to preserve referential integrity and aggregate reporting.
- The anonymized email uses a pattern (`[REDACTED_SECRET]`) that is unique and clearly non-functional.
- Keep the audit log entry documenting that erasure happened (the erasure itself is a compliance event).

### Data Retention Policies

Define retention periods per data type:

| Data Type | Suggested Retention | Action After Expiry |
|-----------|-------------------|-------------------|
| Active CRM records | Indefinite | N/A |
| Soft-deleted records | 90 days | Hard purge or archive |
| Audit logs | 2-7 years | Archive to cold storage |
| Activity logs | 1-3 years | Archive or aggregate |
| Email/campaign events | 1-2 years | Aggregate into summary tables |

Implement automated retention jobs that run daily or weekly to enforce these policies.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)