Skill: ningwang770/database-agent (version 1.0.0)

--- name: database-agent description: 本技能应用于 Java 后端开发场景中的数据库操作。提供慢 SQL 智能分析、表结构规范巡检、安全数据订正与测试数据自动生成的自动化辅助能力。当用户请求数据库优化、schema 验证、安全数据更新或需要为数据库生成测试数据时使用此技能。 --- # Database Agent 数据库智能助手 ## 概述 本技能为 Java 后端开发提供智能数据库辅助能力，支持慢 SQL 分析、表结构规范巡检、安全数据订正和测试数据生成等自动化功能。可自动识别索引缺失、全表扫描、字段不规范等问题，生成优化建议与修复脚本；在数据订正时自动校验风险，防止无 WHERE 条件更新删除；并根据表结构智能生成业务合规的测试数据，大幅提升数据库开发与维护效率。 ## 核心能力 ### 1. 慢 SQL 分析 **何时使用：** 用户报告慢查询、性能问题或需要 SQL 优化时。 **功能特性：** - 解析 SQL 执行计划，识别性能瓶颈 - 检测全表扫描、缺失索引和低效 JOIN - 分析慢查询日志，定位问题查询 - 生成优化建议和索引创建脚本 **使用方法：** 1. 使用 `scripts/analyze_slow_sql.py` 分析 SQL 语句或执行计划 2. 参考 `references/sql_optimization_rules.md` 了解优化模式 3. 生成包含具体建议的报告 **示例场景：** - "分析这条慢 SQL：SELECT * FROM orders WHERE user_id = ?" - "优化这个执行太慢的查询" - "查找这个查询为什么做全表扫描" ### 2. 表结构规范巡检 **何时使用：** 用户需要根据最佳实践和标准验证数据库 schema 时。 **功能特性：** - 检查字段命名规范（snake_case、无保留字） - 验证数据类型和长度 - 检测缺失的主键或索引 - 识别不合规的默认值或可空字段 - 生成规范报告和修复脚本 **使用方法：** 1. 使用 `scripts/check_schema_compliance.py` 检查表结构 2. 参考 `references/database_standards.md` 了解规范规则 3. 使用 `scripts/generate_report.py` 导出结果到 Excel 报告 **示例场景：** - "检查这个表结构是否符合最佳实践" - "验证我们的数据库 schema 是否符合标准" - "为所有表生成规范报告" ### 3. 安全数据订正 **何时使用：** 用户需要安全地执行批量数据更新或删除时。 **功能特性：** - 验证 UPDATE/DELETE 语句的安全风险 - 检测缺失的 WHERE 子句（防止全表更新/删除） - 识别危险操作（如更新主键） - 要求对高风险操作进行确认 - 生成备份和回滚脚本 **使用方法：** 1. 使用 `scripts/validate_data_correction.py` 检查 SQL 安全性 2. 参考 `references/safe_operation_guidelines.md` 了解风险模式 3. 执行前必须生成回滚脚本 **示例场景：** - "将所有用户状态更新为激活" - "删除 1 年前的记录" - "帮我安全地订正这些数据" **关键安全规则：** - 永远不要执行没有 WHERE 子句的 UPDATE/DELETE - 始终生成备份脚本 - 对影响超过 1000 行的操作要求明确确认 - 记录所有数据订正操作 ### 4. 测试数据生成 **何时使用：** 用户需要用真实数据填充测试环境时。 **功能特性：** - 根据表结构智能生成测试数据 - 遵循外键关系 - 支持自定义数据模式（邮箱、电话、日期） - 生成业务合规数据（如有效的订单金额、真实的姓名） - 按正确的依赖顺序创建数据 **使用方法：** 1. 使用 `scripts/generate_test_data.py` 创建测试数据 2. 参考 `references/test_data_patterns.md` 了解数据生成规则 3. 通过 JSON 配置文件配置数量和模式 **示例场景：** - "为数据库生成 1000 个测试用户" - "创建包含真实数据的样本订单" - "用样本数据填充测试环境" ## 工作流程 ### SQL 分析任务 1. 接收 SQL 语句或执行计划 2. 运行 `scripts/analyze_slow_sql.py` 识别问题 3. 生成优化建议 4. 可选地创建索引创建脚本 ### Schema 验证任务 1. 连接数据库或接收 DDL 语句 2. 运行 `scripts/check_schema_compliance.py` 3. 生成规范报告 4. 可选地生成修复脚本 ### 数据订正任务 1. 接收 UPDATE/DELETE SQL 语句 2. 运行 `scripts/validate_data_correction.py` 进行安全检查 3. 如果安全，生成备份脚本 4. 要求用户确认 5. 执行并记录日志 ### 测试数据生成任务 1. 接收表结构或连接数据库 2. 配置数据模式和数量 3. 运行 `scripts/generate_test_data.py` 4. 按依赖顺序生成 INSERT 语句 ## 资源 ### scripts/ 脚本目录 - `analyze_slow_sql.py` - 分析 SQL 性能并生成优化建议 - `check_schema_compliance.py` - 根据标准验证表结构 - `validate_data_correction.py` - 验证数据修改操作的安全性 - `generate_test_data.py` - 根据表结构生成智能测试数据 - `generate_report.py` - 导出分析结果到 Excel 格式 - `database_connector.py` - JDBC 数据库连接工具 ### references/ 参考文档 - `sql_optimization_rules.md` - SQL 优化最佳实践和模式 - `database_standards.md` - 数据库命名约定和设计标准 - `safe_operation_guidelines.md` - 数据操作的风险模式和安全规则 - `test_data_patterns.md` - 数据生成模式和业务规则 ### assets/ 资产文件 - `compliance_rules.json` - 可配置的规范规则定义 - `test_data_config.json` - 测试数据生成配置模板 - `report_template.xlsx` - 规范报告的 Excel 报告模板 ## 重要提示 - 执行数据库操作前始终验证用户权限 - 记录所有操作以便审计追踪 - 为任何数据修改生成回滚脚本 - 首先在非生产环境中测试生成的脚本 - 遵守数据库连接限制和查询超时 - 对大数据量使用批量操作

path: references/test_data_patterns.md
content: # Test Data Generation Patterns

## Overview

This document defines patterns and rules for generating realistic, business-compliant test data. The goal is to create data that:
- Maintains referential integrity
- Follows business rules
- Looks realistic for testing
- Covers edge cases

## Data Type Patterns

### String Patterns

#### Email Addresses
```python
# Pattern: user{number}@test.com
[REDACTED_SECRET]
[REDACTED_SECRET]

# With domain variety
[REDACTED_SECRET]
[REDACTED_SECRET]

# Special cases
[REDACTED_SECRET]  # Plus addressing
```

#### Phone Numbers
```python
# Chinese mobile: 1 + 3/5/7/8/9 + 8 digits
13800138000
15912345678
18600001111

# International format
+8613800138000
+1-555-123-4567
```

#### Names
```python
# Chinese names (2-4 characters)
张三、李四、王五、赵六、钱七

# Western names
John Smith, Jane Doe, Bob Johnson

# Mix of both for international testing
张伟 (Zhang Wei), 李娜 (Li Na)
```

#### Addresses
```python
# Chinese format
北京市朝阳区建国路88号
上海市浦东新区张江高科技园区

# Western format
123 Main St, Apt 4B, New York, NY 10001
```

#### URLs
```python
# Test URLs
https://test.example.com
http://localhost:8080/api/v1
https://staging.company.com/products/123
```

### Numeric Patterns

#### IDs
```python
# Primary keys (auto-increment simulation)
1, 2, 3, 4, 5, ...

# Foreign keys (reference existing IDs)
# If users table has IDs 1-100
user_id: random.randint(1, 100)
```

#### Money/Prices
```python
# Reasonable price ranges
product_price: random.uniform(10, 10000)  # $10 - $10,000
order_total: random.uniform(100, 50000)   # $100 - $50,000

# Round to 2 decimal places
price: round(random.uniform(10, 1000), 2)
```

#### Quantities
```python
# Small quantities
quantity: random.randint(1, 100)

# Stock levels
stock: random.randint(0, 1000)

# Percentages
discount_rate: random.uniform(0, 1)  # 0.0 - 1.0
```

#### Age/Ratings
```python
# Age (reasonable range)
age: random.randint(18, 70)

# Ratings (1-5 stars)
rating: random.randint(1, 5)

# Scores (0-100)
score: random.randint(0, 100)
```

### Date/Time Patterns

#### Timestamps
```python
# Created at (past)
created_at: now - timedelta(days=random.randint(0, 365))

# Updated at (after created_at)
updated_at: created_at + timedelta(days=random.randint(0, 30))

# Future dates
expires_at: now + timedelta(days=random.randint(30, 365))
```

#### Date Ranges
```python
# Birth dates (18-70 years ago)
birth_date: now - timedelta(days=random.randint(18*365, 70*365))

# Event dates (future)
event_date: now + timedelta(days=random.randint(1, 90))

# Historical dates
transaction_date: now - timedelta(days=random.randint(0, 365))
```

#### Time Patterns
```python
# Business hours (9 AM - 6 PM)
business_time: time(random.randint(9, 18), random.randint(0, 59))

# Time slots (on the hour)
time_slot: f"{random.randint(9, 17)}:00:00"
```

### Boolean Patterns

```python
# Common status flags
is_active: random.choice([True, False])  # or weight toward True
is_verified: random.choice([True, False])
is_deleted: False  # Usually False, test soft-delete separately

# With probability weights
is_premium: random.choices([True, False], weights=[20, 80])[0]  # 20% premium users
```

## Business Domain Patterns

### E-commerce

#### Products
```python
{
    'id': auto_increment,
    'name': f"测试商品_{random.randint(1, 999)}",
    'sku': f"SKU{random.randint(100000, 999999)}",
    'price': round(random.uniform(10, 1000), 2),
    'stock': random.randint(0, 1000),
    'category_id': random.randint(1, 10),
    'status': random.choice(['active', 'inactive', 'out_of_stock']),
    'created_at': now - timedelta(days=random.randint(0, 365))
}
```

#### Orders
```python
{
    'id': auto_increment,
    'order_number': f"ORD{timestamp}{random.randint(1000, 9999)}",
    'user_id': random.randint(1, 1000),  # FK to users
    'total_amount': round(random.uniform(100, 5000), 2),
    'status': random.choice(['pending', 'paid', 'shipped', 'completed', 'cancelled']),
    'payment_method': random.choice(['credit_card', 'paypal', 'bank_transfer']),
    'shipping_address': f"测试地址{random.randint(1, 999)}号",
    'created_at': now - timedelta(days=random.randint(0, 90))
}
```

#### Order Items
```python
{
    'id': auto_increment,
    'order_id': random.randint(1, 10000),  # FK to orders
    'product_id': random.randint(1, 500),  # FK to products
    'quantity': random.randint(1, 5),
    'unit_price': round(random.uniform(10, 500), 2),
    'subtotal': quantity * unit_price
}
```

### User Management

#### Users
```python
{
    'id': auto_increment,
    'username': f"user{random.randint(1000, 9999)}",
    'email': f"user{random.randint(1, 10000)}@test.com",
    'phone': f"1{random.choice([3,5,7,8,9])}{random.randint(100000000, 999999999)}",
    'password_hash': hashlib.md5(f"password{random.randint(1, 100)}".encode()).hexdigest(),
    'nickname': random.choice(['张三', '李四', '王五', '赵六']),
    'age': random.randint(18, 60),
    'gender': random.choice(['M', 'F', 'Other']),
    'status': random.choice(['active', 'inactive', 'banned']),
    'is_verified': random.choice([True, False]),
    'created_at': now - timedelta(days=random.randint(0, 365))
}
```

#### User Profiles
```python
{
    'id': auto_increment,
    'user_id': random.randint(1, 1000),  # FK to users (unique)
    'avatar': f"https://test.example.com/avatars/{random.randint(1, 100)}.jpg",
    'bio': f"这是用户简介_{random.randint(1, 100)}",
    'address': f"测试地址{random.randint(1, 999)}号",
    'company': random.choice(['测试公司A', '测试公司B', '测试公司C']),
    'website': f"https://user{random.randint(1, 100)}.test.com"
}
```

### Content Management

#### Articles/Posts
```python
{
    'id': auto_increment,
    'title': f"测试文章_{random.randint(1, 999)}: 这是一篇测试文章",
    'content': f"这是文章内容...{lorem_ipsum_text}",
    'author_id': random.randint(1, 100),  # FK to users
    'category': random.choice(['tech', 'life', 'news', 'entertainment']),
    'tags': random.sample(['python', 'java', 'database', 'web', 'mobile'], k=random.randint(1, 3)),
    'status': random.choice(['draft', 'published', 'archived']),
    'view_count': random.randint(0, 10000),
    'like_count': random.randint(0, 1000),
    'published_at': now - timedelta(days=random.randint(0, 365)),
    'created_at': now - timedelta(days=random.randint(0, 365))
}
```

#### Comments
```python
{
    'id': auto_increment,
    'article_id': random.randint(1, 500),  # FK to articles
    'user_id': random.randint(1, 100),  # FK to users
    'parent_id': random.choice([None, random.randint(1, 100)]),  # For nested comments
    'content': f"这是评论内容_{random.randint(1, 999)}",
    'status': random.choice(['active', 'hidden', 'deleted']),
    'created_at': now - timedelta(days=random.randint(0, 90))
}
```

### Financial

#### Transactions
```python
{
    'id': auto_increment,
    'transaction_id': f"TXN{timestamp}{random.randint(10000, 99999)}",
    'user_id': random.randint(1, 1000),
    'type': random.choice(['deposit', 'withdrawal', 'transfer', 'payment']),
    'amount': round(random.uniform(100, 10000), 2),
    'balance_before': round(random.uniform(1000, 50000), 2),
    'balance_after': balance_before + amount,  # Or minus for withdrawal
    'status': random.choice(['pending', 'completed', 'failed', 'cancelled']),
    'description': f"交易描述_{random.randint(1, 999)}",
    'created_at': now - timedelta(days=random.randint(0, 365))
}
```

#### Accounts
```python
{
    'id': auto_increment,
    'user_id': random.randint(1, 1000),  # FK to users (unique)
    'account_number': f"ACC{random.randint(100000000, 999999999)}",
    'balance': round(random.uniform(0, 100000), 2),
    'currency': 'CNY',
    'status': random.choice(['active', 'frozen', 'closed']),
    'created_at': now - timedelta(days=random.randint(0, 365))
}
```

## Referential Integrity Patterns

### Parent-Child Relationships

#### Generate in Dependency Order
```python
# 1. Generate parent records first
for i in range(100):
    users.append(generate_user())

# 2. Then generate child records with valid FK references
for i in range(500):
    orders.append(generate_order(user_id=random.choice(users).id))

# 3. Then generate grandchildren
for i in range(1000):
    order_items.append(generate_order_item(
        order_id=random.choice(orders).id,
        product_id=random.choice(products).id
    ))
```

### Circular References

#### Handle self-referencing tables
```python
# Categories with parent_category_id
# Generate root categories first (parent_id = NULL)
root_categories = [generate_category(parent_id=None) for _ in range(5)]

# Then generate child categories
child_categories = []
for parent in root_categories:
    children = [generate_category(parent_id=parent.id) for _ in range(3)]
    child_categories.extend(children)
```

## Edge Cases and Special Values

### Boundary Values

```python
# Minimum values
quantity: 0  # Zero quantity
price: 0.01  # Minimum price
age: 18  # Minimum age

# Maximum values
quantity: 999999  # Large quantity
price: 999999.99  # Maximum price
age: 120  # Maximum realistic age

# Just above/below limits
stock: -1  # Invalid (for testing validation)
price: 0.00  # Free item
```

### NULL Values

```python
# Optional fields (10-20% NULL)
nickname: random.choice([None, f"昵称{random.randint(1, 100)}"])
bio: random.choice([None, f"简介{random.randint(1, 100)}"])

# Required fields (never NULL)
user_id: random.randint(1, 100)  # Always has value
created_at: now - timedelta(days=random.randint(0, 365))  # Always has value
```

### Unicode and Special Characters

```python
# Chinese characters
name: random.choice(['张三', '李四', '王五'])

# Emoji (for modern apps)
content: f"测试内容 {random.choice(['😀', '🎉', '👍', '❤️'])}"

# Special characters
description: f"特殊字符测试: <>&\"'{random.randint(1, 100)}"

# Mixed languages
title: f"测试Title{random.randint(1, 100)}_测试"
```

## Data Volume Guidelines

### Development Environment
```
Users: 100-1000
Products: 500-2000
Orders: 1000-5000
Order Items: 3000-10000
Comments: 500-2000
```

### Staging Environment
```
Users: 1000-10000
Products: 2000-10000
Orders: 10000-50000
Order Items: 30000-100000
Comments: 5000-20000
```

### Performance Testing
```
Users: 100000+
Products: 50000+
Orders: 1000000+
Order Items: 5000000+
```

## Example: Complete Table Generation

```python
# Define schema
user_schema = {
    'table_name': 'users',
    'columns': [
        {'name': 'id', 'type': 'BIGINT', 'is_primary_key': True},
        {'name': 'username', 'type': 'VARCHAR(50)'},
        {'name': 'email', 'type': 'VARCHAR(100)'},
        {'name': 'phone', 'type': 'VARCHAR(20)'},
        {'name': 'age', 'type': 'INT'},
        {'name': 'status', 'type': 'TINYINT'},
        {'name': 'created_at', 'type': 'TIMESTAMP'}
    ]
}

# Generate data
generator = TestDataGenerator()
users = generator.generate_for_table(user_schema, count=100)

# Generate SQL
statements = generator.generate_batch_insert('users', users, batch_size=100)

# Output
for stmt in statements:
    print(stmt)
```

## Best Practices

1. ✅ **Generate parent tables first** - Maintain referential integrity
2. ✅ **Use realistic value ranges** - Match production data distribution
3. ✅ **Include edge cases** - Test boundary conditions
4. ✅ **Respect constraints** - Don't violate FK or unique constraints
5. ✅ **Use batch inserts** - Better performance for large datasets
6. ✅ **Make data deterministic** - Use seeds for reproducible tests
7. ✅ **Include NULL values** - Test optional fields
8. ✅ **Use consistent formats** - Same phone format throughout
9. ✅ **Document patterns** - Comment special generation rules
10. ✅ **Validate after generation** - Check FK references exist


path: references/safe_operation_guidelines.md
content: # Safe Database Operation Guidelines

## Overview

Database operations, especially bulk updates and deletions, carry significant risk. This guide establishes safety protocols to prevent data loss, ensure reversibility, and maintain data integrity.

## Core Safety Principles

### 1. Always Have a Backup

**Before any data modification:**
```sql
-- Create backup table
CREATE TABLE users_backup_20240101 AS SELECT * FROM users;

-- Or export to file
SELECT * INTO OUTFILE '/tmp/users_backup.csv'
FIELDS TERMINATED BY ','
FROM users;
```

### 2. Always Use Transactions

```sql
-- Start transaction
START TRANSACTION;

-- Perform operations
UPDATE users SET status = 1 WHERE created_at < '2024-01-01';

-- Verify results
SELECT COUNT(*) FROM users WHERE status = 1;

-- Commit only if everything is correct
COMMIT;
-- Or rollback if issues found
-- ROLLBACK;
```

### 3. Test in Non-Production First

- Always test SQL scripts in development/staging environment
- Verify data volume and execution time
- Check for unexpected side effects

### 4. Get Confirmation for Large Operations

- Operations affecting >1000 rows require review
- Operations affecting >10000 rows require approval
- Always estimate affected rows before execution

## Risk Assessment Matrix

| Operation Type | Risk Level | Required Actions |
|---------------|------------|------------------|
| SELECT queries | Low | None |
| Single-row INSERT | Low | None |
| Single-row UPDATE with WHERE by PK | Low | Backup recommended |
| Multi-row UPDATE (1-1000 rows) | Medium | Backup + Transaction + Review |
| Multi-row UPDATE (1000+ rows) | High | Backup + Transaction + Approval + Batch Processing |
| DELETE with WHERE by PK | Medium | Backup + Transaction + Review |
| DELETE with complex WHERE | High | Backup + Transaction + Approval |
| DELETE without WHERE | CRITICAL | NEVER ALLOWED |
| TRUNCATE TABLE | CRITICAL | NEVER ALLOWED |
| DROP TABLE | CRITICAL | NEVER ALLOWED |

## Dangerous Operations Checklist

### ❌ NEVER Do These

1. **UPDATE/DELETE without WHERE clause**
   ```sql
   -- NEVER DO THIS
   UPDATE users SET status = 1;  -- Updates ALL rows!
   DELETE FROM orders;  -- Deletes ALL rows!
   
   -- ALWAYS use WHERE
   UPDATE users SET status = 1 WHERE id = 123;
   DELETE FROM orders WHERE id = 456;
   ```

2. **WHERE clause that's always true**
   ```sql
   -- NEVER DO THIS
   DELETE FROM users WHERE 1=1;
   DELETE FROM users WHERE TRUE;
   
   -- Use specific conditions
   DELETE FROM users WHERE created_at < '2020-01-01' AND status = 'inactive';
   ```

3. **OR 1=1 injection pattern**
   ```sql
   -- NEVER DO THIS
   DELETE FROM users WHERE id = 123 OR 1=1;
   ```

4. **TRUNCATE without backup**
   ```sql
   -- AVOID THIS - irreversible
   TRUNCATE TABLE orders;
   
   -- Use DELETE instead (slower but safer)
   DELETE FROM orders WHERE created_at < '2023-01-01';
   ```

5. **DROP TABLE without backup**
   ```sql
   -- AVOID THIS - irreversible
   DROP TABLE old_data;
   
   -- Rename instead
   RENAME TABLE old_data TO old_data_backup_20240101;
   ```

### ⚠️ REQUIRES Extra Caution

1. **Updating primary keys**
   ```sql
   -- Dangerous - may break foreign key relationships
   UPDATE users SET id = 999 WHERE id = 1;
   
   -- Better: Don't update primary keys
   -- Or ensure all FKs are updated first
   ```

2. **Deleting by non-indexed column**
   ```sql
   -- Slow on large tables, may lock table
   DELETE FROM orders WHERE YEAR(created_at) = 2023;
   
   -- Better: Use indexed column
   DELETE FROM orders 
   WHERE created_at BETWEEN '2023-01-01' AND '2023-12-31'
   LIMIT 1000;  -- Batch delete
   ```

3. **Cascading deletes**
   ```sql
   -- Dangerous - may delete more than expected
   CREATE TABLE orders (
       user_id INT,
       FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
   );
   
   -- Better: Use RESTRICT and delete manually
   FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE RESTRICT
   ```

## Safe Operation Procedures

### For UPDATE Operations

#### Step 1: Estimate Affected Rows
```sql
-- Count before executing
SELECT COUNT(*) FROM users 
WHERE last_login < '2023-01-01' AND status = 'active';

-- Expected: 1234 rows
```

#### Step 2: Create Backup
```sql
-- Create backup table with timestamp
CREATE TABLE users_backup_20240101_143000 AS 
SELECT * FROM users 
WHERE last_login < '2023-01-01' AND status = 'active';
```

#### Step 3: Start Transaction
```sql
START TRANSACTION;
```

#### Step 4: Execute UPDATE
```sql
-- For large operations, batch process
UPDATE users 
SET status = 'inactive' 
WHERE last_login < '2023-01-01' 
  AND status = 'active'
LIMIT 1000;

-- Repeat until all rows updated
-- Check rows affected after each batch
```

#### Step 5: Verify Results
```sql
-- Check the update
SELECT status, COUNT(*) 
FROM users 
WHERE last_login < '2023-01-01'
GROUP BY status;

-- Verify count matches expected
```

#### Step 6: Commit or Rollback
```sql
-- If verification passed
COMMIT;

-- If issues found
ROLLBACK;
-- Restore from backup if needed
INSERT INTO users SELECT * FROM users_backup_20240101_143000;
```

### For DELETE Operations

#### Step 1: Estimate and Backup
```sql
-- Count
SELECT COUNT(*) FROM logs WHERE created_at < '2023-01-01';

-- Backup (IMPORTANT!)
CREATE TABLE logs_backup_20240101 AS 
SELECT * FROM logs WHERE created_at < '2023-01-01';
```

#### Step 2: Batch Delete with LIMIT
```sql
-- Start transaction
START TRANSACTION;

-- Batch delete (repeat until 0 rows affected)
DELETE FROM logs 
WHERE created_at < '2023-01-01' 
LIMIT 1000;

-- Repeat...
-- When done, commit
COMMIT;
```

#### Step 3: Verify and Rollback Script Ready
```sql
-- Verification
SELECT COUNT(*) FROM logs WHERE created_at < '2023-01-01';
-- Should be 0

-- Rollback script (prepare before operation)
INSERT INTO logs SELECT * FROM logs_backup_20240101;
```

### For Bulk INSERT Operations

#### Use Batch Insert
```sql
-- Bad: Individual inserts
INSERT INTO users (name) VALUES ('User1');
INSERT INTO users (name) VALUES ('User2');
-- ... slow and inefficient

-- Good: Batch insert
INSERT INTO users (name, email) VALUES
    ('User1', '[REDACTED_SECRET]'),
    ('User2', '[REDACTED_SECRET]'),
    ('User3', '[REDACTED_SECRET]');
-- Up to 1000 rows per batch
```

#### For Very Large Datasets
```sql
-- Use LOAD DATA INFILE (MySQL)
LOAD DATA INFILE '/path/to/data.csv'
INTO TABLE users
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(name, email, created_at);

-- Or use COPY (PostgreSQL)
COPY users(name, email, created_at)
FROM '/path/to/data.csv'
DELIMITER ','
CSV HEADER;
```

## Rollback Procedures

### 1. Using Transaction Rollback
```sql
-- If operation is still in transaction
ROLLBACK;
```

### 2. Using Backup Table
```sql
-- For UPDATE
UPDATE target_table t
JOIN backup_table b ON t.id = b.id
SET t.column = b.column;

-- For DELETE
INSERT INTO target_table 
SELECT * FROM backup_table;

-- For DROP
RENAME TABLE backup_table TO target_table;
```

### 3. Using Binary Logs (MySQL)
```bash
# Find the transaction in binlog
mysqlbinlog /var/lib/mysql/mysql-bin.000123

# Generate rollback SQL
mysqlbinlog --flashback \
  --start-datetime="2024-01-01 14:00:00" \
  --stop-datetime="2024-01-01 14:30:00" \
  /var/lib/mysql/mysql-bin.000123 > rollback.sql

# Execute rollback
mysql -u root -p database < rollback.sql
```

## Monitoring and Logging

### Log All Data Modifications
```sql
-- Create audit log table
CREATE TABLE data_modification_log (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    table_name VARCHAR(50) NOT NULL,
    operation VARCHAR(10) NOT NULL,
    affected_rows INT NOT NULL,
    sql_statement TEXT NOT NULL,
    executed_by VARCHAR(50) NOT NULL,
    executed_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

-- Log after each operation
INSERT INTO data_modification_log 
(table_name, operation, affected_rows, sql_statement, executed_by)
VALUES 
('users', 'UPDATE', 1234, 'UPDATE users SET status = ...', 'admin');
```

### Monitor Long-Running Operations
```sql
-- Check running queries
SHOW PROCESSLIST;

-- Kill long-running query if needed
KILL 12345;  -- Process ID
```

## Emergency Procedures

### Accidental Data Deletion

1. **Stop all writes immediately**
   ```sql
   -- Lock tables (if needed)
   FLUSH TABLES WITH READ LOCK;
   ```

2. **Don't close the connection** - Transaction may still be rollbackable

3. **Check for backup availability**
   - Check backup table
   - Check binary logs
   - Check database backups

4. **Contact DBA immediately**

### Recovery Steps
```sql
-- Option 1: Restore from backup table
INSERT INTO target_table SELECT * FROM backup_table;

-- Option 2: Use binary logs
-- (See Rollback Procedures section)

-- Option 3: Restore from database backup
-- (Contact DBA)
```

## Checklist Before Data Modification

- [ ] Estimated number of affected rows
- [ ] Created backup table or exported data
- [ ] Started transaction
- [ ] Prepared rollback script
- [ ] Tested in non-production environment
- [ ] Verified WHERE clause logic
- [ ] Added LIMIT for large operations
- [ ] Got approval for high-risk operations (>1000 rows)
- [ ] Scheduled during low-traffic period
- [ ] Notified team members
- [ ] Monitoring tools ready
- [ ] Documented the operation

## Best Practices Summary

1. ✅ **ALWAYS use WHERE clause** in UPDATE/DELETE
2. ✅ **ALWAYS use LIMIT** for batch operations
3. ✅ **ALWAYS use transactions** for data modifications
4. ✅ **ALWAYS create backup** before risky operations
5. ✅ **ALWAYS test in staging** before production
6. ✅ **ALWAYS estimate affected rows** first
7. ✅ **ALWAYS verify results** before committing
8. ✅ **ALWAYS have rollback plan** ready
9. ✅ **ALWAYS log operations** for audit trail
10. ✅ **ALWAYS get approval** for large operations

## Prohibited Actions

1. ❌ UPDATE/DELETE without WHERE clause
2. ❌ TRUNCATE TABLE without explicit approval
3. ❌ DROP TABLE without backup
4. ❌ Executing untested SQL in production
5. ❌ Modifying data without transaction
6. ❌ Skipping backup for bulk operations
7. ❌ WHERE 1=1 or WHERE TRUE conditions
8. ❌ OR 1=1 patterns (injection risk)
9. ❌ Updating primary key columns
10. ❌ Disabling foreign key checks


path: references/database_standards.md
content: # Database Design Standards and Naming Conventions

## Naming Conventions

### Table Naming

1. **Use lowercase letters, numbers, and underscores only**
   ```
   ✅ Good: users, order_items, product_categories
   ❌ Bad: Users, OrderItems, ProductCategories
   ```

2. **Use plural names for tables**
   ```
   ✅ Good: users, orders, products
   ❌ Bad: user, order, product
   ```

3. **Use snake_case (not camelCase)**
   ```
   ✅ Good: user_profiles, order_history
   ❌ Bad: userProfiles, orderHistory
   ```

4. **Avoid reserved keywords**
   ```
   ❌ Bad: order, group, select, index, key, value
   ✅ Good: orders, user_groups, selection, indexes, key_value
   ```

5. **Prefix tables by module/domain (optional but recommended)**
   ```
   ✅ Good: 
      - auth_users, auth_roles (authentication module)
      - pay_orders, pay_transactions (payment module)
      - inv_products, inv_inventory (inventory module)
   ```

### Column Naming

1. **Use lowercase letters, numbers, and underscores**
   ```
   ✅ Good: user_id, created_at, order_total
   ❌ Bad: userId, createdAt, orderTotal
   ```

2. **Use singular names for columns**
   ```
   ✅ Good: user_name, email_address
   ❌ Bad: users_name, email_addresses
   ```

3. **Primary key should be `id` or `{table}_id`**
   ```
   ✅ Good: 
      - id (auto-increment)
      - user_id (if more descriptive needed)
   ```

4. **Foreign key should be `{referenced_table}_id`**
   ```
   ✅ Good: user_id, order_id, product_id
   ❌ Bad: uid, oid, pid
   ```

5. **Boolean columns should use `is_`, `has_`, or `can_` prefix**
   ```
   ✅ Good: is_active, has_verified, can_edit
   ❌ Bad: active, verified, editable
   ```

6. **Date/time columns should have appropriate suffixes**
   ```
   ✅ Good: 
      - created_at, updated_at, deleted_at (timestamps)
      - birth_date, start_date, end_date (dates)
   ```

### Index Naming

1. **Primary key index: `pk_{table}`**
   ```
   ✅ Good: pk_users, pk_orders
   ```

2. **Unique index: `uk_{table}_{columns}`**
   ```
   ✅ Good: uk_users_email, uk_orders_order_number
   ```

3. **Regular index: `idx_{table}_{columns}`**
   ```
   ✅ Good: idx_users_name, idx_orders_user_id
   ```

4. **Composite index: `idx_{table}_{col1}_{col2}`**
   ```
   ✅ Good: idx_orders_user_created, idx_products_category_status
   ```

5. **Foreign key index: `fk_{table}_{referenced_table}`**
   ```
   ✅ Good: fk_orders_users, fk_order_items_orders
   ```

## Data Type Standards

### Numeric Types

| Type | Usage | Example |
|------|-------|---------|
| TINYINT | Small integers, flags, status | status TINYINT DEFAULT 0 |
| SMALLINT | Small range values | age SMALLINT |
| INT | Primary keys, counts, IDs | id INT AUTO_INCREMENT PRIMARY KEY |
| BIGINT | Large IDs, counts, timestamps | id BIGINT AUTO_INCREMENT PRIMARY KEY |
| DECIMAL(p,s) | Money, precise decimals | price DECIMAL(10,2) |
| FLOAT | Approximate decimals | distance FLOAT |

**Rules:**
- Always use INT or BIGINT for primary keys
- Use DECIMAL for money, never FLOAT
- Choose smallest type that fits the range

### String Types

| Type | Usage | Max Length |
|------|-------|------------|
| CHAR(n) | Fixed-length strings | 255 |
| VARCHAR(n) | Variable-length strings | 65535 |
| TEXT | Long text content | 65535 |
| MEDIUMTEXT | Medium-length text | 16MB |
| LONGTEXT | Very long text | 4GB |

**Rules:**
- Use VARCHAR for variable-length strings up to 255 chars
- Use TEXT for longer content
- Specify reasonable length limits
  - email: VARCHAR(100)
  - phone: VARCHAR(20)
  - name: VARCHAR(50)
  - title: VARCHAR(200)
  - description: TEXT

### Date and Time Types

| Type | Format | Usage |
|------|--------|-------|
| DATE | YYYY-MM-DD | Birth dates, events |
| TIME | HH:MM:SS | Time only |
| DATETIME | YYYY-MM-DD HH:MM:SS | Timestamps with time zone |
| TIMESTAMP | Unix timestamp | Auto-updating timestamps |

**Rules:**
- Use TIMESTAMP for created_at, updated_at (auto-update)
- Use DATETIME for business dates that should not auto-update
- Use DATE for date-only values (birth_date, event_date)

## Primary Key Standards

### Rules

1. **Every table MUST have a primary key**
   ```sql
   ✅ Good:
   CREATE TABLE users (
       id INT AUTO_INCREMENT PRIMARY KEY,
       name VARCHAR(50)
   );
   
   ❌ Bad:
   CREATE TABLE users (
       name VARCHAR(50)
   );
   ```

2. **Use auto-increment integers or UUIDs**
   ```sql
   -- Auto-increment (recommended)
   id INT AUTO_INCREMENT PRIMARY KEY
   
   -- Or UUID (for distributed systems)
   id CHAR(36) PRIMARY KEY DEFAULT (UUID())
   ```

3. **Keep primary key immutable**
   - Never update primary key values
   - Don't use business data as primary key

## Foreign Key Standards

### Rules

1. **Always define foreign key constraints**
   ```sql
   ✅ Good:
   CREATE TABLE orders (
       id INT AUTO_INCREMENT PRIMARY KEY,
       user_id INT NOT NULL,
       FOREIGN KEY (user_id) REFERENCES users(id) 
           ON DELETE RESTRICT
           ON UPDATE CASCADE
   );
   ```

2. **Choose appropriate ON DELETE action**
   - `RESTRICT`: Prevent deletion (default, safest)
   - `CASCADE`: Delete related records
   - `SET NULL`: Set to NULL (if column allows NULL)
   - `NO ACTION`: Same as RESTRICT

3. **Index foreign key columns**
   ```sql
   CREATE INDEX idx_orders_user_id ON orders(user_id);
   ```

## NOT NULL Standards

### Rules

1. **Primary keys: always NOT NULL**
   ```sql
   id INT NOT NULL AUTO_INCREMENT PRIMARY KEY
   ```

2. **Foreign keys: prefer NOT NULL**
   ```sql
   user_id INT NOT NULL  -- Required relationship
   deleted_by INT NULL   -- Optional relationship
   ```

3. **Business-critical fields: NOT NULL with DEFAULT**
   ```sql
   status TINYINT NOT NULL DEFAULT 0,
   created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
   ```

4. **Optional fields: allow NULL**
   ```sql
   nickname VARCHAR(50) NULL,
   bio TEXT NULL,
   deleted_at TIMESTAMP NULL
   ```

## Default Value Standards

### Rules

1. **Always provide defaults for NOT NULL columns**
   ```sql
   status TINYINT NOT NULL DEFAULT 0,
   is_active BOOLEAN NOT NULL DEFAULT TRUE,
   created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
   ```

2. **Use meaningful defaults**
   ```sql
   ✅ Good:
   role VARCHAR(20) NOT NULL DEFAULT 'user',
   currency VARCHAR(3) NOT NULL DEFAULT 'CNY'
   
   ❌ Bad:
   role VARCHAR(20) NOT NULL DEFAULT ''
   ```

3. **TIMESTAMP auto-update**
   ```sql
   created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
   updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
   ```

## Index Standards

### When to Create Indexes

1. **Primary key** (automatically indexed)
2. **Foreign key columns**
3. **Columns in WHERE clauses**
4. **Columns in ORDER BY / GROUP BY**
5. **Columns used in JOINs**
6. **Unique constraint columns**

### Index Design Rules

1. **Use composite indexes for multiple conditions**
   ```sql
   -- Query: WHERE status = ? AND created_at > ?
   CREATE INDEX idx_orders_status_created ON orders(status, created_at);
   ```

2. **Order matters in composite indexes**
   ```sql
   -- Good for: WHERE status = ? AND user_id = ?
   -- Good for: WHERE status = ?
   -- Bad for: WHERE user_id = ?
   CREATE INDEX idx_orders_status_user ON orders(status, user_id);
   ```

3. **Avoid redundant indexes**
   ```sql
   ❌ Bad:
   INDEX idx_user (user_id),
   INDEX idx_user_status (user_id, status)  -- idx_user is redundant
   
   ✅ Good:
   INDEX idx_user_status (user_id, status)  -- Covers both queries
   ```

## Table Structure Example

```sql
CREATE TABLE orders (
    -- Primary key
    id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,
    
    -- Foreign keys
    user_id BIGINT NOT NULL COMMENT 'User ID',
    product_id BIGINT NOT NULL COMMENT 'Product ID',
    
    -- Business fields
    order_number VARCHAR(32) NOT NULL COMMENT 'Order number',
    total_amount DECIMAL(10,2) NOT NULL COMMENT 'Total amount',
    status TINYINT NOT NULL DEFAULT 0 COMMENT 'Order status: 0-pending, 1-paid, 2-shipped, 3-completed, 4-cancelled',
    
    -- Timestamps
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Creation time',
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT 'Update time',
    deleted_at TIMESTAMP NULL COMMENT 'Soft delete time',
    
    -- Unique constraints
    UNIQUE KEY uk_order_number (order_number),
    
    -- Indexes
    KEY idx_user_id (user_id),
    KEY idx_product_id (product_id),
    KEY idx_status_created (status, created_at),
    
    -- Foreign keys
    CONSTRAINT fk_orders_users FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE RESTRICT,
    CONSTRAINT fk_orders_products FOREIGN KEY (product_id) REFERENCES products(id) ON DELETE RESTRICT
    
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Orders table';
```

## Common Violations

### 1. Missing Primary Key
```sql
❌ Bad:
CREATE TABLE user_settings (
    user_id INT,
    setting_key VARCHAR(50),
    setting_value TEXT
);

✅ Good:
CREATE TABLE user_settings (
    id INT AUTO_INCREMENT PRIMARY KEY,
    user_id INT NOT NULL,
    setting_key VARCHAR(50) NOT NULL,
    setting_value TEXT,
    UNIQUE KEY uk_user_setting (user_id, setting_key)
);
```

### 2. Inconsistent Naming
```sql
❌ Bad:
CREATE TABLE UserProfiles (
    UserID INT,
    FirstName VARCHAR(50),
    created_Time TIMESTAMP
);

✅ Good:
CREATE TABLE user_profiles (
    id INT AUTO_INCREMENT PRIMARY KEY,
    user_id INT NOT NULL,
    first_name VARCHAR(50),
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
```

### 3. Wrong Data Type
```sql
❌ Bad:
price VARCHAR(20),  -- Money stored as string
age TINYINT,        -- OK, but could be inconsistent
phone INT           -- Leading zeros lost

✅ Good:
price DECIMAL(10,2),
age TINYINT UNSIGNED,
phone VARCHAR(20)
```

### 4. Missing Foreign Key Constraints
```sql
❌ Bad:
CREATE TABLE orders (
    id INT PRIMARY KEY,
    user_id INT  -- No constraint
);

✅ Good:
CREATE TABLE orders (
    id INT PRIMARY KEY,
    user_id INT NOT NULL,
    FOREIGN KEY (user_id) REFERENCES users(id)
);
```

### 5. Missing Indexes on Foreign Keys
```sql
❌ Bad:
CREATE TABLE orders (
    id INT PRIMARY KEY,
    user_id INT,
    FOREIGN KEY (user_id) REFERENCES users(id)
    -- Missing index on user_id
);

✅ Good:
CREATE TABLE orders (
    id INT PRIMARY KEY,
    user_id INT,
    FOREIGN KEY (user_id) REFERENCES users(id),
    KEY idx_user_id (user_id)
);
```


path: references/sql_optimization_rules.md
content: # SQL Optimization Rules and Best Practices

## Index Optimization

### When to Create Indexes

1. **Columns frequently used in WHERE clauses**
   ```sql
   -- Good: Add index on frequently queried column
   CREATE INDEX idx_user_id ON orders(user_id);
   
   -- Query benefits from index
   SELECT * FROM orders WHERE user_id = 123;
   ```

2. **Columns used in JOIN conditions**
   ```sql
   -- Add index on foreign key columns
   CREATE INDEX idx_order_user_fk ON orders(user_id);
   
   -- JOIN performs better
   SELECT o.*, u.name 
   FROM orders o 
   JOIN users u ON o.user_id = u.id;
   ```

3. **Columns used in ORDER BY and GROUP BY**
   ```sql
   -- Composite index for ORDER BY
   CREATE INDEX idx_status_created ON orders(status, created_at);
   
   -- Query with both WHERE and ORDER BY
   SELECT * FROM orders 
   WHERE status = 'pending' 
   ORDER BY created_at DESC;
   ```

### When NOT to Create Indexes

1. **Small tables** (< 1000 rows)
   - Full table scan may be faster than index lookup

2. **Columns with low cardinality**
   - Gender, status with few distinct values
   - MySQL may ignore the index anyway

3. **Frequently updated columns**
   - Index maintenance overhead
   - Write performance degradation

4. **Columns with many NULL values**
   - Index may not be effective

## Query Optimization Patterns

### 1. Avoid SELECT *

```sql
-- Bad: Retrieves unnecessary columns
SELECT * FROM users WHERE id = 1;

-- Good: Select only needed columns
SELECT id, name, email FROM users WHERE id = 1;
```

### 2. Use Index-Friendly LIKE

```sql
-- Bad: Leading wildcard prevents index usage
SELECT * FROM users WHERE name LIKE '%john%';

-- Good: Trailing wildcard can use index
SELECT * FROM users WHERE name LIKE 'john%';

-- Better: Use full-text search for complex patterns
SELECT * FROM users WHERE MATCH(name) AGAINST('john');
```

### 3. Avoid Functions on Indexed Columns

```sql
-- Bad: Function prevents index usage
SELECT * FROM orders WHERE DATE(created_at) = '2024-01-01';

-- Good: Use range query
SELECT * FROM orders 
WHERE created_at >= '2024-01-01' 
  AND created_at < '2024-01-02';

-- Or use BETWEEN
SELECT * FROM orders 
WHERE created_at BETWEEN '2024-01-01' AND '2024-01-01 23:59:59';
```

### 4. Optimize OR Conditions

```sql
-- Bad: OR may prevent index usage
SELECT * FROM orders 
WHERE user_id = 1 OR user_id = 2 OR user_id = 3;

-- Good: Use IN clause
SELECT * FROM orders WHERE user_id IN (1, 2, 3);

-- Or use UNION for complex conditions
SELECT * FROM orders WHERE user_id = 1
UNION
SELECT * FROM orders WHERE status = 'pending';
```

### 5. Use EXISTS Instead of IN for Subqueries

```sql
-- Bad: IN with subquery may be slow
SELECT * FROM users 
WHERE id IN (SELECT user_id FROM orders WHERE total > 1000);

-- Good: EXISTS is usually faster
SELECT u.* 
FROM users u
WHERE EXISTS (
    SELECT 1 FROM orders o 
    WHERE o.user_id = u.id AND o.total > 1000
);
```

### 6. Avoid DISTINCT When Possible

```sql
-- Bad: DISTINCT indicates duplicate data
SELECT DISTINCT user_id FROM orders;

-- Good: Use GROUP BY or fix the join
SELECT user_id FROM orders GROUP BY user_id;

-- Or check join conditions
SELECT DISTINCT o.user_id 
FROM orders o 
JOIN order_items oi ON o.id = oi.order_id;
-- May need: AND oi.deleted = 0
```

### 7. Optimize Pagination

```sql
-- Bad: Large offset is slow
SELECT * FROM orders ORDER BY id LIMIT 10000, 20;

-- Good: Use cursor-based pagination
SELECT * FROM orders 
WHERE id > 10000 
ORDER BY id 
LIMIT 20;

-- Or use deferred join for complex queries
SELECT o.* 
FROM orders o
INNER JOIN (
    SELECT id FROM orders ORDER BY id LIMIT 10000, 20
) tmp ON o.id = tmp.id;
```

### 8. Use Covering Indexes

```sql
-- Create covering index with all selected columns
CREATE INDEX idx_user_covering ON users(status, name, email);

-- Query can be satisfied by index only
SELECT name, email FROM users WHERE status = 'active';
-- No need to access table data
```

## Join Optimization

### 1. Use Appropriate Join Types

```sql
-- INNER JOIN: Only matching rows
SELECT o.*, u.name 
FROM orders o 
INNER JOIN users u ON o.user_id = u.id;

-- LEFT JOIN: All from left, matching from right
SELECT u.*, o.id as order_id
FROM users u
LEFT JOIN orders o ON u.id = o.user_id;

-- Use LEFT JOIN to find users without orders
SELECT u.* 
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
WHERE o.id IS NULL;
```

### 2. Join Order Matters

```sql
-- Start with smallest table or most selective filter
SELECT o.*, u.name, p.title
FROM orders o  -- Filter first
JOIN users u ON o.user_id = u.id
JOIN products p ON o.product_id = p.id
WHERE o.created_at > '2024-01-01';  -- Reduces rows early
```

### 3. Avoid Cartesian Products

```sql
-- Bad: Missing join condition
SELECT u.name, o.total
FROM users u, orders o;

-- Good: Always specify join condition
SELECT u.name, o.total
FROM users u
INNER JOIN orders o ON u.id = o.user_id;
```

## Subquery Optimization

### 1. Correlated vs Non-Correlated Subqueries

```sql
-- Correlated: Executed for each row (slow)
SELECT * FROM users u
WHERE (SELECT COUNT(*) FROM orders o WHERE o.user_id = u.id) > 5;

-- Better: Use JOIN with GROUP BY
SELECT u.* 
FROM users u
INNER JOIN orders o ON u.id = o.user_id
GROUP BY u.id
HAVING COUNT(o.id) > 5;
```

### 2. Use Derived Tables for Complex Queries

```sql
-- Derived table for intermediate results
SELECT u.name, order_stats.total_orders
FROM users u
INNER JOIN (
    SELECT user_id, COUNT(*) as total_orders
    FROM orders
    GROUP BY user_id
) order_stats ON u.id = order_stats.user_id;
```

## Execution Plan Analysis

### Key Metrics to Check

1. **type** (MySQL) / **Scan Type** (PostgreSQL)
   - `ALL`: Full table scan (usually bad)
   - `index`: Full index scan
   - `range`: Index range scan (good)
   - `ref`: Index lookup (good)
   - `const`: Single row lookup (best)

2. **Extra** field warnings
   - `Using filesort`: Slow sorting
   - `Using temporary`: Temporary table created
   - `Using index`: Good - covering index used
   - `Using where`: Filtering in progress

### Example Analysis

```sql
EXPLAIN SELECT o.*, u.name 
FROM orders o 
JOIN users u ON o.user_id = u.id 
WHERE o.status = 'pending';

-- Look for:
-- 1. No 'ALL' type scans
-- 2. Appropriate indexes used (key column)
-- 3. Reasonable rows estimate
-- 4. No filesort or temporary table warnings
```

## Common Anti-Patterns

### 1. N+1 Query Problem

```java
// Bad: Query inside loop
List<User> users = getUsers();
for (User user : users) {
    List<Order> orders = getOrdersByUser(user.getId());
}

-- Good: Single query with JOIN
SELECT u.*, o.id as order_id, o.total
FROM users u
LEFT JOIN orders o ON u.id = o.user_id;
```

### 2. Unnecessary Subqueries

```sql
-- Bad: Subquery for simple value
SELECT * FROM orders 
WHERE user_id = (SELECT id FROM users WHERE email = '[REDACTED_SECRET]');

-- Good: Use JOIN
SELECT o.* 
FROM orders o
JOIN users u ON o.user_id = u.id
WHERE u.email = '[REDACTED_SECRET]';
```

### 3. Implicit Type Conversion

```sql
-- Bad: String comparison with number column
SELECT * FROM orders WHERE user_id = '123';  -- user_id is INT

-- Good: Use correct type
SELECT * FROM orders WHERE user_id = 123;
```

## Performance Monitoring Queries

### Find Slow Queries

```sql
-- MySQL: Query performance schema
SELECT * FROM sys.statements_with_runtimes_in_95th_percentile;

-- PostgreSQL: pg_stat_statements
SELECT query, calls, total_time, mean_time
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;
```

### Find Missing Indexes

```sql
-- MySQL: Unused indexes
SELECT * FROM sys.schema_unused_indexes;

-- PostgreSQL: Suggested indexes
SELECT * FROM pg_stat_user_indexes 
WHERE idx_scan = 0 AND idx_tup_read = 0;
```

### Table Size Analysis

```sql
-- MySQL
SELECT 
    table_name,
    ROUND((data_length + index_length) / 1024 / 1024, 2) AS size_mb
FROM information_schema.tables
WHERE table_schema = DATABASE()
ORDER BY (data_length + index_length) DESC;

-- PostgreSQL
SELECT 
    relname AS table_name,
    pg_size_pretty(pg_total_relation_size(relid)) AS size
FROM pg_stat_user_tables
ORDER BY pg_total_relation_size(relid) DESC;
```


path: scripts/database_connector.py
content: #!/usr/bin/env python3
"""
Database Connector
Provides JDBC-like database connection utilities for Python.
"""

import argparse
import json
from typing import Dict, List, Any, Optional


class DatabaseConnector:
    """Database connection and query utilities."""
    
    def __init__(self, config: Dict):
        """
        Initialize database connector.
        
        Args:
            config: Database configuration dictionary
                {
                    'host': str,
                    'port': int,
                    'database': str,
                    'user': str,
                    'password': str,
                    'type': str (mysql, postgresql, etc.)
                }
        """
        self.config = config
        self.connection = None
        
    def connect(self):
        """Establish database connection."""
        try:
            import pymysql
            db_type = self.config.get('type', 'mysql')
            
            if db_type == 'mysql':
                self.connection = pymysql.connect(
                    host=self.config['host'],
                    port=self.config.get('port', 3306),
                    user=self.config['user'],
                    [REDACTED_SECRET]'password'],
                    database=self.config['database'],
                    charset='utf8mb4',
                    cursorclass=pymysql.cursors.DictCursor
                )
                print(f"Connected to MySQL database: {self.config['database']}")
            
            elif db_type == 'postgresql':
                import psycopg2
                self.connection = psycopg2.connect(
                    host=self.config['host'],
                    port=self.config.get('port', 5432),
                    user=self.config['user'],
                    [REDACTED_SECRET]'password'],
                    database=self.config['database']
                )
                print(f"Connected to PostgreSQL database: {self.config['database']}")
            
            else:
                raise ValueError(f"Unsupported database type: {db_type}")
                
        except ImportError as e:
            print(f"Error: Required database driver not installed. {e}")
            print("For MySQL: pip install pymysql")
            print("For PostgreSQL: pip install psycopg2-binary")
            raise
    
    def close(self):
        """Close database connection."""
        if self.connection:
            self.connection.close()
            print("Database connection closed")
    
    def execute_query(self, sql: str, params: tuple = None) -> List[Dict]:
        """
        Execute a query and return results.
        
        Args:
            sql: SQL query
            params: Query parameters
            
        Returns:
            List of dictionaries representing rows
        """
        if not self.connection:
            raise RuntimeError("Not connected to database")
        
        with self.connection.cursor() as cursor:
            cursor.execute(sql, params)
            return cursor.fetchall()
    
    def execute_update(self, sql: str, params: tuple = None) -> int:
        """
        Execute an UPDATE/INSERT/DELETE statement.
        
        Args:
            sql: SQL statement
            params: Statement parameters
            
        Returns:
            Number of affected rows
        """
        if not self.connection:
            raise RuntimeError("Not connected to database")
        
        with self.connection.cursor() as cursor:
            affected_rows = cursor.execute(sql, params)
            self.connection.commit()
            return affected_rows
    
    def get_table_schema(self, table_name: str) -> Dict:
        """
        Get schema information for a table.
        
        Args:
            table_name: Name of the table
            
        Returns:
            Dictionary containing table metadata
        """
        db_type = self.config.get('type', 'mysql')
        
        if db_type == 'mysql':
            return self._get_mysql_table_schema(table_name)
        elif db_type == 'postgresql':
            return self._get_postgresql_table_schema(table_name)
        else:
            raise ValueError(f"Unsupported database type: {db_type}")
    
    def _get_mysql_table_schema(self, table_name: str) -> Dict:
        """Get MySQL table schema."""
        # Get column information
        columns_sql = f"""
            SELECT 
                COLUMN_NAME as name,
                COLUMN_TYPE as type,
                IS_NULLABLE as nullable,
                COLUMN_KEY as key_type,
                COLUMN_DEFAULT as default_value,
                EXTRA as extra
            FROM INFORMATION_SCHEMA.COLUMNS
            WHERE TABLE_SCHEMA = %s AND TABLE_NAME = %s
            ORDER BY ORDINAL_POSITION
        """
        
        columns = self.execute_query(
            columns_sql, 
            (self.config['database'], table_name)
        )
        
        # Get index information
        indexes_sql = f"""
            SELECT 
                INDEX_NAME as index_name,
                COLUMN_NAME as column_name,
                NON_UNIQUE as non_unique
            FROM INFORMATION_SCHEMA.STATISTICS
            WHERE TABLE_SCHEMA = %s AND TABLE_NAME = %s
            ORDER BY INDEX_NAME, SEQ_IN_INDEX
        """
        
        indexes = self.execute_query(
            indexes_sql,
            (self.config['database'], table_name)
        )
        
        # Format columns
        formatted_columns = []
        for col in columns:
            formatted_columns.append({
                'name': col['name'],
                'type': col['type'],
                'nullable': col['nullable'] == 'YES',
                'is_primary_key': col['key_type'] == 'PRI',
                'default': col['default_value'],
                'auto_increment': 'auto_increment' in (col['extra'] or '').lower()
            })
        
        return {
            'table_name': table_name,
            'columns': formatted_columns,
            'indexes': indexes
        }
    
    def _get_postgresql_table_schema(self, table_name: str) -> Dict:
        """Get PostgreSQL table schema."""
        # Get column information
        columns_sql = f"""
            SELECT 
                column_name as name,
                data_type as type,
                is_nullable as nullable,
                column_default as default_value,
                character_maximum_length as max_length
            FROM information_schema.columns
            WHERE table_schema = 'public' AND table_name = %s
            ORDER BY ordinal_position
        """
        
        columns = self.execute_query(columns_sql, (table_name,))
        
        # Get primary key
        pk_sql = f"""
            SELECT a.attname as column_name
            FROM pg_index i
            JOIN pg_attribute a ON a.attrelid = i.indrelid AND a.attnum = ANY(i.indkey)
            WHERE i.indrelid = %s::regclass AND i.indisprimary
        """
        
        primary_keys = self.execute_query(pk_sql, (table_name,))
        pk_columns = [pk['column_name'] for pk in primary_keys]
        
        # Format columns
        formatted_columns = []
        for col in columns:
            col_type = col['type']
            if col['max_length']:
                col_type = f"{col_type}({col['max_length']})"
            
            formatted_columns.append({
                'name': col['name'],
                'type': col_type,
                'nullable': col['nullable'] == 'YES',
                'is_primary_key': col['name'] in pk_columns,
                'default': col['default_value']
            })
        
        return {
            'table_name': table_name,
            'columns': formatted_columns
        }
    
    def get_all_tables(self) -> List[str]:
        """Get list of all tables in the database."""
        db_type = self.config.get('type', 'mysql')
        
        if db_type == 'mysql':
            sql = """
                SELECT TABLE_NAME 
                FROM INFORMATION_SCHEMA.TABLES 
                WHERE TABLE_SCHEMA = %s AND TABLE_TYPE = 'BASE TABLE'
            """
            result = self.execute_query(sql, (self.config['database'],))
        
        elif db_type == 'postgresql':
            sql = """
                SELECT tablename 
                FROM pg_tables 
                WHERE schemaname = 'public'
            """
            result = self.execute_query(sql)
        
        return [row['TABLE_NAME' if db_type == 'mysql' else 'tablename'] for row in result]
    
    def explain_query(self, sql: str) -> str:
        """Get execution plan for a query."""
        db_type = self.config.get('type', 'mysql')
        
        if db_type == 'mysql':
            explain_sql = f"EXPLAIN {sql}"
        elif db_type == 'postgresql':
            explain_sql = f"EXPLAIN ANALYZE {sql}"
        
        result = self.execute_query(explain_sql)
        return json.dumps(result, indent=2)


def main():
    parser = argparse.ArgumentParser(description='Database connector utilities')
    parser.add_argument('--config', required=True, help='JSON file containing database config')
    parser.add_argument('--action', required=True, 
                       choices=['tables', 'schema', 'query', 'explain'],
                       help='Action to perform')
    parser.add_argument('--table', help='Table name (for schema action)')
    parser.add_argument('--sql', help='SQL query (for query/explain actions)')
    
    args = parser.parse_args()
    
    # Load config
    with open(args.config, 'r') as f:
        config = json.load(f)
    
    connector = DatabaseConnector(config)
    
    try:
        connector.connect()
        
        if args.action == 'tables':
            tables = connector.get_all_tables()
            print("Tables in database:")
            for table in tables:
                print(f"  - {table}")
        
        elif args.action == 'schema':
            if not args.table:
                print("Error: --table required for schema action")
                return
            
            schema = connector.get_table_schema(args.table)
            print(json.dumps(schema, indent=2))
        
        elif args.action == 'query':
            if not args.sql:
                print("Error: --sql required for query action")
                return
            
            result = connector.execute_query(args.sql)
            print(json.dumps(result, indent=2))
        
        elif args.action == 'explain':
            if not args.sql:
                print("Error: --sql required for explain action")
                return
            
            plan = connector.explain_query(args.sql)
            print(plan)
    
    finally:
        connector.close()


if __name__ == '__main__':
    main()


path: scripts/generate_test_data.py
content: #!/usr/bin/env python3
"""
Test Data Generator
Intelligently generates test data based on table structures and business rules.
"""

import argparse
import json
import random
import re
from datetime import datetime, timedelta
from typing import Dict, List, Any
import hashlib


class TestDataGenerator:
    """Generates intelligent test data based on table schema."""
    
    def __init__(self, config: Dict = None):
        self.config = config or self._default_config()
        self.generated_ids = {}  # Track generated IDs for foreign keys
        
    def _default_config(self) -> Dict:
        """Default test data generation configuration."""
        return {
            'string_patterns': {
                'email': lambda: f"user{random.randint(1, 10000)}@test.com",
                'phone': lambda: f"1{random.randint(3000000000, 9999999999)}",
                'name': lambda: random.choice(['张三', '李四', '王五', '赵六', '钱七']),
                'address': lambda: f"测试地址{random.randint(1, 999)}号",
                'uuid': lambda: hashlib.md5(str(random.random()).encode()).hexdigest()
            },
            'number_ranges': {
                'age': (18, 70),
                'price': (1, 10000),
                'quantity': (1, 100),
                'status': (0, 1)
            },
            'date_ranges': {
                'past_days': 365,
                'future_days': 30
            }
        }
    
    def generate_for_table(self, table_schema: Dict, count: int = 100) -> List[Dict]:
        """
        Generate test data for a table.
        
        Args:
            table_schema: Table metadata including columns
            count: Number of rows to generate
            
        Returns:
            List of dictionaries representing test data rows
        """
        rows = []
        table_name = table_schema.get('table_name', '')
        columns = table_schema.get('columns', [])
        
        for i in range(count):
            row = {}
            for column in columns:
                col_name = column.get('name', '')
                col_type = column.get('type', '').upper()
                is_pk = column.get('is_primary_key', False)
                
                # Generate value for this column
                value = self._generate_column_value(
                    col_name, col_type, is_pk, table_name
                )
                row[col_name] = value
            
            rows.append(row)
        
        return rows
    
    def _generate_column_value(self, col_name: str, col_type: str, 
                               is_pk: bool, table_name: str) -> Any:
        """Generate a value for a specific column."""
        col_lower = col_name.lower()
        
        # Primary key handling
        if is_pk:
            if 'AUTO_INCREMENT' in col_type.upper():
                return None  # Will be auto-generated
        
        # Pattern-based generation
        if 'email' in col_lower:
            return self.config['string_patterns']['email']()
        elif 'phone' in col_lower or 'mobile' in col_lower:
            return self.config['string_patterns']['phone']()
        elif 'name' in col_lower and 'user' in col_lower:
            return self.config['string_patterns']['name']()
        elif 'address' in col_lower:
            return self.config['string_patterns']['address']()
        elif col_lower.endswith('_id') and not is_pk:
            # Foreign key - return a valid reference
            return self._generate_foreign_key_reference(col_lower)
        
        # Type-based generation
        if 'INT' in col_type:
            if 'age' in col_lower:
                return random.randint(*self.config['number_ranges']['age'])
            elif 'status' in col_lower:
                return random.randint(*self.config['number_ranges']['status'])
            elif 'price' in col_lower or 'amount' in col_lower:
                return random.randint(*self.config['number_ranges']['price'])
            else:
                return random.randint(1, 10000)
        
        elif 'VARCHAR' in col_type or 'CHAR' in col_type:
            # Extract length from type
            match = re.search(r'\((\d+)\)', col_type)
            max_length = int(match.group(1)) if match else 255
            
            # Generate appropriate string
            if max_length <= 10:
                return f"TEST{random.randint(1, 999):03d}"
            else:
                return f"测试数据_{random.randint(1, 9999)}"
        
        elif 'TEXT' in col_type:
            return f"测试文本内容_{random.randint(1, 100)}: 这是自动生成的测试数据"
        
        elif 'DATE' in col_type:
            days_ago = random.randint(0, self.config['date_ranges']['past_days'])
            return (datetime.now() - timedelta(days=days_ago)).strftime('%Y-%m-%d')
        
        elif 'DATETIME' in col_type or 'TIMESTAMP' in col_type:
            days_ago = random.randint(0, self.config['date_ranges']['past_days'])
            hours = random.randint(0, 23)
            minutes = random.randint(0, 59)
            dt = datetime.now() - timedelta(days=days_ago, hours=hours, minutes=minutes)
            return dt.strftime('%Y-%m-%d %H:%M:%S')
        
        elif 'DECIMAL' in col_type or 'FLOAT' in col_type or 'DOUBLE' in col_type:
            return round(random.uniform(1, 10000), 2)
        
        elif 'BOOLEAN' in col_type or 'TINYINT' in col_type:
            return random.choice([0, 1])
        
        else:
            # Default: generate a string
            return f"test_value_{random.randint(1, 1000)}"
    
    def _generate_foreign_key_reference(self, col_name: str) -> int:
        """Generate a valid foreign key reference."""
        # Extract table name from column name (e.g., user_id -> user)
        ref_table = col_name.replace('_id', '')
        
        # If we haven't generated IDs for this table, generate some
        if ref_table not in self.generated_ids:
            self.generated_ids[ref_table] = list(range(1, 101))
        
        # Return a random ID from the generated set
        return random.choice(self.generated_ids[ref_table])
    
    def generate_insert_statements(self, table_name: str, rows: List[Dict]) -> List[str]:
        """Generate INSERT SQL statements from test data."""
        statements = []
        
        for row in rows:
            columns = ', '.join(row.keys())
            values = []
            
            for value in row.values():
                if value is None:
                    values.append('NULL')
                elif isinstance(value, str):
                    # Escape quotes
                    escaped = value.replace("'", "''")
                    values.append(f"'{escaped}'")
                elif isinstance(value, (int, float)):
                    values.append(str(value))
                else:
                    values.append(f"'{value}'")
            
            values_str = ', '.join(values)
            sql = f"INSERT INTO {table_name} ({columns}) VALUES ({values_str});"
            statements.append(sql)
        
        return statements
    
    def generate_batch_insert(self, table_name: str, rows: List[Dict], 
                             batch_size: int = 100) -> List[str]:
        """Generate batch INSERT statements for better performance."""
        statements = []
        
        for i in range(0, len(rows), batch_size):
            batch = rows[i:i + batch_size]
            
            if not batch:
                continue
            
            columns = ', '.join(batch[0].keys())
            all_values = []
            
            for row in batch:
                values = []
                for value in row.values():
                    if value is None:
                        values.append('NULL')
                    elif isinstance(value, str):
                        escaped = value.replace("'", "''")
                        values.append(f"'{escaped}'")
                    elif isinstance(value, (int, float)):
                        values.append(str(value))
                    else:
                        values.append(f"'{value}'")
                all_values.append(f"({', '.join(values)})")
            
            values_str = ',\n'.join(all_values)
            sql = f"INSERT INTO {table_name} ({columns}) VALUES\n{values_str};"
            statements.append(sql)
        
        return statements


def main():
    parser = argparse.ArgumentParser(description='Generate test data for database tables')
    parser.add_argument('--schema', required=True, help='JSON file containing table schema')
    parser.add_argument('--count', type=int, default=100, help='Number of rows to generate')
    parser.add_argument('--output', default='sql', choices=['sql', 'json'],
                       help='Output format')
    parser.add_argument('--batch-size', type=int, default=100,
                       help='Batch size for INSERT statements')
    parser.add_argument('--config', help='JSON file containing generation config')
    
    args = parser.parse_args()
    
    # Load table schema
    with open(args.schema, 'r') as f:
        table_schema = json.load(f)
    
    # Load config if provided
    config = None
    if args.config:
        with open(args.config, 'r') as f:
            config = json.load(f)
    
    generator = TestDataGenerator(config)
    rows = generator.generate_for_table(table_schema, args.count)
    
    if args.output == 'json':
        print(json.dumps(rows, indent=2, ensure_ascii=False))
    else:
        table_name = table_schema.get('table_name', 'table')
        statements = generator.generate_batch_insert(
            table_name, rows, args.batch_size
        )
        
        print(f"-- Generated {len(rows)} test records for table '{table_name}'")
        print(f"-- Generated at: {datetime.now().isoformat()}\n")
        
        for stmt in statements:
            print(stmt)
            print()


if __name__ == '__main__':
    main()


path: scripts/analyze_slow_sql.py
content: #!/usr/bin/env python3
"""
Slow SQL Analysis Script
Analyzes SQL statements and execution plans to identify performance issues.
"""

import argparse
import json
import re
from typing import Dict, List, Tuple


class SlowSQLAnalyzer:
    """Analyzes SQL statements for performance issues."""
    
    def __init__(self):
        self.issues = []
        self.recommendations = []
        
    def analyze_sql(self, sql: str, execution_plan: str = None) -> Dict:
        """
        Analyze SQL statement for performance issues.
        
        Args:
            sql: SQL statement to analyze
            execution_plan: Optional execution plan output
            
        Returns:
            Dictionary containing analysis results
        """
        self.issues = []
        self.recommendations = []
        
        # Check for common anti-patterns
        self._check_select_star(sql)
        self._check_where_clause(sql)
        self._check_like_pattern(sql)
        self._check_or_conditions(sql)
        self._check_functions_in_where(sql)
        self._check_distinct(sql)
        
        # Analyze execution plan if provided
        if execution_plan:
            self._analyze_execution_plan(execution_plan)
        
        return {
            'sql': sql,
            'issues': self.issues,
            'recommendations': self.recommendations,
            'severity': self._calculate_severity()
        }
    
    def _check_select_star(self, sql: str):
        """Check for SELECT * usage."""
        if re.search(r'\bSELECT\s+\*', sql, re.IGNORECASE):
            self.issues.append({
                'type': 'SELECT_STAR',
                'message': 'SELECT * retrieves all columns, which may transfer unnecessary data',
                'severity': 'MEDIUM'
            })
            self.recommendations.append({
                'suggestion': 'Specify only required columns in SELECT clause',
                'example': 'Replace SELECT * with SELECT col1, col2, ...'
            })
    
    def _check_where_clause(self, sql: str):
        """Check for missing WHERE clause in SELECT/UPDATE/DELETE."""
        if re.search(r'\b(SELECT|UPDATE|DELETE)\b', sql, re.IGNORECASE):
            if not re.search(r'\bWHERE\b', sql, re.IGNORECASE):
                self.issues.append({
                    'type': 'MISSING_WHERE',
                    'message': 'Statement lacks WHERE clause, may affect entire table',
                    'severity': 'HIGH'
                })
                self.recommendations.append({
                    'suggestion': 'Add WHERE clause to limit affected rows',
                    'example': 'WHERE created_at > DATE_SUB(NOW(), INTERVAL 30 DAY)'
                })
    
    def _check_like_pattern(self, sql: str):
        """Check for inefficient LIKE patterns."""
        if re.search(r"LIKE\s+['\"]%", sql, re.IGNORECASE):
            self.issues.append({
                'type': 'LEADING_WILDCARD',
                'message': 'LIKE with leading wildcard prevents index usage',
                'severity': 'HIGH'
            })
            self.recommendations.append({
                'suggestion': 'Avoid leading wildcard or use full-text search',
                'example': "LIKE 'prefix%' instead of LIKE '%pattern'"
            })
    
    def _check_or_conditions(self, sql: str):
        """Check for OR conditions that may prevent index usage."""
        if re.search(r'\bOR\b', sql, re.IGNORECASE):
            self.issues.append({
                'type': 'OR_CONDITIONS',
                'message': 'OR conditions may prevent effective index usage',
                'severity': 'MEDIUM'
            })
            self.recommendations.append({
                'suggestion': 'Consider using UNION or IN clause instead',
                'example': 'SELECT ... WHERE id IN (1, 2, 3) instead of id=1 OR id=2 OR id=3'
            })
    
    def _check_functions_in_where(self, sql: str):
        """Check for functions in WHERE clause that prevent index usage."""
        if re.search(r'WHERE.*\w+\s*\([^)]*\)', sql, re.IGNORECASE):
            self.issues.append({
                'type': 'FUNCTION_ON_COLUMN',
                'message': 'Function on column in WHERE clause may prevent index usage',
                'severity': 'MEDIUM'
            })
            self.recommendations.append({
                'suggestion': 'Rewrite condition to avoid function on indexed column',
                'example': 'WHERE date_col >= "2024-01-01" instead of WHERE DATE(date_col) = "2024-01-01"'
            })
    
    def _check_distinct(self, sql: str):
        """Check for DISTINCT that may indicate schema issues."""
        if re.search(r'\bDISTINCT\b', sql, re.IGNORECASE):
            self.issues.append({
                'type': 'DISTINCT_USAGE',
                'message': 'DISTINCT may indicate duplicate data or missing JOIN conditions',
                'severity': 'LOW'
            })
            self.recommendations.append({
                'suggestion': 'Verify if DISTINCT is necessary, consider GROUP BY or improving JOIN logic',
                'example': 'Review schema and query logic to eliminate duplicates at source'
            })
    
    def _analyze_execution_plan(self, plan: str):
        """Analyze execution plan for performance issues."""
        # Check for full table scan
        if re.search(r'(type:\s*ALL|FULL TABLE SCAN|Seq Scan)', plan, re.IGNORECASE):
            self.issues.append({
                'type': 'FULL_TABLE_SCAN',
                'message': 'Query performs full table scan - no suitable index found',
                'severity': 'HIGH'
            })
            self.recommendations.append({
                'suggestion': 'Add index on columns used in WHERE/JOIN clauses',
                'example': 'CREATE INDEX idx_table_column ON table_name(column_name)'
            })
        
        # Check for temporary tables
        if re.search(r'Using temporary', plan, re.IGNORECASE):
            self.issues.append({
                'type': 'TEMPORARY_TABLE',
                'message': 'Query creates temporary table',
                'severity': 'MEDIUM'
            })
            self.recommendations.append({
                'suggestion': 'Consider optimizing GROUP BY or ORDER BY clauses',
                'example': 'Add composite index on GROUP BY/ORDER BY columns'
            })
        
        # Check for filesort
        if re.search(r'Using filesort', plan, re.IGNORECASE):
            self.issues.append({
                'type': 'FILESORT',
                'message': 'Query requires filesort operation',
                'severity': 'MEDIUM'
            })
            self.recommendations.append({
                'suggestion': 'Add index on ORDER BY columns',
                'example': 'CREATE INDEX idx_order ON table_name(order_column)'
            })
    
    def _calculate_severity(self) -> str:
        """Calculate overall severity based on issues found."""
        if not self.issues:
            return 'NONE'
        
        severity_scores = {'HIGH': 3, 'MEDIUM': 2, 'LOW': 1}
        max_severity = max(severity_scores.get(issue['severity'], 0) for issue in self.issues)
        
        if max_severity >= 3:
            return 'HIGH'
        elif max_severity >= 2:
            return 'MEDIUM'
        else:
            return 'LOW'
    
    def generate_index_suggestions(self, table_name: str, columns: List[str]) -> List[str]:
        """Generate index creation suggestions."""
        suggestions = []
        
        for col in columns:
            suggestions.append(
                f"CREATE INDEX idx_{table_name}_{col} ON {table_name}({col});"
            )
        
        return suggestions


def main():
    parser = argparse.ArgumentParser(description='Analyze SQL for performance issues')
    parser.add_argument('--sql', required=True, help='SQL statement to analyze')
    parser.add_argument('--plan', help='Execution plan output (optional)')
    parser.add_argument('--output', default='json', choices=['json', 'text'], 
                       help='Output format')
    
    args = parser.parse_args()
    
    analyzer = SlowSQLAnalyzer()
    result = analyzer.analyze_sql(args.sql, args.plan)
    
    if args.output == 'json':
        print(json.dumps(result, indent=2))
    else:
        print(f"SQL Analysis Report")
        print("=" * 60)
        print(f"SQL: {result['sql']}")
        print(f"Severity: {result['severity']}")
        print("\nIssues Found:")
        for issue in result['issues']:
            print(f"  [{issue['severity']}] {issue['type']}: {issue['message']}")
        print("\nRecommendations:")
        for rec in result['recommendations']:
            print(f"  - {rec['suggestion']}")
            print(f"    Example: {rec['example']}")


if __name__ == '__main__':
    main()


path: scripts/generate_report.py
content: #!/usr/bin/env python3
"""
Excel Report Generator
Generates Excel reports for database analysis results.
"""

import argparse
import json
from datetime import datetime
from typing import Dict, List


class ReportGenerator:
    """Generates Excel reports for database compliance and analysis."""
    
    def generate_compliance_report(self, results: List[Dict], output_file: str):
        """
        Generate Excel compliance report.
        
        Args:
            results: List of compliance check results
            output_file: Output Excel file path
        """
        try:
            import openpyxl
            from openpyxl.styles import Font, Alignment, PatternFill
            from openpyxl.utils import get_column_letter
        except ImportError:
            print("Error: openpyxl library required. Install with: pip install openpyxl")
            return
        
        wb = openpyxl.Workbook()
        
        # Summary sheet
        ws_summary = wb.active
        ws_summary.title = "Summary"
        
        # Headers
        headers = ['Table Name', 'Compliance Score', 'Violations', 'Warnings', 'Status']
        ws_summary.append(headers)
        
        # Style headers
        for col_num, header in enumerate(headers, 1):
            cell = ws_summary.cell(row=1, column=col_num)
            cell.font = Font(bold=True, color="FFFFFF")
            cell.fill = PatternFill(start_color="4472C4", end_color="4472C4", fill_type="solid")
            cell.alignment = Alignment(horizontal="center")
        
        # Add data
        total_violations = 0
        total_warnings = 0
        
        for result in results:
            violations_count = len(result.get('violations', []))
            warnings_count = len(result.get('warnings', []))
            score = result.get('compliance_score', 0)
            
            total_violations += violations_count
            total_warnings += warnings_count
            
            status = 'PASS' if score >= 80 else 'FAIL'
            
            ws_summary.append([
                result.get('table_name', ''),
                score,
                violations_count,
                warnings_count,
                status
            ])
        
        # Add summary row
        ws_summary.append([])
        ws_summary.append(['Total', '', total_violations, total_warnings])
        
        # Adjust column widths
        for col_num in range(1, len(headers) + 1):
            ws_summary.column_dimensions[get_column_letter(col_num)].width = 20
        
        # Details sheet
        ws_details = wb.create_sheet("Details")
        
        detail_headers = ['Table Name', 'Issue Type', 'Severity', 'Message', 'Suggestion']
        ws_details.append(detail_headers)
        
        # Style detail headers
        for col_num, header in enumerate(detail_headers, 1):
            cell = ws_details.cell(row=1, column=col_num)
            cell.font = Font(bold=True, color="FFFFFF")
            cell.fill = PatternFill(start_color="70AD47", end_color="70AD47", fill_type="solid")
            cell.alignment = Alignment(horizontal="center")
        
        # Add violations and warnings
        for result in results:
            table_name = result.get('table_name', '')
            
            for violation in result.get('violations', []):
                ws_details.append([
                    table_name,
                    violation.get('type', ''),
                    violation.get('severity', ''),
                    violation.get('message', ''),
                    violation.get('suggestion', '')
                ])
            
            for warning in result.get('warnings', []):
                ws_details.append([
                    table_name,
                    warning.get('type', ''),
                    warning.get('severity', ''),
                    warning.get('message', ''),
                    warning.get('suggestion', '')
                ])
        
        # Adjust column widths
        for col_num in range(1, len(detail_headers) + 1):
            ws_details.column_dimensions[get_column_letter(col_num)].width = 30
        
        # Save workbook
        wb.save(output_file)
        print(f"Report generated: {output_file}")
    
    def generate_sql_analysis_report(self, results: List[Dict], output_file: str):
        """
        Generate Excel report for SQL analysis.
        
        Args:
            results: List of SQL analysis results
            output_file: Output Excel file path
        """
        try:
            import openpyxl
            from openpyxl.styles import Font, Alignment, PatternFill
            from openpyxl.utils import get_column_letter
        except ImportError:
            print("Error: openpyxl library required. Install with: pip install openpyxl")
            return
        
        wb = openpyxl.Workbook()
        ws = wb.active
        ws.title = "SQL Analysis"
        
        # Headers
        headers = ['SQL', 'Severity', 'Issue Type', 'Issue Message', 'Recommendation']
        ws.append(headers)
        
        # Style headers
        for col_num, header in enumerate(headers, 1):
            cell = ws.cell(row=1, column=col_num)
            cell.font = Font(bold=True, color="FFFFFF")
            cell.fill = PatternFill(start_color="FF6B6B", end_color="FF6B6B", fill_type="solid")
            cell.alignment = Alignment(horizontal="center")
        
        # Add data
        for result in results:
            sql = result.get('sql', '')
            severity = result.get('severity', '')
            
            for issue in result.get('issues', []):
                ws.append([
                    sql[:100] + '...' if len(sql) > 100 else sql,
                    issue.get('severity', ''),
                    issue.get('type', ''),
                    issue.get('message', ''),
                    ''  # Recommendations in separate column
                ])
        
        # Adjust column widths
        ws.column_dimensions['A'].width = 50
        ws.column_dimensions['B'].width = 15
        ws.column_dimensions['C'].width = 25
        ws.column_dimensions['D'].width = 50
        ws.column_dimensions['E'].width = 50
        
        # Save workbook
        wb.save(output_file)
        print(f"SQL analysis report generated: {output_file}")


def main():
    parser = argparse.ArgumentParser(description='Generate Excel reports')
    parser.add_argument('--input', required=True, help='JSON file containing analysis results')
    parser.add_argument('--output', required=True, help='Output Excel file path')
    parser.add_argument('--type', choices=['compliance', 'sql_analysis'], 
                       default='compliance', help='Report type')
    
    args = parser.parse_args()
    
    # Load results
    with open(args.input, 'r') as f:
        results = json.load(f)
    
    generator = ReportGenerator()
    
    if args.type == 'compliance':
        generator.generate_compliance_report(results, args.output)
    elif args.type == 'sql_analysis':
        generator.generate_sql_analysis_report(results, args.output)


if __name__ == '__main__':
    main()


path: scripts/validate_data_correction.py
content: #!/usr/bin/env python3
"""
Data Correction Safety Validator
Validates UPDATE/DELETE operations to prevent unsafe data modifications.
"""

import argparse
import json
import re
from datetime import datetime
from typing import Dict, List, Tuple


class DataCorrectionValidator:
    """Validates data modification operations for safety."""
    
    def __init__(self):
        self.risks = []
        self.is_safe = True
        
    def validate_sql(self, sql: str, affected_rows_estimate: int = None) -> Dict:
        """
        Validate UPDATE/DELETE SQL for safety risks.
        
        Args:
            sql: SQL statement to validate
            affected_rows_estimate: Optional estimate of affected rows
            
        Returns:
            Dictionary containing validation results
        """
        self.risks = []
        self.is_safe = True
        
        sql_upper = sql.upper()
        
        # Determine operation type
        operation = self._get_operation_type(sql_upper)
        
        # Perform safety checks
        self._check_where_clause(sql, operation)
        self._check_limit_clause(sql, operation)
        self._check_dangerous_patterns(sql, operation)
        self._check_primary_key_modification(sql, operation)
        
        # Check affected rows estimate if provided
        if affected_rows_estimate is not None:
            self._check_affected_rows(affected_rows_estimate)
        
        return {
            'sql': sql,
            'operation': operation,
            'is_safe': self.is_safe,
            'risks': self.risks,
            'requires_backup': len(self.risks) > 0,
            'timestamp': datetime.now().isoformat()
        }
    
    def _get_operation_type(self, sql_upper: str) -> str:
        """Determine the type of SQL operation."""
        if 'UPDATE' in sql_upper:
            return 'UPDATE'
        elif 'DELETE' in sql_upper:
            return 'DELETE'
        elif 'INSERT' in sql_upper:
            return 'INSERT'
        return 'UNKNOWN'
    
    def _check_where_clause(self, sql: str, operation: str):
        """Check for missing WHERE clause."""
        if operation in ['UPDATE', 'DELETE']:
            if not re.search(r'\bWHERE\b', sql, re.IGNORECASE):
                self.risks.append({
                    'type': 'MISSING_WHERE',
                    'severity': 'CRITICAL',
                    'message': f'{operation} without WHERE clause affects entire table',
                    'recommendation': 'Add WHERE clause to limit affected rows'
                })
                self.is_safe = False
    
    def _check_limit_clause(self, sql: str, operation: str):
        """Check for missing LIMIT clause on DELETE."""
        if operation == 'DELETE':
            if not re.search(r'\bLIMIT\b', sql, re.IGNORECASE):
                self.risks.append({
                    'type': 'NO_LIMIT',
                    'severity': 'HIGH',
                    'message': 'DELETE without LIMIT may affect more rows than expected',
                    'recommendation': 'Consider adding LIMIT clause or use transactions'
                })
    
    def _check_dangerous_patterns(self, sql: str, operation: str):
        """Check for dangerous SQL patterns."""
        sql_upper = sql.upper()
        
        # Check for TRUE conditions
        if re.search(r'\bWHERE\s+(TRUE|1\s*=\s*1)\b', sql, re.IGNORECASE):
            self.risks.append({
                'type': 'ALWAYS_TRUE_WHERE',
                'severity': 'CRITICAL',
                'message': 'WHERE clause is always true - affects all rows',
                'recommendation': 'Review WHERE clause logic'
            })
            self.is_safe = False
        
        # Check for OR 1=1 injection pattern
        if re.search(r'\bOR\s+1\s*=\s*1\b', sql, re.IGNORECASE):
            self.risks.append({
                'type': 'SQL_INJECTION_PATTERN',
                'severity': 'CRITICAL',
                'message': 'Detected potential SQL injection pattern: OR 1=1',
                'recommendation': 'Review query construction'
            })
            self.is_safe = False
        
        # Check for UPDATE with id in SET clause
        if operation == 'UPDATE':
            if re.search(r'\bSET\b.*\bid\s*=', sql, re.IGNORECASE):
                self.risks.append({
                    'type': 'UPDATING_PRIMARY_KEY',
                    'severity': 'HIGH',
                    'message': 'Attempting to update primary key column',
                    'recommendation': 'Primary key updates may break referential integrity'
                })
    
    def _check_primary_key_modification(self, sql: str, operation: str):
        """Check if operation modifies primary key or unique constraints."""
        sql_upper = sql.upper()
        
        # Check for deleting by non-primary key
        if operation == 'DELETE':
            # If WHERE doesn't contain id or primary key columns
            if not re.search(r'\bWHERE\b.*\bid\b', sql, re.IGNORECASE):
                self.risks.append({
                    'type': 'DELETE_BY_NON_PK',
                    'severity': 'MEDIUM',
                    'message': 'DELETE by non-primary key may be slower on large tables',
                    'recommendation': 'Consider deleting by primary key for better performance'
                })
    
    def _check_affected_rows(self, affected_rows: int):
        """Check if affected rows count exceeds safety threshold."""
        WARNING_THRESHOLD = 1000
        CRITICAL_THRESHOLD = 10000
        
        if affected_rows >= CRITICAL_THRESHOLD:
            self.risks.append({
                'type': 'MASS_OPERATION',
                'severity': 'CRITICAL',
                'message': f'Operation affects {affected_rows} rows (>= {CRITICAL_THRESHOLD})',
                'recommendation': 'Requires explicit confirmation and approval'
            })
            self.is_safe = False
        elif affected_rows >= WARNING_THRESHOLD:
            self.risks.append({
                'type': 'LARGE_OPERATION',
                'severity': 'HIGH',
                'message': f'Operation affects {affected_rows} rows (>= {WARNING_THRESHOLD})',
                'recommendation': 'Consider batch processing with LIMIT'
            })
    
    def generate_backup_sql(self, sql: str, table_name: str = None) -> str:
        """Generate backup SQL statement."""
        # Extract table name from SQL if not provided
        if not table_name:
            table_match = re.search(r'\bFROM\s+(\w+)', sql, re.IGNORECASE)
            if table_match:
                table_name = table_match.group(1)
            else:
                table_name = 'unknown_table'
        
        # Extract WHERE clause
        where_match = re.search(r'\bWHERE\s+(.+)', sql, re.IGNORECASE)
        where_clause = where_match.group(1) if where_match else '1=1'
        
        # Generate backup table name
        timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
        backup_table = f"{table_name}_backup_{timestamp}"
        
        return f"CREATE TABLE {backup_table} AS SELECT * FROM {table_name} WHERE {where_clause};"
    
    def generate_rollback_sql(self, sql: str, table_name: str = None) -> List[str]:
        """Generate rollback SQL statements."""
        rollback_statements = []
        
        # Extract table name
        if not table_name:
            table_match = re.search(r'\bFROM\s+(\w+)', sql, re.IGNORECASE)
            table_name = table_match.group(1) if table_match else 'unknown_table'
        
        timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
        backup_table = f"{table_name}_backup_{timestamp}"
        
        # Generate rollback based on operation type
        if 'DELETE' in sql.upper():
            rollback_statements.append(
                f"-- Rollback: Restore deleted records\n"
                f"INSERT INTO {table_name} SELECT * FROM {backup_table};"
            )
        elif 'UPDATE' in sql.upper():
            # This is simplified - real rollback would need to know original values
            rollback_statements.append(
                f"-- Rollback: Restore updated records\n"
                f"-- Requires backup table with original values\n"
                f"UPDATE {table_name} t SET ... FROM {backup_table} b WHERE t.id = b.id;"
            )
        
        return rollback_statements


def main():
    parser = argparse.ArgumentParser(description='Validate data correction SQL for safety')
    parser.add_argument('--sql', required=True, help='SQL statement to validate')
    parser.add_argument('--affected-rows', type=int, help='Estimated affected rows (optional)')
    parser.add_argument('--output', default='json', choices=['json', 'text'],
                       help='Output format')
    
    args = parser.parse_args()
    
    validator = DataCorrectionValidator()
    result = validator.validate_sql(args.sql, args.affected_rows)
    
    if args.output == 'json':
        print(json.dumps(result, indent=2))
    else:
        print(f"Data Correction Safety Report")
        print("=" * 60)
        print(f"Operation: {result['operation']}")
        print(f"Safe: {'YES' if result['is_safe'] else 'NO'}")
        print(f"\nRisks Identified: {len(result['risks'])}")
        for risk in result['risks']:
            print(f"  [{risk['severity']}] {risk['type']}")
            print(f"    {risk['message']}")
            print(f"    Recommendation: {risk['recommendation']}")
        
        if result['requires_backup']:
            print("\n⚠️  BACKUP REQUIRED")
            backup_sql = validator.generate_backup_sql(args.sql)
            print(f"\nBackup SQL:\n{backup_sql}")


if __name__ == '__main__':
    main()


path: scripts/check_schema_compliance.py
content: #!/usr/bin/env python3
"""
Schema Compliance Checker
Validates database table structures against best practices and standards.
"""

import argparse
import json
import re
from typing import Dict, List


class SchemaComplianceChecker:
    """Validates table structures against database standards."""
    
    def __init__(self, rules_config: Dict = None):
        self.violations = []
        self.warnings = []
        self.rules = rules_config or self._default_rules()
        
    def _default_rules(self) -> Dict:
        """Default compliance rules."""
        return {
            'naming': {
                'table_pattern': r'^[a-z][a-z0-9_]*$',
                'column_pattern': r'^[a-z][a-z0-9_]*$',
                'reserved_keywords': ['order', 'group', 'select', 'index', 'key', 'value']
            },
            'primary_key': {
                'required': True,
                'naming_pattern': r'^id$|^[a-z_]+_id$'
            },
            'data_types': {
                'string_max_length': 255,
                'text_for_long_strings': True
            },
            'defaults': {
                'not_null_preferred': True,
                'explicit_defaults_required': False
            }
        }
    
    def check_table(self, table_schema: Dict) -> Dict:
        """
        Check table schema for compliance issues.
        
        Args:
            table_schema: Dictionary containing table metadata
                {
                    'table_name': str,
                    'columns': [
                        {
                            'name': str,
                            'type': str,
                            'nullable': bool,
                            'default': any,
                            'is_primary_key': bool
                        }
                    ],
                    'indexes': [...]
                }
        
        Returns:
            Dictionary with compliance results
        """
        self.violations = []
        self.warnings = []
        
        table_name = table_schema.get('table_name', '')
        columns = table_schema.get('columns', [])
        
        # Check table naming
        self._check_table_naming(table_name)
        
        # Check primary key
        self._check_primary_key(columns)
        
        # Check each column
        for column in columns:
            self._check_column_naming(table_name, column)
            self._check_column_type(table_name, column)
            self._check_column_nullable(table_name, column)
        
        return {
            'table_name': table_name,
            'violations': self.violations,
            'warnings': self.warnings,
            'compliance_score': self._calculate_compliance_score()
        }
    
    def _check_table_naming(self, table_name: str):
        """Check if table name follows naming conventions."""
        pattern = self.rules['naming']['table_pattern']
        
        if not re.match(pattern, table_name):
            self.violations.append({
                'type': 'TABLE_NAMING',
                'message': f"Table name '{table_name}' doesn't follow naming convention",
                'severity': 'HIGH',
                'suggestion': f"Use lowercase letters, numbers, and underscores only"
            })
        
        # Check reserved keywords
        if table_name.lower() in self.rules['naming']['reserved_keywords']:
            self.violations.append({
                'type': 'RESERVED_KEYWORD',
                'message': f"Table name '{table_name}' is a reserved keyword",
                'severity': 'HIGH',
                'suggestion': f"Prefix with 'tbl_' or use alternative name"
            })
    
    def _check_primary_key(self, columns: List[Dict]):
        """Check if table has proper primary key."""
        has_pk = any(col.get('is_primary_key', False) for col in columns)
        
        if not has_pk:
            self.violations.append({
                'type': 'MISSING_PRIMARY_KEY',
                'message': 'Table lacks primary key',
                'severity': 'HIGH',
                'suggestion': 'Add an auto-increment ID column as primary key'
            })
    
    def _check_column_naming(self, table_name: str, column: Dict):
        """Check column naming conventions."""
        col_name = column.get('name', '')
        pattern = self.rules['naming']['column_pattern']
        
        if not re.match(pattern, col_name):
            self.violations.append({
                'type': 'COLUMN_NAMING',
                'message': f"Column '{col_name}' in table '{table_name}' doesn't follow naming convention",
                'severity': 'MEDIUM',
                'suggestion': 'Use lowercase letters, numbers, and underscores only'
            })
        
        # Check reserved keywords
        if col_name.lower() in self.rules['naming']['reserved_keywords']:
            self.violations.append({
                'type': 'RESERVED_KEYWORD',
                'message': f"Column '{col_name}' is a reserved keyword",
                'severity': 'HIGH',
                'suggestion': f"Use alternative name like '{col_name}_value'"
            })
    
    def _check_column_type(self, table_name: str, column: Dict):
        """Check column data type appropriateness."""
        col_name = column.get('name', '')
        col_type = column.get('type', '').upper()
        
        # Check for VARCHAR length
        if 'VARCHAR' in col_type or 'CHAR' in col_type:
            match = re.search(r'\((\d+)\)', col_type)
            if match:
                length = int(match.group(1))
                if length > self.rules['data_types']['string_max_length']:
                    self.warnings.append({
                        'type': 'LONG_VARCHAR',
                        'message': f"Column '{col_name}' has VARCHAR({length}), consider using TEXT",
                        'severity': 'LOW',
                        'suggestion': 'Use TEXT type for strings longer than 255 characters'
                    })
    
    def _check_column_nullable(self, table_name: str, column: Dict):
        """Check nullable column practices."""
        col_name = column.get('name', '')
        nullable = column.get('nullable', True)
        default = column.get('default')
        
        # Recommend NOT NULL for critical columns
        if nullable and col_name.endswith(('_id', '_time', '_date', '_status')):
            self.warnings.append({
                'type': 'NULLABLE_COLUMN',
                'message': f"Column '{col_name}' should typically be NOT NULL",
                'severity': 'MEDIUM',
                'suggestion': 'Consider adding NOT NULL constraint with default value'
            })
    
    def _calculate_compliance_score(self) -> int:
        """Calculate compliance score (0-100)."""
        if not self.violations and not self.warnings:
            return 100
        
        # Each violation reduces score by 20, each warning by 5
        score = 100 - (len(self.violations) * 20) - (len(self.warnings) * 5)
        return max(0, score)
    
    def generate_fix_sql(self, violations: List[Dict], table_name: str) -> List[str]:
        """Generate SQL statements to fix violations."""
        fix_statements = []
        
        for violation in violations:
            if violation['type'] == 'MISSING_PRIMARY_KEY':
                fix_statements.append(
                    f"ALTER TABLE {table_name} ADD COLUMN id INT AUTO_INCREMENT PRIMARY KEY FIRST;"
                )
        
        return fix_statements


def main():
    parser = argparse.ArgumentParser(description='Check database schema compliance')
    parser.add_argument('--schema', required=True, help='JSON file containing table schema')
    parser.add_argument('--rules', help='JSON file containing custom rules (optional)')
    parser.add_argument('--output', default='json', choices=['json', 'text'],
                       help='Output format')
    
    args = parser.parse_args()
    
    # Load table schema
    with open(args.schema, 'r') as f:
        table_schema = json.load(f)
    
    # Load custom rules if provided
    rules = None
    if args.rules:
        with open(args.rules, 'r') as f:
            rules = json.load(f)
    
    checker = SchemaComplianceChecker(rules)
    result = checker.check_table(table_schema)
    
    if args.output == 'json':
        print(json.dumps(result, indent=2))
    else:
        print(f"Schema Compliance Report for '{result['table_name']}'")
        print("=" * 60)
        print(f"Compliance Score: {result['compliance_score']}/100")
        print("\nViolations:")
        for violation in result['violations']:
            print(f"  [{violation['severity']}] {violation['type']}")
            print(f"    {violation['message']}")
            print(f"    Suggestion: {violation['suggestion']}")
        print("\nWarnings:")
        for warning in result['warnings']:
            print(f"  [{warning['severity']}] {warning['type']}")
            print(f"    {warning['message']}")


if __name__ == '__main__':
    main()


path: assets/compliance_rules.json
content: {
  "naming": {
    "table_pattern": "^[a-z][a-z0-9_]*$",
    "column_pattern": "^[a-z][a-z0-9_]*$",
    "reserved_keywords": [
      "order", "group", "select", "index", "key", "value", "user",
      "table", "column", "database", "schema", "grant", "revoke"
    ]
  },
  "primary_key": {
    "required": true,
    "naming_pattern": "^id$|^[a-z_]+_id$",
    "auto_increment_preferred": true,
    "type_preference": ["BIGINT", "INT"]
  },
  "foreign_keys": {
    "required": true,
    "naming_pattern": "^[a-z_]+_id$",
    "index_required": true,
    "on_delete_default": "RESTRICT",
    "on_update_default": "CASCADE"
  },
  "data_types": {
    "string_max_length": 255,
    "text_for_long_strings": true,
    "decimal_for_money": true,
    "timestamp_for_datetime": true
  },
  "defaults": {
    "not_null_preferred": true,
    "explicit_defaults_required": false,
    "created_at_default": true,
    "updated_at_default": true,
    "soft_delete_column": "deleted_at"
  },
  "indexes": {
    "foreign_key_index_required": true,
    "unique_constraint_naming": "uk_{table}_{columns}",
    "regular_index_naming": "idx_{table}_{columns}",
    "composite_index_limit": 5
  },
  "validation_rules": [
    {
      "rule": "missing_primary_key",
      "severity": "HIGH",
      "message": "Table lacks primary key",
      "auto_fix": true
    },
    {
      "rule": "reserved_keyword_table",
      "severity": "HIGH",
      "message": "Table name uses reserved keyword",
      "auto_fix": false
    },
    {
      "rule": "reserved_keyword_column",
      "severity": "HIGH",
      "message": "Column name uses reserved keyword",
      "auto_fix": false
    },
    {
      "rule": "invalid_table_naming",
      "severity": "MEDIUM",
      "message": "Table name doesn't follow naming convention",
      "auto_fix": false
    },
    {
      "rule": "invalid_column_naming",
      "severity": "MEDIUM",
      "message": "Column name doesn't follow naming convention",
      "auto_fix": false
    },
    {
      "rule": "missing_foreign_key_index",
      "severity": "MEDIUM",
      "message": "Foreign key column lacks index",
      "auto_fix": true
    },
    {
      "rule": "nullable_critical_field",
      "severity": "LOW",
      "message": "Critical field should be NOT NULL",
      "auto_fix": false
    }
  ]
}


path: assets/test_data_config.json
content: {
  "string_patterns": {
    "email": {
      "pattern": "user{n}@test.com",
      "n_range": [1, 10000]
    },
    "phone": {
      "pattern": "1{prefix}{number}",
      "prefixes": ["3", "5", "7", "8", "9"],
      "number_length": 9
    },
    "name": {
      "values": ["张三", "李四", "王五", "赵六", "钱七", "孙八", "周九", "吴十"]
    },
    "address": {
      "pattern": "{city}{district}{street}{number}号",
      "cities": ["北京市", "上海市", "广州市", "深圳市", "杭州市"],
      "districts": ["朝阳区", "浦东新区", "天河区", "南山区", "西湖区"],
      "streets": ["建国路", "南京路", "中山大道", "科技园路", "文三路"],
      "number_range": [1, 999]
    },
    "url": {
      "domains": ["test.example.com", "staging.company.com", "localhost:8080"],
      "protocols": ["http", "https"]
    }
  },
  "number_ranges": {
    "age": {"min": 18, "max": 70},
    "price": {"min": 1, "max": 10000, "decimals": 2},
    "quantity": {"min": 1, "max": 100},
    "stock": {"min": 0, "max": 1000},
    "status": {"min": 0, "max": 1},
    "rating": {"min": 1, "max": 5},
    "score": {"min": 0, "max": 100}
  },
  "date_ranges": {
    "past_days": 365,
    "future_days": 30,
    "business_hours": {"start": 9, "end": 18}
  },
  "boolean_weights": {
    "is_active": {"true": 80, "false": 20},
    "is_verified": {"true": 70, "false": 30},
    "is_deleted": {"true": 5, "false": 95},
    "is_premium": {"true": 20, "false": 80}
  },
  "business_domains": {
    "user": {
      "username_pattern": "user{n}",
      "username_range": [1000, 9999],
      "gender_values": ["M", "F", "Other"],
      "status_values": ["active", "inactive", "banned"]
    },
    "product": {
      "sku_pattern": "SKU{n}",
      "sku_range": [100000, 999999],
      "status_values": ["active", "inactive", "out_of_stock"]
    },
    "order": {
      "order_number_pattern": "ORD{timestamp}{n}",
      "status_values": ["pending", "paid", "shipped", "completed", "cancelled"],
      "payment_methods": ["credit_card", "paypal", "bank_transfer", "alipay", "wechat"]
    },
    "article": {
      "categories": ["tech", "life", "news", "entertainment", "business"],
      "tags": ["python", "java", "database", "web", "mobile", "ai", "cloud"],
      "status_values": ["draft", "published", "archived"]
    },
    "transaction": {
      "transaction_id_pattern": "TXN{timestamp}{n}",
      "types": ["deposit", "withdrawal", "transfer", "payment"],
      "status_values": ["pending", "completed", "failed", "cancelled"],
      "currencies": ["CNY", "USD", "EUR"]
    }
  },
  "generation_settings": {
    "batch_size": 100,
    "null_probability": 0.15,
    "unicode_enabled": true,
    "emoji_enabled": true,
    "special_chars_enabled": true,
    "deterministic_seed": null
  },
  "volume_guidelines": {
    "development": {
      "users": "100-1000",
      "products": "500-2000",
      "orders": "1000-5000",
      "comments": "500-2000"
    },
    "staging": {
      "users": "1000-10000",
      "products": "2000-10000",
      "orders": "10000-50000",
      "comments": "5000-20000"
    },
    "performance": {
      "users": "100000+",
      "products": "50000+",
      "orders": "1000000+",
      "comments": "100000+"
    }
  }
}


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)