Skill: gdp6539/data-analysis-skill (version 1.0.0)

--- name: data-analysis-skill description: 数据分析技能包 - 自动抓取、清洗、可视化、生成报告。适合数据分析师、运营人员，告别 Excel 手工操作。 version: 1.0.0 author: AI-Company tags: [data, analysis, visualization, report, automation, 数据分析, 可视化] license: MIT --- # 数据分析技能包 DataMaster Pro ## 一句话介绍 自动抓取数据、智能清洗、一键可视化、生成专业报告。 ## 解决什么问题？ - 数据分散：多个来源手动复制粘贴 → 一键抓取聚合 - 清洗耗时：Excel 公式搞半天 → 自动识别并清洗 - 图表难看：不会做可视化 → 一键生成专业图表 - 报告难写：不会写分析 → AI 自动生成洞察 ## 功能清单 - 🌐 数据抓取：网页爬取、API 对接、数据库连接 - 🧹 数据清洗：去重、补缺、格式化、异常检测 - 📊 可视化：折线图、柱状图、饼图、热力图、散点图 - 📝 报告生成：自动生成数据分析报告（Markdown/HTML/PDF） - 🔄 定时任务：支持定时抓取和分析 ## 快速开始 ### 安装 ```bash # 进入技能包目录 cd data-analysis-skill npm install ``` ### 使用命令 ```bash # 网页数据抓取 /data-fetch <URL> --selector "table.data" # API 数据获取 /data-api <API_URL> --method GET --output data.json # 数据清洗 /data-clean data.json --rules clean-rules.json # 生成可视化图表 /data-viz data.csv --type bar --title "销售趋势" # 完整分析报告 /data-report data.csv --template business --output report.md ``` ### 配置示例 ```json { "sources": [ { "type": "web", "url": "https://example.com/data", "selector": "table tbody tr", "schedule": "0 9 * * *" }, { "type": "api", "url": "https://api.example.com/v1/data", "headers": { "Authorization": "Bearer TOKEN" } } ], "cleaning": { "removeDuplicates": true, "fillMissing": "mean", "normalizeColumns": ["price", "quantity"] }, "visualization": { "defaultChartType": "bar", "colors": ["#4CAF50", "#2196F3", "#FF9800"], "width": 800, "height": 400 } } ``` ## 文件结构 ``` data-analysis-skill/ ├── SKILL.md # 技能定义（本文件） ├── README.md # 产品说明 ├── TUTORIAL.md # 傻瓜式教程 ├── install.bat # 一键安装 ├── run.bat # 一键运行 ├── config.json # 配置示例 ├── package.json # 依赖管理 ├── scripts/ # 核心代码 │ ├── data-fetch.js # 数据抓取 │ ├── data-clean.js # 数据清洗 │ ├── data-viz.js # 可视化生成 │ └── data-report.js # 报告生成 ├── templates/ # 报告模板 │ ├── business.md # 商业报告模板 │ ├── technical.md # 技术报告模板 │ └── weekly.md # 周报模板 └── examples/ # 示例文件 ├── sample-data.csv └── sample-report.md ``` ## 核心脚本说明 ### data-fetch.js - 数据抓取 支持三种数据源： - **网页抓取**：CSS 选择器提取表格/列表数据 - **API 请求**：GET/POST 请求，支持认证 - **数据库**：MySQL/PostgreSQL/MongoDB 连接 ### data-clean.js - 数据清洗 - 去重、去空值 - 格式标准化（日期、数字、文本） - 异常值检测与处理 - 数据类型转换 ### data-viz.js - 可视化 - 自动推荐最佳图表类型 - 支持自定义样式 - 输出 SVG/PNG/HTML ### data-report.js - 报告生成 - AI 驱动的数据洞察 - 多种模板可选 - 支持导出 Markdown/HTML/PDF ## 适用人群 - 数据分析师 - 运营人员 - 市场研究员 - 产品经理 - 财务人员 ## 价格 - 基础版：¥99（网页抓取+基础清洗+5种图表） - 进阶版：¥199（API对接+高级清洗+10种图表+报告模板） - 专业版：¥299（数据库连接+AI洞察+定制模板+1对1指导） --- *开发者：AI-Company* *联系：通过ClawHub*

path: README.md
content: # DataMaster Pro - 数据分析技能包

> 自动抓取、智能清洗、一键可视化、生成专业报告

## 🎯 产品定位

专为数据分析师、运营人员打造的自动化数据分析工具包。告别重复的 Excel 操作，让数据分析从小时级变成分钟级。

## ✨ 核心功能

### 1. 数据抓取 (Data Fetch)

```bash
# 网页表格抓取
node scripts/data-fetch.js --url "https://example.com/data" --selector "table.data"

# API 数据获取
node scripts/data-fetch.js --api "https://api.example.com/v1/data" --method GET

# 数据库连接
node scripts/data-fetch.js --db mysql://user:pass@host/db --query "SELECT * FROM sales"
```

支持的数据源：
- ✅ 网页表格/列表
- ✅ REST API
- ✅ MySQL / PostgreSQL / MongoDB
- ✅ CSV / JSON / Excel 文件

### 2. 数据清洗 (Data Clean)

```bash
# 自动清洗
node scripts/data-clean.js --input data.json --output clean.csv

# 使用清洗规则
node scripts/data-clean.js --input data.json --rules clean-rules.json
```

清洗功能：
- 去重（按指定列）
- 空值处理（删除/填充/插值）
- 格式标准化（日期、货币、百分比）
- 异常值检测（IQR/Z-score）
- 数据类型转换

### 3. 数据可视化 (Data Viz)

```bash
# 生成柱状图
node scripts/data-viz.js --input data.csv --type bar --output chart.png

# 生成多图
node scripts/data-viz.js --input data.csv --type line,bar,pie --output charts/
```

支持的图表类型：
- 📊 柱状图（Bar）
- 📈 折线图（Line）
- 🥧 饼图（Pie）
- 📉 面积图（Area）
- 🔵 散点图（Scatter）
- 🔥 热力图（Heatmap）
- 📉 箱线图（Box Plot）

### 4. 报告生成 (Data Report)

```bash
# 生成 Markdown 报告
node scripts/data-report.js --input data.csv --template business --output report.md

# 生成 HTML 报告
node scripts/data-report.js --input data.csv --template weekly --output report.html
```

报告模板：
- `business` - 商业分析报告
- `technical` - 技术数据报告
- `weekly` - 周报模板
- `custom` - 自定义模板

## 📦 安装

### 方式一：一键安装
```bash
install.bat
```

### 方式二：手动安装
```bash
cd data-analysis-skill
npm install
```

## 🚀 快速开始

### 完整流程示例

```bash
# 1. 抓取数据
node scripts/data-fetch.js --url "https://example.com/sales" --selector "table" --output raw.json

# 2. 清洗数据
node scripts/data-clean.js --input raw.json --output clean.csv

# 3. 生成可视化
node scripts/data-viz.js --input clean.csv --type bar,line --output charts/

# 4. 生成报告
node scripts/data-report.js --input clean.csv --template business --output report.md
```

### 一键运行
```bash
run.bat
```

## 📁 文件结构

```
data-analysis-skill/
├── SKILL.md           # 技能定义
├── README.md          # 产品说明（本文件）
├── TUTORIAL.md        # 傻瓜式教程
├── install.bat        # 一键安装
├── run.bat            # 一键运行
├── config.json        # 配置示例
├── package.json       # 依赖管理
├── scripts/           # 核心代码
│   ├── data-fetch.js  # 数据抓取
│   ├── data-clean.js  # 数据清洗
│   ├── data-viz.js    # 可视化生成
│   └── data-report.js # 报告生成
├── templates/         # 报告模板
└── examples/          # 示例文件
```

## ⚙️ 配置说明

编辑 `config.json` 进行配置：

```json
{
  "sources": [
    {
      "type": "web",
      "url": "https://example.com/data",
      "selector": "table tbody tr",
      "schedule": "0 9 * * *"
    }
  ],
  "cleaning": {
    "removeDuplicates": true,
    "fillMissing": "mean",
    "detectOutliers": true
  },
  "visualization": {
    "defaultChartType": "bar",
    "colors": ["#4CAF50", "#2196F3"],
    "width": 800,
    "height": 400
  },
  "report": {
    "template": "business",
    "includeCharts": true,
    "aiInsights": true
  }
}
```

## 🔧 高级用法

### 定时任务
```bash
# 每天早上9点抓取数据
node scripts/data-fetch.js --schedule "0 9 * * *"
```

### 批量处理
```bash
# 批量处理多个URL
node scripts/data-fetch.js --batch urls.txt
```

### 自定义清洗规则
```json
{
  "rules": [
    {"column": "price", "type": "number", "decimals": 2},
    {"column": "date", "type": "date", "format": "YYYY-MM-DD"},
    {"column": "name", "type": "string", "trim": true}
  ]
}
```

## 📊 使用案例

### 案例1：电商销售数据分析
```bash
# 抓取店铺销售数据
node scripts/data-fetch.js --api "https://shop.example.com/api/sales" --output sales.json

# 清洗并分析
node scripts/data-clean.js --input sales.json --output clean.csv
node scripts/data-report.js --input clean.csv --template business
```

### 案例2：竞品监控报告
```bash
# 抓取竞品价格
node scripts/data-fetch.js --url "https://competitor.example.com/prices" --selector ".price-table"

# 生成对比报告
node scripts/data-report.js --input prices.csv --template competitive
```

### 案例3：周报自动化
```bash
# 设置定时任务，每周一生成周报
node scripts/data-report.js --input weekly.csv --template weekly --schedule "0 9 * * 1"
```

## 🛡️ 注意事项

1. **合规使用** - 仅抓取公开、允许抓取的数据
2. **API 限制** - 注意 API 调用频率限制
3. **数据安全** - 敏感数据请使用环境变量
4. **版权问题** - 生成的报告请自行审核

## 💰 价格

| 版本 | 价格 | 功能 |
|------|------|------|
| 基础版 | ¥99 | 网页抓取 + 基础清洗 + 5种图表 |
| 进阶版 | ¥199 | API对接 + 高级清洗 + 10种图表 + 报告模板 |
| 专业版 | ¥299 | 数据库连接 + AI洞察 + 定制模板 + 1对1指导 |

## 🆘 常见问题

**Q: 支持哪些数据格式？**
A: 支持 CSV、JSON、Excel、XML 等常见格式。

**Q: 可以抓取需要登录的网站吗？**
A: 进阶版及以上支持 Cookie 认证和 Headers 设置。

**Q: 报告可以导出什么格式？**
A: 支持 Markdown、HTML、PDF 格式。

**Q: 有使用限制吗？**
A: 无限制，一次购买永久使用。

---

*开发者：AI-Company*
*版本：1.0.0*
*联系：通过ClawHub*

path: scripts/data-clean.js
content: /**
 * DataMaster Pro - 数据清洗脚本
 * 支持去重、补缺、格式化、异常检测
 */

const fs = require('fs');
const path = require('path');

// 解析命令行参数
function parseArgs() {
  const args = process.argv.slice(2);
  const options = {
    input: null,
    output: 'clean.csv',
    rules: null,
    removeDuplicates: true,
    fillMissing: 'mean',
    detectOutliers: true,
    normalizeColumns: []
  };

  for (let i = 0; i < args.length; i++) {
    switch (args[i]) {
      case '--input':
      case '-i':
        options.input = args[++i];
        break;
      case '--output':
      case '-o':
        options.output = args[++i];
        break;
      case '--rules':
        options.rules = args[++i];
        break;
      case '--remove-duplicates':
        options.removeDuplicates = args[++i] === 'true';
        break;
      case '--fill-missing':
        options.fillMissing = args[++i];
        break;
      case '--detect-outliers':
        options.detectOutliers = args[++i] === 'true';
        break;
      case '--normalize':
        options.normalizeColumns = args[++i].split(',');
        break;
    }
  }

  return options;
}

// 读取数据文件
function readData(filePath) {
  const ext = path.extname(filePath).toLowerCase();
  const content = fs.readFileSync(filePath, 'utf-8');

  if (ext === '.json') {
    return JSON.parse(content);
  } else if (ext === '.csv') {
    return parseCSV(content);
  } else {
    throw new Error(`不支持的文件格式: ${ext}`);
  }
}

// 解析CSV
function parseCSV(content) {
  const lines = content.trim().split('\n');
  if (lines.length === 0) return [];

  const headers = parseCSVLine(lines[0]);
  const data = [];

  for (let i = 1; i < lines.length; i++) {
    const values = parseCSVLine(lines[i]);
    const row = {};
    headers.forEach((header, index) => {
      row[header] = values[index] || '';
    });
    data.push(row);
  }

  return data;
}

// 解析CSV行
function parseCSVLine(line) {
  const result = [];
  let current = '';
  let inQuotes = false;

  for (let i = 0; i < line.length; i++) {
    const char = line[i];
    
    if (char === '"') {
      inQuotes = !inQuotes;
    } else if (char === ',' && !inQuotes) {
      result.push(current.trim());
      current = '';
    } else {
      current += char;
    }
  }
  
  result.push(current.trim());
  return result;
}

// 转换为CSV
function toCSV(data) {
  if (!Array.isArray(data) || data.length === 0) return '';

  const headers = Object.keys(data[0]);
  const lines = [headers.join(',')];

  data.forEach(row => {
    const values = headers.map(h => {
      const value = row[h];
      if (value === null || value === undefined) return '';
      const str = String(value);
      return str.includes(',') || str.includes('"') ? `"${str}"` : str;
    });
    lines.push(values.join(','));
  });

  return lines.join('\n');
}

// 去重
function removeDuplicates(data, keyColumns = null) {
  console.log('🔍 执行去重...');
  const seen = new Set();
  const result = [];

  data.forEach(row => {
    const key = keyColumns 
      ? keyColumns.map(col => row[col]).join('|')
      : JSON.stringify(row);
    
    if (!seen.has(key)) {
      seen.add(key);
      result.push(row);
    }
  });

  console.log(`   去重前: ${data.length} 条`);
  console.log(`   去重后: ${result.length} 条`);
  console.log(`   移除: ${data.length - result.length} 条重复数据`);

  return result;
}

// 填充缺失值
function fillMissing(data, strategy = 'mean') {
  console.log('🔧 填充缺失值...');
  
  // 统计缺失情况
  const columns = Object.keys(data[0] || {});
  const missingStats = {};
  
  columns.forEach(col => {
    const values = data.map(row => row[col]).filter(v => v !== null && v !== undefined && v !== '');
    const missing = data.length - values.length;
    if (missing > 0) {
      missingStats[col] = { total: data.length, missing, values };
    }
  });

  // 根据策略填充
  Object.entries(missingStats).forEach(([col, stats]) => {
    const numericValues = stats.values
      .map(v => parseFloat(v))
      .filter(v => !isNaN(v));

    let fillValue;
    
    if (strategy === 'mean' && numericValues.length > 0) {
      fillValue = numericValues.reduce((a, b) => a + b, 0) / numericValues.length;
      fillValue = Math.round(fillValue * 100) / 100;
    } else if (strategy === 'median' && numericValues.length > 0) {
      const sorted = numericValues.sort((a, b) => a - b);
      fillValue = sorted[Math.floor(sorted.length / 2)];
    } else if (strategy === 'mode') {
      const counts = {};
      stats.values.forEach(v => {
        counts[v] = (counts[v] || 0) + 1;
      });
      fillValue = Object.entries(counts).sort((a, b) => b[1] - a[1])[0][0];
    } else if (strategy === 'zero' || strategy === '0') {
      fillValue = 0;
    } else {
      fillValue = '';
    }

    // 应用填充
    data.forEach(row => {
      if (row[col] === null || row[col] === undefined || row[col] === '') {
        row[col] = fillValue;
      }
    });

    console.log(`   列 "${col}": 填充 ${stats.missing} 个缺失值为 ${fillValue}`);
  });

  return data;
}

// 异常值检测 (IQR方法)
function detectOutliers(data, columns = null) {
  console.log('📊 检测异常值...');
  
  const allColumns = columns || Object.keys(data[0] || {});
  const outlierCount = {};

  allColumns.forEach(col => {
    const values = data
      .map(row => parseFloat(row[col]))
      .filter(v => !isNaN(v));

    if (values.length === 0) return;

    values.sort((a, b) => a - b);
    
    const q1 = values[Math.floor(values.length * 0.25)];
    const q3 = values[Math.floor(values.length * 0.75)];
    const iqr = q3 - q1;
    
    const lower = q1 - 1.5 * iqr;
    const upper = q3 + 1.5 * iqr;

    let count = 0;
    data.forEach(row => {
      const val = parseFloat(row[col]);
      if (!isNaN(val) && (val < lower || val > upper)) {
        row[`${col}_outlier`] = true;
        count++;
      }
    });

    if (count > 0) {
      outlierCount[col] = count;
      console.log(`   列 "${col}": 发现 ${count} 个异常值 (范围: ${lower.toFixed(2)} ~ ${upper.toFixed(2)})`);
    }
  });

  return data;
}

// 数据标准化
function normalize(data, columns) {
  console.log('📏 标准化数据...');
  
  columns.forEach(col => {
    const values = data
      .map(row => parseFloat(row[col]))
      .filter(v => !isNaN(v));

    if (values.length === 0) return;

    const min = Math.min(...values);
    const max = Math.max(...values);
    const range = max - min;

    if (range === 0) return;

    data.forEach(row => {
      const val = parseFloat(row[col]);
      if (!isNaN(val)) {
        row[`${col}_normalized`] = (val - min) / range;
      }
    });

    console.log(`   列 "${col}": 标准化到 [0, 1]`);
  });

  return data;
}

// 数据类型转换
function convertTypes(data, rules) {
  console.log('🔄 转换数据类型...');

  Object.entries(rules).forEach(([col, type]) => {
    data.forEach(row => {
      const val = row[col];
      
      if (val === null || val === undefined || val === '') return;

      switch (type) {
        case 'number':
          row[col] = parseFloat(val) || 0;
          break;
        case 'integer':
          row[col] = parseInt(val) || 0;
          break;
        case 'string':
          row[col] = String(val);
          break;
        case 'boolean':
          row[col] = ['true', '1', 'yes', '是'].includes(String(val).toLowerCase());
          break;
        case 'date':
          row[col] = new Date(val).toISOString().split('T')[0];
          break;
      }
    });

    console.log(`   列 "${col}": 转换为 ${type}`);
  });

  return data;
}

// 数据质量报告
function generateReport(data, originalCount) {
  console.log('\n📋 数据质量报告');
  console.log('═'.repeat(40));
  console.log(`原始数据量: ${originalCount} 条`);
  console.log(`清洗后数据量: ${data.length} 条`);
  console.log(`数据保留率: ${((data.length / originalCount) * 100).toFixed(1)}%`);
  
  const columns = Object.keys(data[0] || {});
  console.log(`\n字段数量: ${columns.length}`);
  
  // 完整性检查
  console.log('\n字段完整性:');
  columns.forEach(col => {
    const filled = data.filter(row => row[col] !== null && row[col] !== undefined && row[col] !== '').length;
    const pct = ((filled / data.length) * 100).toFixed(1);
    const bar = '█'.repeat(Math.floor(pct / 5)) + '░'.repeat(20 - Math.floor(pct / 5));
    console.log(`  ${col.padEnd(20)} [${bar}] ${pct}%`);
  });

  console.log('═'.repeat(40));
}

// 主函数
async function main() {
  console.log('╔════════════════════════════════════╗');
  console.log('║   DataMaster Pro - 数据清洗工具    ║');
  console.log('╚════════════════════════════════════╝\n');

  const options = parseArgs();

  if (!options.input) {
    console.log('使用方法:');
    console.log('  node data-clean.js --input <文件> [选项]');
    console.log('\n选项:');
    console.log('  --output, -o          输出文件 (默认: clean.csv)');
    console.log('  --remove-duplicates   是否去重 (默认: true)');
    console.log('  --fill-missing        填充策略: mean/median/mode/zero (默认: mean)');
    console.log('  --detect-outliers     检测异常值 (默认: true)');
    console.log('  --normalize           标准化列 (逗号分隔)');
    console.log('\n示例:');
    console.log('  node data-clean.js --input data.json --output clean.csv');
    console.log('  node data-clean.js -i data.csv --fill-missing median');
    return;
  }

  try {
    console.log(`📂 读取数据: ${options.input}`);
    const data = readData(options.input);
    const originalCount = Array.isArray(data) ? data.length : 1;
    console.log(`✅ 读取 ${originalCount} 条数据\n`);

    let cleanedData = [...data];

    // 执行清洗步骤
    if (options.removeDuplicates) {
      cleanedData = removeDuplicates(cleanedData);
    }

    if (options.fillMissing) {
      cleanedData = fillMissing(cleanedData, options.fillMissing);
    }

    if (options.detectOutliers) {
      cleanedData = detectOutliers(cleanedData);
    }

    if (options.normalizeColumns.length > 0) {
      cleanedData = normalize(cleanedData, options.normalizeColumns);
    }

    // 生成报告
    generateReport(cleanedData, originalCount);

    // 保存数据
    const ext = path.extname(options.output);
    const content = ext === '.json' 
      ? JSON.stringify(cleanedData, null, 2)
      : toCSV(cleanedData);
    
    fs.writeFileSync(options.output, content);
    console.log(`\n💾 数据已保存到: ${options.output}`);
    console.log('✨ 清洗完成！');

  } catch (error) {
    console.error('\n❌ 清洗失败:', error.message);
    process.exit(1);
  }
}

main();

path: scripts/data-fetch.js
content: /**
 * DataMaster Pro - 数据抓取脚本
 * 支持网页、API、数据库三种数据源
 */

const https = require('https');
const http = require('http');
const fs = require('fs');
const path = require('path');

// 解析命令行参数
function parseArgs() {
  const args = process.argv.slice(2);
  const options = {
    url: null,
    api: null,
    db: null,
    selector: 'table',
    method: 'GET',
    headers: {},
    output: 'data.json',
    query: null
  };

  for (let i = 0; i < args.length; i++) {
    switch (args[i]) {
      case '--url':
        options.url = args[++i];
        break;
      case '--api':
        options.api = args[++i];
        break;
      case '--db':
        options.db = args[++i];
        break;
      case '--selector':
        options.selector = args[++i];
        break;
      case '--method':
        options.method = args[++i];
        break;
      case '--headers':
        options.headers = JSON.parse(args[++i]);
        break;
      case '--output':
        options.output = args[++i];
        break;
      case '--query':
        options.query = args[++i];
        break;
    }
  }

  return options;
}

// HTTP 请求封装
function httpRequest(url, options = {}) {
  return new Promise((resolve, reject) => {
    const urlObj = new URL(url);
    const lib = urlObj.protocol === 'https:' ? https : http;

    const reqOptions = {
      hostname: urlObj.hostname,
      port: urlObj.port || (urlObj.protocol === 'https:' ? 443 : 80),
      path: urlObj.pathname + urlObj.search,
      method: options.method || 'GET',
      headers: {
        'User-Agent': 'DataMaster-Pro/1.0',
        ...options.headers
      }
    };

    const req = lib.request(reqOptions, (res) => {
      let data = '';
      res.on('data', chunk => data += chunk);
      res.on('end', () => {
        if (res.statusCode >= 200 && res.statusCode < 300) {
          resolve(data);
        } else {
          reject(new Error(`HTTP ${res.statusCode}: ${data}`));
        }
      });
    });

    req.on('error', reject);
    req.end();
  });
}

// 网页抓取 - 简化版HTML解析
function parseHTML(html, selector) {
  const data = [];
  
  // 简单的表格提取
  if (selector.includes('table')) {
    const tableRegex = /<table[^>]*>([\s\S]*?)<\/table>/gi;
    const rowRegex = /<tr[^>]*>([\s\S]*?)<\/tr>/gi;
    const cellRegex = /<t[dh][^>]*>([\s\S]*?)<\/t[dh]>/gi;
    
    let tableMatch;
    while ((tableMatch = tableRegex.exec(html)) !== null) {
      const table = tableMatch[1];
      let rowMatch;
      
      while ((rowMatch = rowRegex.exec(table)) !== null) {
        const row = [];
        const rowContent = rowMatch[1];
        let cellMatch;
        
        while ((cellMatch = cellRegex.exec(rowContent)) !== null) {
          // 清理HTML标签
          let cell = cellMatch[1]
            .replace(/<[^>]+>/g, '')
            .replace(/&nbsp;/g, ' ')
            .replace(/&lt;/g, '<')
            .replace(/&gt;/g, '>')
            .replace(/&amp;/g, '&')
            .trim();
          row.push(cell);
        }
        
        if (row.length > 0) {
          data.push(row);
        }
      }
    }
  }
  
  // 简单的列表提取
  if (selector.includes('li') || selector.includes('list')) {
    const liRegex = /<li[^>]*>([\s\S]*?)<\/li>/gi;
    let match;
    
    while ((match = liRegex.exec(html)) !== null) {
      let text = match[1]
        .replace(/<[^>]+>/g, '')
        .replace(/&nbsp;/g, ' ')
        .trim();
      if (text) {
        data.push({ text });
      }
    }
  }
  
  return data;
}

// API 数据获取
async function fetchAPI(url, method, headers) {
  console.log(`📡 正在请求 API: ${url}`);
  
  try {
    const response = await httpRequest(url, { method, headers });
    const data = JSON.parse(response);
    console.log(`✅ 成功获取 ${Array.isArray(data) ? data.length : 1} 条数据`);
    return data;
  } catch (error) {
    console.error(`❌ API 请求失败: ${error.message}`);
    throw error;
  }
}

// 网页数据抓取
async function fetchWeb(url, selector) {
  console.log(`🌐 正在抓取网页: ${url}`);
  
  try {
    const html = await httpRequest(url);
    const data = parseHTML(html, selector);
    console.log(`✅ 成功解析 ${data.length} 行数据`);
    return data;
  } catch (error) {
    console.error(`❌ 网页抓取失败: ${error.message}`);
    throw error;
  }
}

// 模拟数据库查询（需要实际数据库驱动）
async function fetchDatabase(dbString, query) {
  console.log(`🗄️ 正在连接数据库...`);
  console.log(`📝 执行查询: ${query}`);
  
  // 这里是示例实现
  // 实际使用需要安装对应数据库驱动
  // 如: mysql2, pg, mongodb
  
  console.log('⚠️ 数据库功能需要安装对应驱动');
  console.log('   MySQL: npm install mysql2');
  console.log('   PostgreSQL: npm install pg');
  console.log('   MongoDB: npm install mongodb');
  
  throw new Error('数据库功能需要配置驱动');
}

// 保存数据
function saveData(data, outputFile) {
  const ext = path.extname(outputFile);
  
  if (ext === '.json') {
    fs.writeFileSync(outputFile, JSON.stringify(data, null, 2));
  } else if (ext === '.csv') {
    const csv = convertToCSV(data);
    fs.writeFileSync(outputFile, csv);
  } else {
    fs.writeFileSync(outputFile, JSON.stringify(data, null, 2));
  }
  
  console.log(`💾 数据已保存到: ${outputFile}`);
}

// 转换为CSV
function convertToCSV(data) {
  if (!Array.isArray(data) || data.length === 0) return '';
  
  // 处理对象数组
  if (typeof data[0] === 'object' && !Array.isArray(data[0])) {
    const headers = Object.keys(data[0]);
    const rows = data.map(item => headers.map(h => `"${item[h] || ''}"`).join(','));
    return [headers.join(','), ...rows].join('\n');
  }
  
  // 处理二维数组
  return data.map(row => row.map(cell => `"${cell || ''}"`).join(',')).join('\n');
}

// 主函数
async function main() {
  console.log('╔════════════════════════════════════╗');
  console.log('║   DataMaster Pro - 数据抓取工具    ║');
  console.log('╚════════════════════════════════════╝\n');
  
  const options = parseArgs();
  let data = null;
  
  try {
    if (options.url) {
      data = await fetchWeb(options.url, options.selector);
    } else if (options.api) {
      data = await fetchAPI(options.api, options.method, options.headers);
    } else if (options.db) {
      data = await fetchDatabase(options.db, options.query);
    } else {
      console.log('使用方法:');
      console.log('  node data-fetch.js --url <URL> --selector <选择器>');
      console.log('  node data-fetch.js --api <API_URL> --method <GET|POST>');
      console.log('  node data-fetch.js --db <连接字符串> --query <SQL>');
      console.log('\n示例:');
      console.log('  node data-fetch.js --url "https://example.com/data" --selector "table"');
      console.log('  node data-fetch.js --api "https://api.example.com/v1/data" --output data.json');
      return;
    }
    
    if (data) {
      saveData(data, options.output);
      console.log('\n✨ 抓取完成！');
    }
  } catch (error) {
    console.error('\n❌ 抓取失败:', error.message);
    process.exit(1);
  }
}

main();

path: scripts/data-viz.js
content: /**
 * DataMaster Pro - 数据可视化脚本
 * 支持多种图表类型，输出 SVG/PNG/HTML
 */

const fs = require('fs');
const path = require('path');

// 解析命令行参数
function parseArgs() {
  const args = process.argv.slice(2);
  const options = {
    input: null,
    output: 'chart',
    type: 'bar',
    title: 'Data Chart',
    xColumn: null,
    yColumn: null,
    width: 800,
    height: 400,
    colors: ['#4CAF50', '#2196F3', '#FF9800', '#E91E63', '#9C27B0', '#00BCD4'],
    format: 'svg'
  };

  for (let i = 0; i < args.length; i++) {
    switch (args[i]) {
      case '--input':
      case '-i':
        options.input = args[++i];
        break;
      case '--output':
      case '-o':
        options.output = args[++i];
        break;
      case '--type':
      case '-t':
        options.type = args[++i];
        break;
      case '--title':
        options.title = args[++i];
        break;
      case '--x':
        options.xColumn = args[++i];
        break;
      case '--y':
        options.yColumn = args[++i];
        break;
      case '--width':
        options.width = parseInt(args[++i]);
        break;
      case '--height':
        options.height = parseInt(args[++i]);
        break;
      case '--colors':
        options.colors = args[++i].split(',');
        break;
      case '--format':
        options.format = args[++i];
        break;
    }
  }

  return options;
}

// 读取数据
function readData(filePath) {
  const ext = path.extname(filePath).toLowerCase();
  const content = fs.readFileSync(filePath, 'utf-8');

  if (ext === '.json') {
    return JSON.parse(content);
  } else if (ext === '.csv') {
    return parseCSV(content);
  }
  throw new Error(`不支持的文件格式: ${ext}`);
}

// 解析CSV
function parseCSV(content) {
  const lines = content.trim().split('\n');
  if (lines.length < 2) return [];

  const headers = parseCSVLine(lines[0]);
  const data = [];

  for (let i = 1; i < lines.length; i++) {
    const values = parseCSVLine(lines[i]);
    const row = {};
    headers.forEach((header, index) => {
      const val = values[index] || '';
      row[header] = isNaN(val) ? val : parseFloat(val);
    });
    data.push(row);
  }

  return data;
}

function parseCSVLine(line) {
  const result = [];
  let current = '';
  let inQuotes = false;

  for (let i = 0; i < line.length; i++) {
    const char = line[i];
    if (char === '"') {
      inQuotes = !inQuotes;
    } else if (char === ',' && !inQuotes) {
      result.push(current.trim());
      current = '';
    } else {
      current += char;
    }
  }
  result.push(current.trim());
  return result;
}

// SVG 生成器
class SVGChart {
  constructor(width, height) {
    this.width = width;
    this.height = height;
    this.margin = { top: 40, right: 30, bottom: 50, left: 60 };
    this.elements = [];
  }

  header() {
    return `<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 ${this.width} ${this.height}" width="${this.width}" height="${this.height}">
  <style>
    .axis-label { font-family: Arial, sans-serif; font-size: 12px; fill: #666; }
    .axis-title { font-family: Arial, sans-serif; font-size: 14px; fill: #333; font-weight: bold; }
    .chart-title { font-family: Arial, sans-serif; font-size: 18px; fill: #333; font-weight: bold; }
    .legend { font-family: Arial, sans-serif; font-size: 11px; fill: #666; }
    .grid { stroke: #e0e0e0; stroke-width: 1; }
    .tick { stroke: #ccc; stroke-width: 1; }
  </style>`;
  }

  footer() {
    return '</svg>';
  }

  title(text) {
    return `<text x="${this.width / 2}" y="25" class="chart-title" text-anchor="middle">${this.escape(text)}</text>`;
  }

  escape(text) {
    return String(text)
      .replace(/&/g, '&amp;')
      .replace(/</g, '&lt;')
      .replace(/>/g, '&gt;')
      .replace(/"/g, '&quot;');
  }

  rect(x, y, w, h, fill, options = {}) {
    const attrs = Object.entries(options)
      .map(([k, v]) => `${k}="${this.escape(v)}"`)
      .join(' ');
    return `<rect x="${x}" y="${y}" width="${w}" height="${h}" fill="${fill}" ${attrs}/>`;
  }

  circle(cx, cy, r, fill, options = {}) {
    const attrs = Object.entries(options)
      .map(([k, v]) => `${k}="${this.escape(v)}"`)
      .join(' ');
    return `<circle cx="${cx}" cy="${cy}" r="${r}" fill="${fill}" ${attrs}/>`;
  }

  line(x1, y1, x2, y2, options = {}) {
    const attrs = Object.entries(options)
      .map(([k, v]) => `${k}="${this.escape(v)}"`)
      .join(' ');
    return `<line x1="${x1}" y1="${y1}" x2="${x2}" y2="${y2}" ${attrs}/>`;
  }

  text(x, y, content, options = {}) {
    const attrs = Object.entries(options)
      .map(([k, v]) => `${k}="${this.escape(v)}"`)
      .join(' ');
    return `<text x="${x}" y="${y}" ${attrs}>${this.escape(content)}</text>`;
  }

  path(d, options = {}) {
    const attrs = Object.entries(options)
      .map(([k, v]) => `${k}="${this.escape(v)}"`)
      .join(' ');
    return `<path d="${d}" ${attrs}/>`;
  }
}

// 柱状图
function generateBarChart(data, options) {
  const chart = new SVGChart(options.width, options.height);
  const { margin, width, height } = chart;
  
  const chartWidth = width - margin.left - margin.right;
  const chartHeight = height - margin.top - margin.bottom;

  // 获取数据
  const keys = Object.keys(data[0] || {});
  const xKey = options.xColumn || keys[0];
  const yKey = options.yColumn || keys.find(k => typeof data[0][k] === 'number') || keys[1];

  const xValues = data.map(d => d[xKey]);
  const yValues = data.map(d => parseFloat(d[yKey]) || 0);
  const maxY = Math.max(...yValues) * 1.1;

  // SVG 头部
  let svg = chart.header();
  svg += chart.title(options.title);

  // Y轴网格线
  const yTicks = 5;
  for (let i = 0; i <= yTicks; i++) {
    const y = margin.top + (chartHeight / yTicks) * i;
    svg += chart.line(margin.left, y, width - margin.right, y, { class: 'grid' });
    const val = (maxY - (maxY / yTicks) * i).toFixed(0);
    svg += chart.text(margin.left - 10, y + 4, val, { class: 'axis-label', 'text-anchor': 'end' });
  }

  // 绘制柱状图
  const barWidth = (chartWidth / data.length) * 0.7;
  const gap = (chartWidth / data.length) * 0.3;

  data.forEach((d, i) => {
    const x = margin.left + i * (barWidth + gap) + gap / 2;
    const barHeight = (yValues[i] / maxY) * chartHeight;
    const y = margin.top + chartHeight - barHeight;
    const color = options.colors[i % options.colors.length];

    svg += chart.rect(x, y, barWidth, barHeight, color, { rx: '3' });

    // X轴标签
    svg += chart.text(x + barWidth / 2, height - margin.bottom + 20, String(xValues[i]).slice(0, 10), {
      class: 'axis-label',
      'text-anchor': 'middle',
      transform: `rotate(-45 ${x + barWidth / 2} ${height - margin.bottom + 20})`
    });

    // 数值标签
    svg += chart.text(x + barWidth / 2, y - 5, yValues[i].toFixed(0), {
      class: 'axis-label',
      'text-anchor': 'middle'
    });
  });

  // Y轴标题
  svg += chart.text(15, height / 2, yKey, {
    class: 'axis-title',
    transform: `rotate(-90 15 ${height / 2})`,
    'text-anchor': 'middle'
  });

  svg += chart.footer();
  return svg;
}

// 折线图
function generateLineChart(data, options) {
  const chart = new SVGChart(options.width, options.height);
  const { margin, width, height } = chart;
  
  const chartWidth = width - margin.left - margin.right;
  const chartHeight = height - margin.top - margin.bottom;

  const keys = Object.keys(data[0] || {});
  const xKey = options.xColumn || keys[0];
  const yKey = options.yColumn || keys.find(k => typeof data[0][k] === 'number') || keys[1];

  const yValues = data.map(d => parseFloat(d[yKey]) || 0);
  const maxY = Math.max(...yValues) * 1.1;

  let svg = chart.header();
  svg += chart.title(options.title);

  // Y轴网格线
  const yTicks = 5;
  for (let i = 0; i <= yTicks; i++) {
    const y = margin.top + (chartHeight / yTicks) * i;
    svg += chart.line(margin.left, y, width - margin.right, y, { class: 'grid' });
    const val = (maxY - (maxY / yTicks) * i).toFixed(0);
    svg += chart.text(margin.left - 10, y + 4, val, { class: 'axis-label', 'text-anchor': 'end' });
  }

  // 计算点坐标
  const points = data.map((d, i) => {
    const x = margin.left + (chartWidth / (data.length - 1 || 1)) * i;
    const y = margin.top + chartHeight - (yValues[i] / maxY) * chartHeight;
    return { x, y, label: d[xKey] };
  });

  // 绘制折线
  const pathD = points.map((p, i) => `${i === 0 ? 'M' : 'L'} ${p.x} ${p.y}`).join(' ');
  svg += chart.path(pathD, { fill: 'none', stroke: options.colors[0], 'stroke-width': '2' });

  // 绘制点
  points.forEach((p, i) => {
    svg += chart.circle(p.x, p.y, 4, options.colors[0]);
    svg += chart.text(p.x, height - margin.bottom + 20, String(p.label).slice(0, 8), {
      class: 'axis-label',
      'text-anchor': 'middle'
    });
  });

  svg += chart.footer();
  return svg;
}

// 饼图
function generatePieChart(data, options) {
  const chart = new SVGChart(options.width, options.height);
  const { width, height } = chart;
  
  const keys = Object.keys(data[0] || {});
  const labelKey = options.xColumn || keys[0];
  const valueKey = options.yColumn || keys.find(k => typeof data[0][k] === 'number') || keys[1];

  const values = data.map(d => ({ label: d[labelKey], value: parseFloat(d[valueKey]) || 0 }));
  const total = values.reduce((sum, v) => sum + v.value, 0);

  const cx = width / 2;
  const cy = height / 2;
  const r = Math.min(width, height) / 2 - 60;

  let svg = chart.header();
  svg += chart.title(options.title);

  let startAngle = -Math.PI / 2;
  
  values.forEach((v, i) => {
    const angle = (v.value / total) * 2 * Math.PI;
    const endAngle = startAngle + angle;
    const color = options.colors[i % options.colors.length];

    // 计算路径
    const x1 = cx + r * Math.cos(startAngle);
    const y1 = cy + r * Math.sin(startAngle);
    const x2 = cx + r * Math.cos(endAngle);
    const y2 = cy + r * Math.sin(endAngle);

    const largeArc = angle > Math.PI ? 1 : 0;
    const pathD = `M ${cx} ${cy} L ${x1} ${y1} A ${r} ${r} 0 ${largeArc} 1 ${x2} ${y2} Z`;

    svg += chart.path(pathD, { fill: color, stroke: '#fff', 'stroke-width': '2' });

    // 标签
    const midAngle = startAngle + angle / 2;
    const labelR = r * 0.7;
    const labelX = cx + labelR * Math.cos(midAngle);
    const labelY = cy + labelR * Math.sin(midAngle);

    const pct = ((v.value / total) * 100).toFixed(1);
    if (pct > 5) {
      svg += chart.text(labelX, labelY + 4, `${pct}%`, {
        class: 'axis-label',
        'text-anchor': 'middle',
        fill: '#fff',
        'font-weight': 'bold'
      });
    }

    startAngle = endAngle;
  });

  // 图例
  values.forEach((v, i) => {
    const legendX = width - 120;
    const legendY = 60 + i * 20;
    const color = options.colors[i % options.colors.length];

    svg += chart.rect(legendX, legendY - 8, 12, 12, color);
    svg += chart.text(legendX + 18, legendY, String(v.label).slice(0, 15), { class: 'legend' });
  });

  svg += chart.footer();
  return svg;
}

// 散点图
function generateScatterChart(data, options) {
  const chart = new SVGChart(options.width, options.height);
  const { margin, width, height } = chart;
  
  const chartWidth = width - margin.left - margin.right;
  const chartHeight = height - margin.top - margin.bottom;

  const keys = Object.keys(data[0] || {});
  const numericKeys = keys.filter(k => typeof data[0][k] === 'number');
  const xKey = options.xColumn || numericKeys[0] || keys[0];
  const yKey = options.yColumn || numericKeys[1] || keys[1];

  const xValues = data.map(d => parseFloat(d[xKey]) || 0);
  const yValues = data.map(d => parseFloat(d[yKey]) || 0);

  const maxX = Math.max(...xValues) * 1.1;
  const maxY = Math.max(...yValues) * 1.1;

  let svg = chart.header();
  svg += chart.title(options.title);

  // 网格
  for (let i = 0; i <= 5; i++) {
    const y = margin.top + (chartHeight / 5) * i;
    svg += chart.line(margin.left, y, width - margin.right, y, { class: 'grid' });
    svg += chart.text(margin.left - 10, y + 4, ((maxY / 5) * (5 - i)).toFixed(0), { class: 'axis-label', 'text-anchor': 'end' });

    const x = margin.left + (chartWidth / 5) * i;
    svg += chart.line(x, margin.top, x, height - margin.bottom, { class: 'grid' });
    svg += chart.text(x, height - margin.bottom + 20, ((maxX / 5) * i).toFixed(0), { class: 'axis-label', 'text-anchor': 'middle' });
  }

  // 绘制点
  data.forEach((d, i) => {
    const x = margin.left + (xValues[i] / maxX) * chartWidth;
    const y = margin.top + chartHeight - (yValues[i] / maxY) * chartHeight;
    svg += chart.circle(x, y, 5, options.colors[i % options.colors.length], { opacity: '0.7' });
  });

  svg += chart.footer();
  return svg;
}

// 生成 HTML 包装
function wrapInHTML(svg, title) {
  return `<!DOCTYPE html>
<html lang="zh-CN">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>${title}</title>
  <style>
    body { 
      margin: 0; 
      padding: 20px; 
      background: #f5f5f5; 
      font-family: Arial, sans-serif;
      display: flex;
      justify-content: center;
      align-items: center;
      min-height: 100vh;
    }
    .chart-container {
      background: white;
      padding: 20px;
      border-radius: 8px;
      box-shadow: 0 2px 10px rgba(0,0,0,0.1);
    }
  </style>
</head>
<body>
  <div class="chart-container">
    ${svg}
  </div>
</body>
</html>`;
}

// 主函数
async function main() {
  console.log('╔════════════════════════════════════╗');
  console.log('║   DataMaster Pro - 数据可视化工具  ║');
  console.log('╚════════════════════════════════════╝\n');

  const options = parseArgs();

  if (!options.input) {
    console.log('使用方法:');
    console.log('  node data-viz.js --input <文件> [选项]');
    console.log('\n选项:');
    console.log('  --type, -t       图表类型: bar/line/pie/scatter (默认: bar)');
    console.log('  --output, -o     输出文件名 (默认: chart)');
    console.log('  --title          图表标题');
    console.log('  --x              X轴列名');
    console.log('  --y              Y轴列名');
    console.log('  --width          图表宽度 (默认: 800)');
    console.log('  --height         图表高度 (默认: 400)');
    console.log('  --format         输出格式: svg/html (默认: svg)');
    console.log('\n示例:');
    console.log('  node data-viz.js -i data.csv -t bar --title "销售数据"');
    console.log('  node data-viz.js -i data.json -t pie --x category --y amount');
    return;
  }

  try {
    console.log(`📂 读取数据: ${options.input}`);
    const data = readData(options.input);
    console.log(`✅ 读取 ${data.length} 条数据\n`);

    let svg;
    console.log(`📊 生成${options.type === 'bar' ? '柱状' : options.type === 'line' ? '折线' : options.type === 'pie' ? '饼' : '散点'}图...`);

    switch (options.type) {
      case 'bar':
        svg = generateBarChart(data, options);
        break;
      case 'line':
        svg = generateLineChart(data, options);
        break;
      case 'pie':
        svg = generatePieChart(data, options);
        break;
      case 'scatter':
        svg = generateScatterChart(data, options);
        break;
      default:
        svg = generateBarChart(data, options);
    }

    // 保存文件
    const ext = options.format === 'html' ? 'html' : 'svg';
    const outputFile = `${options.output}.${ext}`;
    const content = options.format === 'html' ? wrapInHTML(svg, options.title) : svg;

    fs.writeFileSync(outputFile, content);
    console.log(`💾 图表已保存到: ${outputFile}`);
    console.log('✨ 可视化完成！');

  } catch (error) {
    console.error('\n❌ 可视化失败:', error.message);
    process.exit(1);
  }
}

main();

path: scripts/report-generate.js
content: #!/usr/bin/env node

/**
 * Data Analysis Skill - Report Generation
 * 自动生成数据报告
 * 
 * 功能：
 * - 从清洗后的数据生成分析报告
 * - 支持多种报告模板
 * - 输出PDF/HTML/Markdown格式
 * - 包含关键指标和洞察
 */

const fs = require('fs');
const path = require('path');
const { execSync } = require('child_process');

// 配置文件路径
const CONFIG_PATH = path.join(__dirname, '..', 'config.json');
const TEMPLATES_DIR = path.join(__dirname, '..', 'templates');
const OUTPUT_DIR = path.join(__dirname, '..', 'output');

// 确保输出目录存在
if (!fs.existsSync(OUTPUT_DIR)) {
    fs.mkdirSync(OUTPUT_DIR, { recursive: true });
}

class ReportGenerator {
    constructor() {
        this.config = this.loadConfig();
    }

    loadConfig() {
        try {
            const configContent = fs.readFileSync(CONFIG_PATH, 'utf8');
            return JSON.parse(configContent);
        } catch (error) {
            console.log('⚠️  未找到配置文件，使用默认配置');
            return {
                reportTemplate: 'default',
                outputFormat: 'html',
                includeCharts: true,
                language: 'zh-CN'
            };
        }
    }

    /**
     * 生成数据报告
     * @param {string} dataPath - 清洗后数据的路径
     * @param {string} reportName - 报告名称
     */
    generateReport(dataPath, reportName = 'data-report') {
        console.log('📊 开始生成数据报告...');
        
        // 读取清洗后的数据
        const rawData = fs.readFileSync(dataPath, 'utf8');
        let data;
        try {
            data = JSON.parse(rawData);
        } catch (error) {
            console.error('❌ 数据格式错误，无法解析JSON');
            return false;
        }

        // 执行数据分析
        const analysis = this.analyzeData(data);
        
        // 生成报告内容
        const reportContent = this.createReportContent(analysis, data);
        
        // 保存报告
        const timestamp = new Date().toISOString().split('T')[0];
        const outputBaseName = `${reportName}-${timestamp}`;
        
        if (this.config.outputFormat === 'pdf') {
            this.saveAsPDF(reportContent, outputBaseName);
        } else if (this.config.outputFormat === 'markdown') {
            this.saveAsMarkdown(reportContent, outputBaseName);
        } else {
            this.saveAsHTML(reportContent, outputBaseName);
        }
        
        console.log(`✅ 报告已生成: ${OUTPUT_DIR}/${outputBaseName}`);
        return true;
    }

    /**
     * 数据分析核心逻辑
     */
    analyzeData(data) {
        const analysis = {
            summary: {},
            insights: [],
            metrics: {},
            trends: []
        };

        // 基础统计信息
        analysis.summary.totalRecords = Array.isArray(data) ? data.length : Object.keys(data).length;
        analysis.summary.dataTypes = this.detectDataTypes(data);
        
        // 关键指标计算
        analysis.metrics = this.calculateMetrics(data);
        
        // 洞察发现
        analysis.insights = this.generateInsights(data, analysis.metrics);
        
        // 趋势分析（如果数据包含时间序列）
        if (this.hasTimeSeries(data)) {
            analysis.trends = this.analyzeTrends(data);
        }

        return analysis;
    }

    /**
     * 检测数据类型
     */
    detectDataTypes(data) {
        const types = {};
        if (Array.isArray(data) && data.length > 0) {
            const firstRecord = data[0];
            if (typeof firstRecord === 'object' && firstRecord !== null) {
                Object.keys(firstRecord).forEach(key => {
                    const value = firstRecord[key];
                    if (typeof value === 'number') {
                        types[key] = 'numeric';
                    } else if (typeof value === 'string') {
                        // 尝试判断是否为日期
                        if (this.isDateString(value)) {
                            types[key] = 'date';
                        } else {
                            types[key] = 'string';
                        }
                    } else if (typeof value === 'boolean') {
                        types[key] = 'boolean';
                    } else {
                        types[key] = typeof value;
                    }
                });
            }
        }
        return types;
    }

    /**
     * 判断字符串是否为日期格式
     */
    isDateString(str) {
        const datePatterns = [
            /^\d{4}-\d{2}-\d{2}$/,
            /^\d{4}\/\d{2}\/\d{2}$/,
            /^\d{2}\/\d{2}\/\d{4}$/,
            /^\d{1,2}-\d{1,2}-\d{4}$/
        ];
        return datePatterns.some(pattern => pattern.test(str));
    }

    /**
     * 计算关键指标
     */
    calculateMetrics(data) {
        const metrics = {};
        
        if (!Array.isArray(data) || data.length === 0) return metrics;

        // 获取数值字段
        const numericFields = [];
        const firstRecord = data[0];
        if (typeof firstRecord === 'object' && firstRecord !== null) {
            Object.keys(firstRecord).forEach(key => {
                if (typeof firstRecord[key] === 'number') {
                    numericFields.push(key);
                }
            });
        }

        // 为每个数值字段计算统计指标
        numericFields.forEach(field => {
            const values = data.map(record => record[field]).filter(val => !isNaN(val) && val !== null);
            if (values.length > 0) {
                metrics[field] = {
                    sum: values.reduce((a, b) => a + b, 0),
                    average: values.reduce((a, b) => a + b, 0) / values.length,
                    min: Math.min(...values),
                    max: Math.max(...values),
                    median: this.calculateMedian(values)
                };
            }
        });

        return metrics;
    }

    /**
     * 计算中位数
     */
    calculateMedian(values) {
        const sorted = [...values].sort((a, b) => a - b);
        const mid = Math.floor(sorted.length / 2);
        return sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid];
    }

    /**
     * 生成业务洞察
     */
    generateInsights(data, metrics) {
        const insights = [];
        
        // 基于指标生成洞察
        Object.keys(metrics).forEach(field => {
            const metric = metrics[field];
            if (metric.max > metric.average * 2) {
                insights.push({
                    type: 'outlier',
                    message: `字段 "${field}" 存在显著异常值，最大值(${metric.max})是平均值(${metric.average.toFixed(2)})的${(metric.max/metric.average).toFixed(1)}倍`,
                    severity: 'high'
                });
            }
            
            if (metric.average > 0 && metric.min === 0) {
                insights.push({
                    type: 'zero_values',
                    message: `字段 "${field}" 包含零值，可能表示缺失数据或特殊状态`,
                    severity: 'medium'
                });
            }
        });

        // 数据完整性检查
        const totalRecords = data.length;
        const completeRecords = data.filter(record => {
            return Object.values(record).every(value => value !== null && value !== undefined && value !== '');
        }).length;
        
        const completeness = (completeRecords / totalRecords) * 100;
        if (completeness < 95) {
            insights.push({
                type: 'data_completeness',
                message: `数据完整性为 ${completeness.toFixed(1)}%，建议检查数据质量`,
                severity: 'medium'
            });
        }

        return insights;
    }

    /**
     * 检查是否包含时间序列数据
     */
    hasTimeSeries(data) {
        if (!Array.isArray(data) || data.length === 0) return false;
        
        const firstRecord = data[0];
        if (typeof firstRecord !== 'object' || firstRecord === null) return false;
        
        return Object.keys(firstRecord).some(key => {
            const value = firstRecord[key];
            return typeof value === 'string' && this.isDateString(value);
        });
    }

    /**
     * 时间序列趋势分析
     */
    analyzeTrends(data) {
        const trends = [];
        
        // 找到日期字段
        let dateField = null;
        const firstRecord = data[0];
        Object.keys(firstRecord).forEach(key => {
            if (typeof firstRecord[key] === 'string' && this.isDateString(firstRecord[key])) {
                dateField = key;
            }
        });
        
        if (!dateField) return trends;
        
        // 按日期排序
        const sortedData = [...data].sort((a, b) => {
            return new Date(a[dateField]) - new Date(b[dateField]);
        });
        
        // 分析数值字段的趋势
        Object.keys(firstRecord).forEach(key => {
            if (typeof firstRecord[key] === 'number') {
                const trend = this.calculateTrend(sortedData, dateField, key);
                if (trend) {
                    trends.push(trend);
                }
            }
        });
        
        return trends;
    }

    /**
     * 计算单个字段的趋势
     */
    calculateTrend(data, dateField, valueField) {
        if (data.length < 2) return null;
        
        const firstValue = data[0][valueField];
        const lastValue = data[data.length - 1][valueField];
        
        if (firstValue === null || lastValue === null || isNaN(firstValue) || isNaN(lastValue)) {
            return null;
        }
        
        const change = lastValue - firstValue;
        const changePercent = (change / firstValue) * 100;
        const direction = change > 0 ? '上升' : change < 0 ? '下降' : '稳定';
        
        return {
            field: valueField,
            direction: direction,
            change: change,
            changePercent: changePercent,
            period: `${data[0][dateField]} 至 ${data[data.length - 1][dateField]}`
        };
    }

    /**
     * 创建报告内容
     */
    createReportContent(analysis, rawData) {
        const templatePath = path.join(TEMPLATES_DIR, `${this.config.reportTemplate}.md`);
        let templateContent;
        
        try {
            templateContent = fs.readFileSync(templatePath, 'utf8');
        } catch (error) {
            console.log('⚠️  未找到模板文件，使用默认模板');
            templateContent = this.getDefaultTemplate();
        }
        
        // 替换模板变量
        let reportContent = templateContent
            .replace('{{REPORT_TITLE}}', '数据分析报告')
            .replace('{{GENERATION_DATE}}', new Date().toLocaleString('zh-CN'))
            .replace('{{TOTAL_RECORDS}}', analysis.summary.totalRecords)
            .replace('{{DATA_TYPES}}', JSON.stringify(analysis.summary.dataTypes, null, 2));
        
        // 添加关键指标
        let metricsContent = '';
        Object.keys(analysis.metrics).forEach(field => {
            const metric = analysis.metrics[field];
            metricsContent += `\n### ${field}\n`;
            metricsContent += `- 总计: ${metric.sum.toLocaleString()}\n`;
            metricsContent += `- 平均值: ${metric.average.toFixed(2)}\n`;
            metricsContent += `- 最小值: ${metric.min}\n`;
            metricsContent += `- 最大值: ${metric.max}\n`;
            metricsContent += `- 中位数: ${metric.median.toFixed(2)}\n`;
        });
        reportContent = reportContent.replace('{{KEY_METRICS}}', metricsContent);
        
        // 添加洞察
        let insightsContent = '';
        if (analysis.insights.length > 0) {
            insightsContent = '\n## 🔍 关键洞察\n';
            analysis.insights.forEach(insight => {
                const severityEmoji = insight.severity === 'high' ? '🔴' : insight.severity === 'medium' ? '🟡' : '🟢';
                insightsContent += `\n${severityEmoji} **${insight.message}**\n`;
            });
        }
        reportContent = reportContent.replace('{{INSIGHTS}}', insightsContent);
        
        // 添加趋势分析
        let trendsContent = '';
        if (analysis.trends.length > 0) {
            trendsContent = '\n## 📈 趋势分析\n';
            analysis.trends.forEach(trend => {
                const trendEmoji = trend.direction === '上升' ? '📈' : trend.direction === '下降' ? '📉' : '➡️';
                trendsContent += `\n${trendEmoji} **${trend.field}**: ${trend.direction} ${trend.changePercent > 0 ? '+' : ''}${trend.changePercent.toFixed(1)}% (${trend.period})\n`;
            });
        }
        reportContent = reportContent.replace('{{TRENDS}}', trendsContent);
        
        return reportContent;
    }

    /**
     * 默认报告模板
     */
    getDefaultTemplate() {
        return `# {{REPORT_TITLE}}

生成时间: {{GENERATION_DATE}}

## 📊 数据概览
- 总记录数: {{TOTAL_RECORDS}}
- 数据类型: 
\`\`\`json
{{DATA_TYPES}}
\`\`\`

## 📏 关键指标
{{KEY_METRICS}}

{{INSIGHTS}}

{{TRENDS}}

---
*本报告由 AI 数据分析技能包自动生成*
`;
    }

    /**
     * 保存为HTML格式
     */
    saveAsHTML(content, fileName) {
        const htmlContent = `
<!DOCTYPE html>
<html lang="zh-CN">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>数据分析报告</title>
    <style>
        body { font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif; line-height: 1.6; color: #333; max-width: 1200px; margin: 0 auto; padding: 20px; }
        h1, h2, h3 { color: #2c3e50; }
        h1 { border-bottom: 2px solid #3498db; padding-bottom: 10px; }
        h2 { border-left: 4px solid #3498db; padding-left: 10px; margin-top: 30px; }
        pre { background: #f8f9fa; padding: 15px; border-radius: 5px; overflow-x: auto; }
        code { background: #f1f1f1; padding: 2px 4px; border-radius: 3px; }
        .insight { margin: 10px 0; padding: 10px; border-left: 4px solid #f39c12; background: #fef9e7; }
        .trend-up { color: #27ae60; }
        .trend-down { color: #e74c3c; }
    </style>
</head>
<body>
${content}
</body>
</html>
        `;
        fs.writeFileSync(path.join(OUTPUT_DIR, `${fileName}.html`), htmlContent, 'utf8');
    }

    /**
     * 保存为Markdown格式
     */
    saveAsMarkdown(content, fileName) {
        fs.writeFileSync(path.join(OUTPUT_DIR, `${fileName}.md`), content, 'utf8');
    }

    /**
     * 保存为PDF格式（需要外部工具）
     */
    saveAsPDF(content, fileName) {
        // 先保存为HTML
        this.saveAsHTML(content, fileName);
        
        // 尝试使用外部工具转换为PDF
        try {
            const htmlPath = path.join(OUTPUT_DIR, `${fileName}.html`);
            const pdfPath = path.join(OUTPUT_DIR, `${fileName}.pdf`);
            
            // 检查是否安装了wkhtmltopdf或其他PDF工具
            execSync('wkhtmltopdf --version', { stdio: 'ignore' });
            execSync(`wkhtmltopdf "${htmlPath}" "${pdfPath}"`, { stdio: 'ignore' });
            console.log('📄 PDF报告已生成');
            
            // 删除临时HTML文件
            fs.unlinkSync(htmlPath);
        } catch (error) {
            console.log('⚠️  未安装PDF生成工具，仅生成HTML报告');
            console.log('💡 安装 wkhtmltopdf 可以生成PDF报告');
        }
    }
}

// 主函数
function main() {
    const args = process.argv.slice(2);
    
    if (args.length === 0) {
        console.log('用法: node report-generate.js <数据文件路径> [报告名称]');
        console.log('示例: node report-generate.js ./data/cleaned-data.json 销售分析报告');
        process.exit(1);
    }
    
    const dataPath = args[0];
    const reportName = args[1] || 'data-report';
    
    if (!fs.existsSync(dataPath)) {
        console.error('❌ 数据文件不存在:', dataPath);
        process.exit(1);
    }
    
    const generator = new ReportGenerator();
    generator.generateReport(dataPath, reportName);
}

if (require.main === module) {
    main();
}

module.exports = ReportGenerator;

path: templates/business.json
content: {
  "name": "商业分析报告",
  "description": "适合销售报告、市场分析、经营决策",
  "version": "1.0",
  
  "report": {
    "title": {
      "template": "{{公司名称}} - {{报告类型}}报告",
      "includeDate": true,
      "dateFormat": "YYYY年MM月DD日"
    },
    
    "sections": [
      {
        "name": "执行摘要",
        "type": "summary",
        "required": true,
        "content": {
          "keyMetrics": true,
          "highlights": true,
          "recommendations": true
        }
      },
      {
        "name": "数据分析",
        "type": "analysis",
        "required": true,
        "content": {
          "trends": true,
          "comparisons": true,
          "correlations": true
        }
      },
      {
        "name": "可视化图表",
        "type": "charts",
        "required": true,
        "content": {
          "mainChart": "bar",
          "secondaryChart": "line",
          "pieChart": true
        }
      },
      {
        "name": "洞察与建议",
        "type": "insights",
        "required": true,
        "content": {
          "aiInsights": true,
          "actionItems": true,
          "risks": true
        }
      },
      {
        "name": "附录",
        "type": "appendix",
        "required": false,
        "content": {
          "rawData": false,
          "methodology": true,
          "glossary": false
        }
      }
    ]
  },
  
  "charts": {
    "main": {
      "type": "bar",
      "title": "主要数据趋势",
      "colors": ["#4CAF50", "#2196F3", "#FF9800", "#9C27B0"],
      "width": 800,
      "height": 400
    },
    "secondary": {
      "type": "line",
      "title": "变化趋势",
      "showTrendLine": true,
      "showDataPoints": true
    },
    "pie": {
      "type": "pie",
      "title": "占比分布",
      "showPercentages": true,
      "showLegend": true
    }
  },
  
  "style": {
    "font": {
      "title": "Microsoft YaHei",
      "body": "Microsoft YaHei",
      "sizes": {
        "h1": 24,
        "h2": 18,
        "h3": 14,
        "body": 12
      }
    },
    "colors": {
      "primary": "#1976D2",
      "secondary": "#424242",
      "accent": "#4CAF50",
      "background": "#FFFFFF",
      "text": "#212121"
    },
    "spacing": {
      "paragraph": 1.5,
      "section": 2
    }
  },
  
  "output": {
    "formats": ["markdown", "html", "pdf"],
    "includeTableOfContents": true,
    "includePageNumbers": true,
    "chartEmbed": true
  },
  
  "aiInsights": {
    "enabled": true,
    "language": "zh-CN",
    "tone": "professional",
    "includeRecommendations": true,
    "maxInsights": 5
  }
}

path: templates/technical.json
content: {
  "name": "技术数据报告",
  "description": "适合性能分析、系统监控、技术指标报告",
  "version": "1.0",
  
  "report": {
    "title": {
      "template": "{{项目名称}} - 技术分析报告",
      "includeDate": true,
      "dateFormat": "YYYY-MM-DD"
    },
    
    "sections": [
      {
        "name": "概述",
        "type": "overview",
        "required": true,
        "content": {
          "purpose": true,
          "scope": true,
          "methodology": true
        }
      },
      {
        "name": "数据采集",
        "type": "collection",
        "required": true,
        "content": {
          "source": true,
          "timeRange": true,
          "sampleSize": true,
          "dataQuality": true
        }
      },
      {
        "name": "统计分析",
        "type": "statistics",
        "required": true,
        "content": {
          "descriptive": true,
          "distributions": true,
          "correlations": true,
          "anomalies": true
        }
      },
      {
        "name": "性能指标",
        "type": "metrics",
        "required": true,
        "content": {
          "kpis": true,
          "benchmarks": true,
          "trends": true,
          "percentiles": true
        }
      },
      {
        "name": "问题诊断",
        "type": "diagnosis",
        "required": false,
        "content": {
          "anomalies": true,
          "rootCause": true,
          "impact": true
        }
      },
      {
        "name": "结论与建议",
        "type": "conclusion",
        "required": true,
        "content": {
          "findings": true,
          "recommendations": true,
          "nextSteps": true
        }
      }
    ]
  },
  
  "charts": {
    "performance": {
      "type": "line",
      "title": "性能趋势",
      "showPercentiles": ["p50", "p90", "p99"],
      "colors": ["#2196F3", "#4CAF50", "#FF9800"],
      "width": 900,
      "height": 350
    },
    "distribution": {
      "type": "histogram",
      "title": "数据分布",
      "bins": 20,
      "showNormalCurve": true
    },
    "heatmap": {
      "type": "heatmap",
      "title": "相关性矩阵",
      "showValues": true,
      "colorScale": "RdYlGn"
    },
    "boxplot": {
      "type": "box",
      "title": "数据分布箱线图",
      "showOutliers": true
    }
  },
  
  "style": {
    "font": {
      "title": "Consolas",
      "body": "Microsoft YaHei",
      "code": "Source Code Pro",
      "sizes": {
        "h1": 22,
        "h2": 16,
        "h3": 14,
        "body": 11,
        "code": 10
      }
    },
    "colors": {
      "primary": "#1976D2",
      "secondary": "#455A64",
      "accent": "#00BCD4",
      "background": "#FAFAFA",
      "text": "#212121",
      "code": "#37474F",
      "codeBackground": "#ECEFF1"
    },
    "code": {
      "highlight": true,
      "lineNumbers": true,
      "theme": "light"
    }
  },
  
  "output": {
    "formats": ["markdown", "html"],
    "includeTableOfContents": true,
    "includeTimestamps": true,
    "chartEmbed": true,
    "rawDataAppendix": true
  },
  
  "aiInsights": {
    "enabled": true,
    "language": "zh-CN",
    "tone": "technical",
    "includeRecommendations": true,
    "focusAreas": ["performance", "reliability", "scalability"]
  }
}

Security scan verdicts — ClawScan: clean | static analysis: suspicious | VirusTotal: clean | SkillSpector: clean (LOW)