Skill: ljw-git-dw/dataworks-diagnoser (version 1.0.0)

--- name: dataworks-diagnoser description: DataWorks task instance log fetcher and diagnostician. Use when: user needs to troubleshoot failed DataWorks task instances, analyze error logs, get diagnostic recommendations for task failures, or understand why a DataWorks node failed. Requires Alibaba Cloud credentials. Supports all DataWorks task types including ODPS SQL, Sync, Shell, Python, etc. metadata: { "openclaw": { "emoji": "🔍", "requires": { "bins": ["curl", "python3"] }, "install": [ { "id": "brew", "kind": "brew", "formula": "curl", "bins": ["curl"], "label": "Install curl (brew)", }, ], }, } --- # DataWorks Task Instance Diagnostician Fetches task instance logs from Alibaba Cloud DataWorks API and provides intelligent diagnostic recommendations. ## Quick Start **Diagnose a failed task:** ```bash python3 scripts/dataworks_diagnose.py <instance_id> ``` **Example:** ```bash python3 scripts/dataworks_diagnose.py 123456789 ``` ## When to Use ✅ **USE this skill when:** - DataWorks task instance failed and you need to know why - You have an instance ID and need to fetch error logs - You want automated diagnosis and solutions for task failures - Troubleshooting ODPS SQL, Data Integration, Shell, Python nodes - Need to analyze error patterns across multiple failures - Preparing incident reports for failed tasks ## When NOT to Use ❌ **DON'T use this skill when:** - You need real-time task monitoring (use DataWorks console) - You want to modify task configurations (use console or API directly) - You need historical analytics across many tasks (use DataWorks reports) - The task is still running (wait for completion first) - You don't have Alibaba Cloud credentials (need AccessKey) ## Prerequisites ### 1. Alibaba Cloud Credentials One of the following is required: **Option A: Environment Variables (Recommended)** ```bash export ALIBABA_CLOUD_ACCESS_KEY_ID=your_access_key export ALIBABA_CLOUD_ACCESS_KEY_SECRET=your_access_secret ``` **Option B: Config File** Create `~/.alibabacloud/credentials`: ```json { "access_key_id": "your_access_key", "access_key_secret": "your_access_secret" } ``` **Option C: Aliyun CLI Config** If you have Aliyun CLI configured, credentials will be loaded automatically. ### 2. Required Permissions The AccessKey needs these permissions: - `dataworks:GetInstanceLog` - Fetch task instance logs - `dataworks:QueryTask` - Query task information ### 3. Network Access - Access to Alibaba Cloud API endpoints - If using VPC, ensure proper network configuration ## Core Workflows ### 1. Quick Diagnosis (Recommended) Fetch log and get diagnosis in one command: ```bash python3 scripts/dataworks_diagnose.py <instance_id> ``` **Example:** ```bash python3 scripts/dataworks_diagnose.py 123456789 ``` **Output:** ``` 🔍 开始诊断 DataWorks 任务实例：123456789 📍 区域：cn-hangzhou ------------------------------------------------------------ 📥 步骤 1/2: 获取任务日志... ✅ 日志获取成功 🔬 步骤 2/2: 分析诊断中... ✅ 诊断完成 ============================================================ 📋 诊断报告 ============================================================ 🔍 DataWorks 任务实例诊断报告 ============================================================ 实例 ID: 123456789 发现问题数：2 ---------------------------------------------------------------------- 🔴 问题 1: 资源配额不足 类型：resource_quota 严重程度：HIGH 相关日志: > ERROR: quota exceeded for resource group 'default' > No available slots in queue 建议解决方案: 1. 检查当前资源组的使用情况，释放闲置资源 2. 联系管理员提升资源配额 3. 优化任务配置，减少资源消耗 4. 考虑错峰调度，避开资源使用高峰 参考文档：https://help.aliyun.com/.../resource-group.html ``` ### 2. Fetch Log Only ```bash python3 scripts/fetch_instance_log.py <instance_id> [options] ``` **Options:** ```bash # Specify region python3 scripts/fetch_instance_log.py 123456789 --region cn-shanghai # Output as JSON python3 scripts/fetch_instance_log.py 123456789 --json # Show full log (default: last 50 lines) python3 scripts/fetch_instance_log.py 123456789 --verbose # Save to file python3 scripts/fetch_instance_log.py 123456789 > log.txt ``` ### 3. Diagnose Existing Log ```bash python3 scripts/diagnose_log.py <log_file> ``` **Examples:** ```bash # From file python3 scripts/diagnose_log.py error.log # From stdin cat log.txt | python3 scripts/diagnose_log.py # With instance ID python3 scripts/diagnose_log.py error.log --instance-id 123456789 # JSON output python3 scripts/diagnose_log.py error.log --json # Summary only python3 scripts/diagnose_log.py error.log --summary ``` ## Scripts This skill includes three scripts: ### `dataworks_diagnose.py` - All-in-One Tool Fetches log and provides diagnosis automatically. **Usage:** ```bash python3 scripts/dataworks_diagnose.py <instance_id> [options] ``` **Options:** - `--region, -r` - Alibaba Cloud region (default: cn-hangzhou) - `--json, -j` - Output as JSON - `--verbose, -v` - Show full log - `--save-log FILE` - Save raw log to file - `--save-report FILE` - Save diagnostic report to file ### `fetch_instance_log.py` - Log Fetcher Fetches task instance log from DataWorks API. **Usage:** ```bash python3 scripts/fetch_instance_log.py <instance_id> [options] ``` **Options:** - `--region, -r` - Region (default: cn-hangzhou) - `--access-key` - Access Key ID - `--access-secret` - Access Key Secret - `--json, -j` - JSON output - `--verbose, -v` - Full log ### `diagnose_log.py` - Log Analyzer Analyzes log content and provides diagnostic recommendations. **Usage:** ```bash python3 scripts/diagnose_log.py <log_file_or_stdin> [options] ``` **Options:** - `--instance-id` - Task instance ID - `--json, -j` - JSON output - `--summary, -s` - Summary only ## Detected Error Patterns The diagnostician recognizes these error types: | Error Type | Severity | Examples | |------------|----------|----------| | 🔴 resource_quota | High | "quota exceeded", "资源不足" | | 🔴 resource_expired | High | "expired", "独享资源组已过期", "bill exception" | | 🔴 connection_timeout | High | "connection timeout", "network unreachable" | | 🔴 permission_denied | High | "permission denied", "access denied" | | 🟡 syntax_error | Medium | "syntax error", "parse error" | | 🟡 table_not_found | Medium | "table not found", "doesn't exist" | | 🟡 data_quality | Medium | "quality check failed" | | 🔴 memory_overflow | High | "out of memory", "heap space" | | 🔴 disk_full | High | "disk full", "no space left" | | 🟡 dependency_failed | Medium | "dependency failed", "upstream failed" | | 🟡 api_rate_limit | Medium | "rate limit exceeded" | See `references/error_codes.md` for detailed error patterns and solutions. ## Common Regions | Region | Code | |--------|------| | 华东 1 (杭州) | cn-hangzhou | | 华东 2 (上海) | cn-shanghai | | 华北 1 (青岛) | cn-qingdao | | 华北 2 (北京) | cn-beijing | | 华南 1 (深圳) | cn-shenzhen | | 香港 | cn-hongkong | | 新加坡 | ap-southeast-1 | ## API Reference **API:** GetTaskInstanceLog **Version:** 2024-05-18 **Endpoint:** `https://dataworks-public.{region}.aliyuncs.com/` **Request Parameters:** - `InstanceId` (required) - Task instance ID - `RegionId` (required) - Region ID **Response:** ```json { "Data": { "LogContent": "...", "InstanceStatus": "FAILED", "CycleTime": "2024-01-15 10:30:00" }, "Code": "200" } ``` **Documentation:** https://api.aliyun.com/api/dataworks-public/2024-05-18/GetTaskInstanceLog ## Examples ### Example 1: Quick Diagnosis ```bash python3 scripts/dataworks_diagnose.py 123456789 ``` ### Example 2: Save Report ```bash python3 scripts/dataworks_diagnose.py 123456789 --save-report diagnosis.txt ``` ### Example 3: Different Region ```bash python3 scripts/dataworks_diagnose.py 123456789 --region cn-shanghai ``` ### Example 4: Analyze Saved Log ```bash python3 scripts/diagnose_log.py saved_log.txt --instance-id 123456789 ``` ### Example 5: Batch Analysis ```bash for id in 123 456 789; do python3 scripts/diagnose_log.py --instance-id $id < log_$id.txt done ``` ## Troubleshooting ### "Credentials not found" ```bash # Set environment variables export ALIBABA_CLOUD_ACCESS_KEY_ID=your_key export ALIBABA_CLOUD_ACCESS_KEY_SECRET=your_secret ``` ### "Instance not found" - Verify the instance ID is correct - Check if the instance exists in DataWorks console - Ensure you're using the correct region ### "Permission denied" - Verify AccessKey has required permissions - Check RAM role configuration - Contact administrator for access ### "Request timeout" - Check network connectivity - Try increasing timeout in script - Verify API endpoint is accessible ## Tips 💡 **Pro tips:** 1. **Save logs for failed tasks** - Use `--save-log` to keep records 2. **Generate reports** - Use `--save-report` for documentation 3. **Batch processing** - Script supports multiple instance IDs 4. **JSON output** - Use `--json` for programmatic processing 5. **Region matters** - Always use the correct region for your workspace ## Security ⚠️ **Important:** - Never commit AccessKeys to version control - Use RAM roles instead of main account keys - Rotate keys regularly - Use environment variables or secure config files - Restrict key permissions to minimum required ## References - `references/error_codes.md` - Complete error code reference - [DataWorks Documentation](https://help.aliyun.com/product/27728.html) - [API Reference](https://api.aliyun.com/api/dataworks-public) - [Error Codes](https://help.aliyun.com/document_detail/dataworks/error-codes.html)

path: README.md
content: # DataWorks Diagnoser 🔍

DataWorks 任务实例智能诊断工具 - 自动获取日志、分析错误、提供解决方案

## 功能特性

- ✅ **自动获取日志** - 通过阿里云 API 获取任务实例日志
- ✅ **智能错误识别** - 自动提取 ODPS、DataX、Java 异常等错误代码
- ✅ **AI 智能分析** - 分析错误原因，提供解决方案
- ✅ **明确的行动指引** - 每个错误都有"下一步"操作建议
- ✅ **简洁清晰的报告** - 无冗余信息，直击问题核心

## 支持的错误类型

| 类型 | 错误代码示例 | 自动识别 |
|------|----------|----------|
| ODPS SQL | `ODPS-0130071` 函数不存在 | ✅ |
| ODPS SQL | `ODPS-0130141` 类型转换错误 | ✅ |
| DataX | `MYSQLErrCode-02` 连接失败 | ✅ |
| Java | `NullPointerException` | ✅ |
| 通用错误 | `ERROR`, `FAILED` | ✅ |

## 安装方法

### 方法 1: OpenClaw Skill（推荐）

```bash
# 1. 下载 skill 包
# 将 dataworks-diagnoser.skill 放到 skills 目录
unzip dataworks-diagnoser.skill -d ~/.openclaw/skills/

# 2. 重启 OpenClaw 或重新加载 skills
openclaw gateway restart
```

### 方法 2: 直接使用脚本

```bash
# 1. 克隆或下载源码
git clone <repo-url>
cd dataworks-diagnoser

# 2. 安装依赖
pip3 install alibabacloud_dataworks_public20240518 alibabacloud_credentials

# 3. 配置阿里云 AK
export ALIBABA_CLOUD_ACCESS_KEY_ID=LTAI5t...
export ALIBABA_CLOUD_ACCESS_KEY_SECRET=xxxxxx

# 4. 运行诊断
python3 scripts/dataworks_diagnose.py <实例 ID>
```

## 使用方法

### 基础用法

```bash
# 配置阿里云 AccessKey
export ALIBABA_CLOUD_ACCESS_KEY_ID=你的 AK
export ALIBABA_CLOUD_ACCESS_KEY_SECRET=你的密钥

# 诊断任务实例
python3 scripts/dataworks_diagnose.py 1004057140616
```

### 输出示例

```
🔍 开始诊断 DataWorks 任务实例：1004057140616
📍 区域：cn-hangzhou
------------------------------------------------------------

📥 步骤 1/2: 获取任务日志...
✅ 日志获取成功
📋 RequestId: B7719DED-6DD1-5ACA-B3EC-A7CE943131DB

📊 任务摘要:
   ❌ FAILED: ODPS-0130071:[11,10] Semantic analysis exception

🔬 步骤 2/2: AI 智能分析中...
✅ 诊断完成

======================================================================
DataWorks 任务诊断报告
======================================================================
实例 ID: 1004057140616
发现错误：2 个

错误 1: ODPS - ODPS-0130071
错误信息:
  Semantic analysis exception - function or view 'bound_index' cannot be resolved
  位置：line 11, col 10

可能原因:
  • SQL 中使用了不存在的函数或视图

解决方案:
  1. 检查 SQL 第 line 11, col 10 处的函数
  2. 确认函数名称拼写正确
  3. 在 DataWorks 数据开发中测试该函数

下一步：在 DataWorks 数据开发面板试运行 SQL
======================================================================
```

### 高级用法

```bash
# 指定区域
python3 scripts/dataworks_diagnose.py 1004057140616 --region cn-shanghai

# 输出 JSON 格式
python3 scripts/dataworks_diagnose.py 1004057140616 --json

# 只分析日志文件（不需要 AK）
python3 scripts/analyze_error.py error.log

# 从管道读取日志
cat task.log | python3 scripts/analyze_error.py
```

## 前置要求

### 1. 阿里云 AccessKey

需要配置阿里云 AccessKey 才能获取任务日志：

**获取 AK：**
1. 登录 [阿里云 RAM 控制台](https://ram.console.aliyun.com/manage/ak)
2. 创建 AccessKey 或使用现有的
3. 记录 `AccessKey ID` 和 `AccessKey Secret`

**配置 AK（3 选 1）：**

```bash
# 方式 1: 环境变量（推荐）
export ALIBABA_CLOUD_ACCESS_KEY_ID=LTAI5t...
export ALIBABA_CLOUD_ACCESS_KEY_SECRET=xxxxxx

# 方式 2: 配置文件
cat > ~/.alibabacloud/credentials << EOF
{
  "access_key_id": "LTAI5t...",
  "access_key_secret": "xxxxxx"
}
EOF

# 方式 3: 命令行参数
python3 scripts/dataworks_diagnose.py <实例 ID> \
  --access-key LTAI5t... \
  --access-secret xxxxxx
```

### 2. RAM 权限

AccessKey 需要以下权限：

```json
{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dataworks:GetTaskInstanceLog",
        "dataworks:QueryTask"
      ],
      "Resource": "*"
    }
  ]
}
```

### 3. Python 依赖

```bash
pip3 install alibabacloud_dataworks_public20240518 alibabacloud_credentials
```

## 文件结构

```
dataworks-diagnoser/
├── scripts/
│   ├── dataworks_diagnose.py      # 主程序：获取日志 + 分析
│   ├── fetch_instance_log.py      # 日志获取（官方 SDK）
│   ├── analyze_error.py           # 错误分析
│   └── diagnose_log.py            # 旧版诊断（兼容）
├── test_logs/                     # 测试日志样本
├── references/
│   └── error_codes.md             # 错误代码参考
├── AK_SETUP.md                    # AK 配置指南
├── QUICK_START.md                 # 快速开始
└── README.md                      # 本文档
```

## 故障排查

### "Credentials not found"

**原因：** 未配置 AK

**解决：**
```bash
export ALIBABA_CLOUD_ACCESS_KEY_ID=你的 AK
export ALIBABA_CLOUD_ACCESS_KEY_SECRET=你的密钥
```

### "InvalidAccessKeyId"

**原因：** AK ID 错误或已禁用

**解决：**
- 检查 AK ID 是否正确
- 在 RAM 控制台确认 AK 状态为"启用"

### "Forbidden"

**原因：** 权限不足

**解决：**
- 在 RAM 控制台授予 `dataworks:GetTaskInstanceLog` 权限

### "Instance not found"

**原因：** 实例 ID 错误或区域不匹配

**解决：**
- 确认实例 ID 正确
- 使用 `--region` 指定正确区域

## 安全提醒

⚠️ **重要安全事项：**

1. **不要使用主账号 AK** - 创建 RAM 子账号
2. **不要提交 AK 到 Git** - 使用 `.gitignore`
3. **定期轮换 AK** - 建议 90 天更换
4. **最小权限原则** - 只授予必要的 API 权限

## 常见问题

### Q: 支持哪些区域？

A: 支持所有阿里云区域，默认 `cn-hangzhou`。使用 `--region` 指定：
```bash
python3 scripts/dataworks_diagnose.py <实例 ID> --region cn-shanghai
```

### Q: 可以诊断历史任务吗？

A: 可以，只要任务日志还在阿里云保留期内（通常 30 天）。

### Q: 支持批量诊断吗？

A: 当前版本支持单个实例诊断。批量诊断可以写脚本循环调用：
```bash
for id in 1004057140616 1004057140617 1004057140618; do
  python3 scripts/dataworks_diagnose.py $id
done
```

### Q: 如何反馈问题？

A: 请提供：
- 实例 ID
- 完整错误信息
- 期望的行为

## 版本历史

| 版本 | 日期 | 更新内容 |
|------|------|----------|
| 1.0 | 2024-01-15 | 初始版本 |
| 1.1 | 2024-01-15 | 添加 ODPS 错误识别 |
| 1.2 | 2024-01-15 | 简化输出，移除假链接 |
| 1.3 | 2024-01-15 | 添加下一步行动指引 |

## 许可证

MIT License

## 联系方式

- GitHub: [你的 GitHub 链接]
- 邮箱：[你的邮箱]
- 阿里云市场：[产品链接]

---

**让 DataWorks 故障排查更简单！** 🚀


path: references/error_codes.md
content: # DataWorks 常见错误代码与解决方案

本参考文档收录了 DataWorks 任务执行中常见的错误类型及其解决方案。

---

## 错误分类索引

| 分类 | 错误类型 | 严重程度 |
|------|----------|----------|
| 🔴 资源类 | 资源配额不足、内存溢出、磁盘空间不足 | 高 |
| 🔴 网络类 | 连接超时、网络不可达、白名单问题 | 高 |
| 🔴 权限类 | 权限不足、认证失败、AK 无效 | 高 |
| 🟡 代码类 | SQL 语法错误、表不存在、字段错误 | 中 |
| 🟡 数据类 | 数据质量检查失败、空值约束违反 | 中 |
| 🟡 依赖类 | 上游任务失败、依赖缺失 | 中 |
| 🟢 限流类 | API 调用超限、QPS 限制 | 中 |

---

## 🔴 资源类错误

### 1. 资源配额不足

**错误特征：**
```
ERROR: quota exceeded
ERROR: resource insufficient
ERROR: no available resource in queue
```

**常见原因：**
- 资源组使用率已达上限
- 并发任务数超过配额
- 资源组配置过小

**解决方案：**
1. 查看资源组使用情况
   ```bash
   # 在 DataWorks 控制台查看资源组监控
   ```
2. 释放闲置资源
   - 停止不必要的测试任务
   - 优化长期运行的任务
3. 提升配额
   - 联系管理员调整资源组配置
   - 升级资源组规格
4. 错峰调度
   - 调整任务调度时间，避开高峰

**参考文档：**
https://help.aliyun.com/document_detail/dataworks/resource-group.html

---

### 2. 内存溢出 (OOM)

**错误特征：**
```
ERROR: OutOfMemoryError
ERROR: Java heap space
ERROR: GC overhead limit exceeded
ERROR: Container killed by YARN for exceeding memory limits
```

**常见原因：**
- 数据处理量过大
- SQL 查询未优化，扫描全表
- 数据倾斜导致单点内存不足

**解决方案：**
1. 增加内存配置
   ```
   任务配置 → 资源设置 → 增加内存 (如 4GB → 8GB)
   ```
2. 优化 SQL
   ```sql
   -- 添加分区裁剪
   SELECT * FROM table WHERE ds = '20240115'
   
   -- 避免 SELECT *
   SELECT col1, col2 FROM table
   
   -- 使用近似计算
   SELECT APPROX_COUNT_DISTINCT(user_id) FROM table
   ```
3. 处理数据倾斜
   ```sql
   -- 添加随机前缀打散数据
   SELECT /*+ MAPJOIN(small_table) */ * FROM large_table JOIN small_table
   ```
4. 分批处理
   - 将大任务拆分为多个小任务
   - 使用增量处理替代全量

**参考文档：**
https://help.aliyun.com/document_detail/dataworks/performance-tuning.html

---

### 3. 磁盘空间不足

**错误特征：**
```
ERROR: No space left on device
ERROR: Disk quota exceeded
ERROR: Storage exhausted
```

**解决方案：**
1. 清理临时文件
2. 检查输出表数据量是否异常
3. 增加磁盘配额
4. 优化任务减少中间数据

---

## 🔴 网络类错误

### 1. 连接超时

**错误特征：**
```
ERROR: Connection timeout
ERROR: Connect timed out
ERROR: Network unreachable
ERROR: Host unreachable
```

**常见原因：**
- 数据源服务器不可达
- 防火墙阻挡
- 白名单未配置
- VPC 网络不通

**解决方案：**
1. 检查网络连通性
   ```bash
   ping your-database-host.com
   telnet your-database-host.com 3306
   ```
2. 配置白名单
   - 添加 DataWorks 服务器 IP 到数据源白名单
   - 参考：https://help.aliyun.com/document_detail/dataworks/ip-ranges.html
3. 检查 VPC 配置
   - 确保 DataWorks 和数据源在同一 VPC
   - 或配置 VPC 对等连接
4. 增加超时时间
   ```
   任务配置 → 高级设置 → 连接超时 (如 30s → 60s)
   ```

---

### 2. 白名单问题

**错误特征：**
```
ERROR: Access denied for IP
ERROR: Host not in whitelist
ERROR: Connection rejected by whitelist
```

**解决方案：**
1. 获取 DataWorks IP 段
   - 查看官方文档 IP 列表
   - 或在控制台查看当前工作区 IP
2. 添加 IP 到白名单
   - 登录数据源管理控制台
   - 添加 DataWorks IP 到访问白名单
3. 使用 VPC 网络（推荐）
   - 配置 VPC 绑定
   - 避免公网传输

---

## 🔴 权限类错误

### 1. 权限不足

**错误特征：**
```
ERROR: Permission denied
ERROR: Access denied
ERROR: Unauthorized
ERROR: Authentication failed
```

**常见原因：**
- 数据源账号权限不足
- RAM 角色未授权
- AccessKey 无效或过期

**解决方案：**
1. 检查账号权限
   ```sql
   -- MySQL 示例
   SHOW GRANTS FOR 'username'@'%';
   ```
2. 验证 AccessKey
   - 在 RAM 控制台检查 AK 状态
   - 确保 AK 有 DataWorks 相关权限
3. 配置 RAM 角色
   - 创建 RAM 角色
   - 授予必要权限
   - 在 DataWorks 中绑定角色

**权限清单：**
| 操作 | 所需权限 |
|------|----------|
| 读表 | SELECT |
| 写表 | INSERT, CREATE |
| 删表 | DROP |
| 执行存储过程 | EXECUTE |

**参考文档：**
https://help.aliyun.com/document_detail/dataworks/permission.html

---

## 🟡 代码类错误

### 1. SQL 语法错误

**错误特征：**
```
ERROR: Syntax error near 'xxx'
ERROR: Parse error at line X
ERROR: Invalid syntax
```

**解决方案：**
1. 使用语法检查工具
2. 检查关键字拼写
3. 验证表名、字段名
4. 检查特殊字符转义

**常见语法错误示例：**
```sql
-- 错误：缺少引号
SELECT * FROM table WHERE name = John

-- 正确
SELECT * FROM table WHERE name = 'John'

-- 错误：关键字未转义
SELECT order FROM table

-- 正确
SELECT `order` FROM table
```

---

### 2. 表/字段不存在

**错误特征：**
```
ERROR: Table 'xxx' doesn't exist
ERROR: Column 'xxx' not found
ERROR: Relation does not exist
```

**解决方案：**
1. 确认表名拼写
2. 检查数据库/Schema
3. 验证表是否被删除
4. 检查分区是否存在

---

## 🟡 数据类错误

### 1. 数据质量检查失败

**错误特征：**
```
ERROR: Data quality check failed
ERROR: Quality rule violation
ERROR: Null value not allowed
```

**常见原因：**
- 源数据存在空值
- 数据类型不匹配
- 违反唯一性约束

**解决方案：**
1. 检查数据质量规则配置
2. 分析源数据异常值
3. 添加数据清洗步骤
4. 配置容错处理

---

## 🟡 依赖类错误

### 1. 上游任务失败

**错误特征：**
```
ERROR: Dependency failed
ERROR: Upstream node failed
ERROR: Parent node not completed
```

**解决方案：**
1. 检查上游节点状态
2. 查看上游错误日志
3. 调整依赖配置
4. 添加重试机制

---

## 🟢 限流类错误

### 1. API 调用超限

**错误特征：**
```
ERROR: Rate limit exceeded
ERROR: Too many requests
ERROR: QPS limit reached
```

**解决方案：**
1. 降低调用频率
2. 添加重试退避
   ```python
   import time
   for i in range(3):
       try:
           call_api()
           break
       except RateLimitError:
           time.sleep(2 ** i)  # 指数退避
   ```
3. 申请提升配额
4. 使用批量接口

---

## 故障排查流程

```
1. 查看错误日志
   ↓
2. 识别错误类型（参考本手册）
   ↓
3. 根据解决方案逐一尝试
   ↓
4. 验证修复效果
   ↓
5. 如未解决，收集以下信息联系支持：
   - 完整错误日志
   - 任务配置截图
   - 相关资源监控数据
```

---

## 快速诊断命令

```bash
# 检查资源使用
dataworks-cli resource status

# 检查网络连接
dataworks-cli network check --target <datasource>

# 检查权限
dataworks-cli permission check --user <username>

# 获取任务日志
dataworks-cli task log --instance <instance_id>
```

---

## 有用的链接

- [DataWorks 官方文档](https://help.aliyun.com/product/27728.html)
- [常见问题 FAQ](https://help.aliyun.com/document_detail/dataworks/faq.html)
- [错误码大全](https://help.aliyun.com/document_detail/dataworks/error-codes.html)
- [社区论坛](https://developer.aliyun.com/group/dataworks)

---

## 更新记录

| 日期 | 更新内容 |
|------|----------|
| 2024-01-15 | 初始版本 |


path: requirements.txt
content: # DataWorks Diagnoser - Python Dependencies

# Alibaba Cloud SDK (required for API access)
alibabacloud_dataworks_public20240518>=1.0.0
alibabacloud_credentials>=0.3.0

# Optional: for HTTP requests to official docs
# requests>=2.28.0

# Optional: for HTML parsing
# beautifulsoup4>=4.11.0


path: scripts/diagnose_log.py
content: #!/usr/bin/env python3
"""
DataWorks Task Instance Log Diagnostician

Analyzes task instance logs and provides diagnostic recommendations.

Usage:
    python3 diagnose_log.py <log_file_or_text> [options]
    cat log.txt | python3 diagnose_log.py
    python3 diagnose_log.py --instance-id 12345678

Examples:
    python3 diagnose_log.py error.log
    python3 diagnose_log.py --instance-id 12345678
    cat log.txt | python3 diagnose_log.py --json
"""

import argparse
import json
import re
import sys
from pathlib import Path


# Common error patterns and solutions
ERROR_PATTERNS = {
    'resource_quota': {
        'patterns': [
            r'quota exceeded',
            r'resource.*insufficient',
            r'no.*resource',
            r'queue.*full',
            r'max.*limit',
            r'资源.*不足',
            r'配额.*超限',
        ],
        'severity': 'high',
        'title': '资源配额不足',
        'solutions': [
            '检查当前资源组的使用情况，释放闲置资源',
            '联系管理员提升资源配额',
            '优化任务配置，减少资源消耗',
            '考虑错峰调度，避开资源使用高峰',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/resource-group.html',
    },
    'resource_expired': {
        'patterns': [
            r'独享资源组.*过期',
            r'资源组.*过期',
            r'bill.*exception',
            r'checking.*bill',
            r'checkexclusivebillavailable',
            r'任务实例.*欠费',
            r'account.*overdue',
        ],
        'severity': 'high',
        'title': '资源组已过期/欠费',
        'solutions': [
            '登录阿里云控制台，检查独享资源组的计费状态',
            '资源组已过期，需要续费或重新购买',
            '检查账户余额，确保有足够资金',
            '如已续费，等待 5-10 分钟后重试任务',
            '联系阿里云客服确认资源组状态',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/billing.html',
    },
    'connection_timeout': {
        'patterns': [
            r'connection.*timeout',
            r'timeout.*connect',
            r'network.*unreachable',
            r'host.*unreachable',
        ],
        'severity': 'high',
        'title': '网络连接超时',
        'solutions': [
            '检查数据源网络连接是否正常',
            '验证白名单配置，确保 DataWorks IP 已添加',
            '检查防火墙规则',
            '增加连接超时时间配置',
            '使用 VPC 网络时，检查网络连通性',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/network.html',
    },
    'permission_denied': {
        'patterns': [
            r'permission.*denied',
            r'access.*denied',
            r'unauthorized',
            r'authentication.*failed',
            r'invalid.*credential',
        ],
        'severity': 'high',
        'title': '权限不足',
        'solutions': [
            '检查数据源账号权限是否足够',
            '验证 AccessKey 是否有效',
            '确认 RAM 角色授权配置',
            '检查数据源白名单设置',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/permission.html',
    },
    'syntax_error': {
        'patterns': [
            r'syntax.*error',
            r'parse.*error',
            r'invalid.*syntax',
            r'line \d+.*error',
        ],
        'severity': 'medium',
        'title': 'SQL/代码语法错误',
        'solutions': [
            '检查 SQL 语法是否正确',
            '验证表名、字段名是否存在',
            '检查特殊字符是否需要转义',
            '使用语法检查工具预验证',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/sql-reference.html',
    },
    'odps_error': {
        'patterns': [
            r'odps-\d+:\[\d+,\d+\]',
            r'semantic.*analysis.*exception',
            r'function.*cannot.*be.*resolved',
            r'view.*cannot.*be.*resolved',
            r'illegal.*implicit.*type.*cast',
            r'table.*does.*not.*exist',
            r'column.*ambiguous',
            r'odpscmd.*failed',
            r'FAILED: ODPS',
        ],
        'severity': 'high',
        'title': 'ODPS SQL 执行失败',
        'solutions': [
            '检查 SQL 中使用的函数是否存在（如 BOUND_INDEX）',
            '验证表名、字段名是否正确',
            '检查数据类型转换是否合法',
            '查看 Log View 获取详细错误信息',
            '在 DataWorks 数据开发中先试运行 SQL',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/odps-sql.html',
    },
    'null_pointer': {
        'patterns': [
            r'nullpointerexception',
            r'null.*pointer',
            r'should not be (blank|null|empty)',
            r'参数.*为空',
            r'缺少.*参数',
        ],
        'severity': 'high',
        'title': '空指针/参数缺失',
        'solutions': [
            '检查任务配置中的必填参数是否已填写',
            '验证数据源连接信息（用户名、密码、URL）是否完整',
            '检查调度参数是否配置正确',
            '查看节点配置，确认所有必填字段都已填写',
            '如果是自定义脚本，检查代码中的空值处理',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/node-configuration.html',
    },
    'missing_access_key': {
        'patterns': [
            r'di_serv_render_006',
            r'没有指定.*accessid',
            r'没有指定.*accesskey',
            r'accessid.*not.*specified',
            r'accesskey.*not.*specified',
            r'writer 插件.*access',
        ],
        'severity': 'high',
        'title': '数据集成缺少 AK 配置',
        'solutions': [
            '检查数据同步任务的 writer 端配置',
            '在 Holo 写入插件中填写 AccessId 和 AccessKey',
            '如使用数据源，检查数据源配置中的 AK 信息',
            '建议使用 RAM 子账号的 AK，不要用主账号',
            '确认 AK 有 Hologres 的写入权限',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/data-integration.html',
    },
    'db_connection_failed': {
        'patterns': [
            r'communications.*link.*failure',
            r'unknown.*host',
            r'cannot.*connect.*database',
            r'connection.*refused',
            r'mysqlerrcode',
            r'datax.*无法连接',
            r'name.*or.*service.*not.*known',
            r'dbutil.*test.*connection.*failed',
        ],
        'severity': 'high',
        'title': '数据库连接失败',
        'solutions': [
            '检查 JDBC URL 是否正确（IP、端口、数据库名）',
            '验证域名是否可以解析（DNS 问题）',
            '检查 RDS 实例是否正在运行',
            '确认白名单配置，添加 DataWorks IP 到 RDS 白名单',
            '检查网络类型（VPC/公网）是否匹配',
            '验证账号密码是否正确',
            '如果是 VPC 网络，确保 DataWorks 和 RDS 在同一 VPC',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/database-connection.html',
    },
    'table_not_found': {
        'patterns': [
            r'table.*not.*found',
            r"table.*doesn't.*exist",
            r'object.*not.*found',
            r'relation.*does not exist',
        ],
        'severity': 'medium',
        'title': '表不存在',
        'solutions': [
            '确认表名拼写是否正确',
            '检查表是否在指定数据库中',
            '验证表是否已被删除或重命名',
            '检查分区表分区是否存在',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/table-management.html',
    },
    'data_quality': {
        'patterns': [
            r'data.*quality.*check.*failed',
            r'quality.*rule.*violation',
            r'null.*not.*allowed',
            r'constraint.*violation',
        ],
        'severity': 'medium',
        'title': '数据质量问题',
        'solutions': [
            '检查数据质量规则配置',
            '分析源数据是否存在异常值',
            '验证数据类型是否匹配',
            '检查是否有空值违反约束',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/data-quality.html',
    },
    'memory_overflow': {
        'patterns': [
            r'out.*of.*memory',
            r'memory.*overflow',
            r'heap.*space',
            r'gc.*overhead',
        ],
        'severity': 'high',
        'title': '内存溢出',
        'solutions': [
            '增加任务内存配置',
            '优化 SQL，减少数据处理量',
            '使用分区裁剪减少扫描数据',
            '检查是否有数据倾斜',
            '考虑分批次处理数据',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/performance-tuning.html',
    },
    'disk_full': {
        'patterns': [
            r'disk.*full',
            r'no.*space.*left',
            r'storage.*exhausted',
        ],
        'severity': 'high',
        'title': '磁盘空间不足',
        'solutions': [
            '清理临时文件和历史数据',
            '检查输出表数据量是否异常',
            '增加磁盘配额',
            '优化任务减少中间数据',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/storage.html',
    },
    'dependency_failed': {
        'patterns': [
            r'dependency.*failed',
            r'upstream.*failed',
            r'parent.*node.*failed',
        ],
        'severity': 'medium',
        'title': '上游依赖失败',
        'solutions': [
            '检查上游节点运行状态',
            '查看上游节点的错误日志',
            '调整依赖关系配置',
            '考虑添加重试机制',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/dependency.html',
    },
    'api_rate_limit': {
        'patterns': [
            r'rate.*limit.*exceeded',
            r'too.*many.*requests',
            r'qps.*limit.*exceeded',
            r'api.*throttl',
            r'request.*rate.*limit',
        ],
        'severity': 'medium',
        'title': 'API 调用限流',
        'solutions': [
            '降低 API 调用频率',
            '添加重试和退避机制',
            '申请提升 API 配额',
            '使用批量接口替代单次调用',
        ],
        'docs': 'https://help.aliyun.com/document_detail/dataworks/api-limits.html',
    },
}

# Error severity levels
SEVERITY_ORDER = {'high': 0, 'medium': 1, 'low': 2}


def analyze_log(log_text):
    """
    Analyze log text and identify error patterns
    
    Returns list of detected issues with recommendations
    """
    issues = []
    log_lower = log_text.lower()
    
    for error_type, config in ERROR_PATTERNS.items():
        matched = False
        matched_pattern = None
        
        for pattern in config['patterns']:
            if re.search(pattern, log_lower, re.IGNORECASE):
                matched = True
                matched_pattern = pattern
                break
        
        if matched:
            # Extract relevant log lines
            relevant_lines = []
            for line in log_text.split('\n'):
                if re.search(matched_pattern, line, re.IGNORECASE):
                    relevant_lines.append(line.strip())
            
            issues.append({
                'type': error_type,
                'title': config['title'],
                'severity': config['severity'],
                'matched_pattern': matched_pattern,
                'relevant_lines': relevant_lines[:5],  # Top 5 matching lines
                'solutions': config['solutions'],
                'docs': config['docs'],
            })
    
    # Sort by severity
    issues.sort(key=lambda x: SEVERITY_ORDER.get(x['severity'], 99))
    
    return issues


def generate_report(issues, instance_id=None):
    """Generate diagnostic report"""
    output = []
    
    output.append("=" * 70)
    output.append("🔍 DataWorks 任务实例诊断报告")
    output.append("=" * 70)
    
    if instance_id:
        output.append(f"实例 ID: {instance_id}")
    
    output.append(f"发现问题数：{len(issues)}")
    output.append("")
    
    if not issues:
        output.append("✅ 未发现已知错误模式")
        output.append("")
        output.append("建议:")
        output.append("  - 检查业务逻辑是否正确")
        output.append("  - 查看完整日志寻找其他错误信息")
        output.append("  - 联系技术支持获取帮助")
    else:
        for i, issue in enumerate(issues, 1):
            severity_icon = {'high': '🔴', 'medium': '🟡', 'low': '🟢'}.get(issue['severity'], '⚪')
            
            output.append("-" * 70)
            output.append(f"{severity_icon} 问题 {i}: {issue['title']}")
            output.append(f"   类型：{issue['type']}")
            output.append(f"   严重程度：{issue['severity'].upper()}")
            output.append("")
            
            if issue['relevant_lines']:
                output.append("   相关日志:")
                for line in issue['relevant_lines'][:3]:
                    output.append(f"     > {line[:100]}")
                output.append("")
            
            output.append("   建议解决方案:")
            for j, solution in enumerate(issue['solutions'], 1):
                output.append(f"     {j}. {solution}")
            output.append("")
            output.append(f"   参考文档：{issue['docs']}")
            output.append("")
    
    output.append("=" * 70)
    output.append("💡 提示：根据上述建议逐一排查，通常可以解决 90% 的常见问题")
    output.append("=" * 70)
    
    return '\n'.join(output)


def main():
    parser = argparse.ArgumentParser(
        description="Diagnose DataWorks task instance log",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  %(prog)s error.log
  %(prog)s --instance-id 12345678
  cat log.txt | %(prog)s
  %(prog)s --json < log.txt
        """
    )
    
    parser.add_argument("log_file", nargs='?', help="Log file to analyze")
    parser.add_argument("--instance-id", help="Task instance ID")
    parser.add_argument("--json", "-j", action="store_true",
                       help="Output as JSON")
    parser.add_argument("--summary", "-s", action="store_true",
                       help="Show summary only")
    
    args = parser.parse_args()
    
    # Read log content
    if args.log_file:
        try:
            log_text = Path(args.log_file).read_text()
        except Exception as e:
            print(f"Error reading file: {e}", file=sys.stderr)
            sys.exit(1)
    elif not sys.stdin.isatty():
        # Read from stdin
        log_text = sys.stdin.read()
    else:
        print("Error: No log content provided. Provide a file or pipe content to stdin.", file=sys.stderr)
        sys.exit(1)
    
    if not log_text.strip():
        print("Error: No log content provided", file=sys.stderr)
        sys.exit(1)
    
    # Analyze
    issues = analyze_log(log_text)
    
    # Output
    if args.json:
        output = {
            'instance_id': args.instance_id,
            'issues_found': len(issues),
            'issues': issues,
        }
        print(json.dumps(output, indent=2, ensure_ascii=False))
    elif args.summary:
        print(f"📊 发现 {len(issues)} 个问题")
        for issue in issues:
            icon = {'high': '🔴', 'medium': '🟡', 'low': '🟢'}.get(issue['severity'], '⚪')
            print(f"  {icon} {issue['title']} ({issue['severity']})")
    else:
        report = generate_report(issues, args.instance_id)
        print(report)


if __name__ == "__main__":
    main()


path: scripts/fetch_instance_log.py
content: #!/usr/bin/env python3
"""
DataWorks Task Instance Log Fetcher using Official Alibaba Cloud SDK

Usage:
    python3 fetch_instance_log.py <instance_id> [options]
"""

import argparse
import json
import os
import sys
from pathlib import Path

try:
    from alibabacloud_credentials.client import Client as CredentialClient
    from alibabacloud_dataworks_public20240518.client import Client as DataWorksClient
    from alibabacloud_tea_openapi import models as open_api_models
    from alibabacloud_dataworks_public20240518 import models as dataworks_models
    from alibabacloud_tea_util import models as util_models
except ImportError:
    print("❌ Alibaba Cloud SDK not installed", file=sys.stderr)
    print("\nInstall with:", file=sys.stderr)
    print("  pip3 install alibabacloud_dataworks_public20240518 alibabacloud_credentials", file=sys.stderr)
    sys.exit(1)


def load_credentials():
    """Load Alibaba Cloud credentials from environment or config file"""
    import os
    
    # Try environment variables first
    access_key = os.getenv('ALIBABA_CLOUD_ACCESS_KEY_ID')
    access_secret = os.getenv('ALIBABA_CLOUD_ACCESS_KEY_SECRET')
    
    if access_key and access_secret:
        return access_key, access_secret
    
    # Try config file
    config_path = Path.home() / '.alibabacloud' / 'credentials'
    if config_path.exists():
        try:
            config = json.loads(config_path.read_text())
            return config.get('access_key_id'), config.get('access_key_secret')
        except Exception:
            pass
    
    # Try current directory credentials.json
    config_path = Path('credentials.json')
    if config_path.exists():
        try:
            config = json.loads(config_path.read_text())
            return config.get('access_key_id'), config.get('access_key_secret')
        except Exception:
            pass
    
    return None, None


def fetch_log(instance_id, region='cn-hangzhou', access_key=None, access_secret=None):
    """
    Fetch task instance log from DataWorks API using official SDK
    
    API: https://api.aliyun.com/api/dataworks-public/2024-05-18/GetTaskInstanceLog
    """
    # Load credentials if not provided
    if not access_key or not access_secret:
        access_key, access_secret = load_credentials()
        if not access_key or not access_secret:
            print("❌ Error: Alibaba Cloud credentials not found", file=sys.stderr)
            print("\nPlease set credentials using one of these methods:", file=sys.stderr)
            print("  1. Environment variables:", file=sys.stderr)
            print("     export ALIBABA_CLOUD_ACCESS_KEY_ID=your_key", file=sys.stderr)
            print("     export ALIBABA_CLOUD_ACCESS_KEY_SECRET=your_secret", file=sys.stderr)
            print("  2. Config file: ~/.alibabacloud/credentials", file=sys.stderr)
            print("  3. Local file: credentials.json", file=sys.stderr)
            return None
    
    try:
        # Create credential
        os.environ['ALIBABA_CLOUD_ACCESS_KEY_ID'] = access_key
        os.environ['ALIBABA_CLOUD_ACCESS_KEY_SECRET'] = access_secret
        credential = CredentialClient()
        
        # Create client config
        config = open_api_models.Config(credential=credential)
        config.endpoint = f'dataworks.{region}.aliyuncs.com'
        
        # Create client
        client = DataWorksClient(config)
        
        # Create request - Note: parameter is 'id' not 'InstanceId'
        request = dataworks_models.GetTaskInstanceLogRequest(id=int(instance_id))
        
        # Set runtime options
        runtime = util_models.RuntimeOptions()
        
        # Send request
        response = client.get_task_instance_log_with_options(request, runtime)
        
        # Convert response to dict
        return response.to_map()
        
    except Exception as e:
        print(f"❌ Error: {e}", file=sys.stderr)
        if hasattr(e, 'data') and e.data:
            print(f"   Recommend: {e.data.get('Recommend', '')}", file=sys.stderr)
        return None


def format_log_output(response, verbose=False):
    """Format API response for display"""
    if not response:
        return None
    
    # Navigate response structure
    body = response.get('body', {})
    data = body.get('Data', {})
    
    log_content = data.get('LogContent', '')
    instance_status = data.get('InstanceStatus', 'Unknown')
    cycle_time = data.get('CycleTime', 'N/A')
    
    output = []
    output.append("=" * 60)
    output.append("📋 DataWorks Task Instance Log")
    output.append("=" * 60)
    output.append(f"Status: {instance_status}")
    output.append(f"Cycle Time: {cycle_time}")
    output.append("-" * 60)
    output.append("Log Content:")
    output.append("-" * 60)
    
    if log_content:
        if verbose:
            output.append(log_content)
        else:
            lines = log_content.split('\n')
            if len(lines) > 50:
                output.append(f"... ({len(lines) - 50} lines omitted)")
                output.extend(lines[-50:])
            else:
                output.extend(lines)
    else:
        output.append("(No log content available)")
    
    output.append("=" * 60)
    return '\n'.join(output)


def main():
    parser = argparse.ArgumentParser(
        description="Fetch DataWorks task instance log using official SDK",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  %(prog)s 12345678
  %(prog)s 12345678 --region cn-shanghai
  %(prog)s 12345678 --json
  %(prog)s 12345678 --verbose
        """
    )
    
    parser.add_argument("instance_id", help="Task instance ID")
    parser.add_argument("--region", "-r", default='cn-hangzhou',
                       help="Alibaba Cloud region (default: cn-hangzhou)")
    parser.add_argument("--access-key", help="Access Key ID")
    parser.add_argument("--access-secret", help="Access Key Secret")
    parser.add_argument("--json", "-j", action="store_true",
                       help="Output as JSON")
    parser.add_argument("--verbose", "-v", action="store_true",
                       help="Show full log (default: last 50 lines)")
    
    args = parser.parse_args()
    
    # Fetch log
    response = fetch_log(
        args.instance_id,
        region=args.region,
        access_key=args.access_key,
        access_secret=args.access_secret
    )
    
    if response is None:
        sys.exit(1)
    
    # Output
    if args.json:
        print(json.dumps(response, indent=2, ensure_ascii=False, default=str))
    else:
        formatted = format_log_output(response, args.verbose)
        if formatted:
            print(formatted)


if __name__ == "__main__":
    main()


path: scripts/analyze_error.py
content: #!/usr/bin/env python3
"""
DataWorks Error Analyzer - Clean and focused error analysis

Extracts errors from logs and provides clear, actionable solutions.
"""

import argparse
import json
import re
import sys
from pathlib import Path


def extract_errors(log_text):
    """
    Extract error codes and key error messages from log
    
    Returns list of error info dicts, sorted by severity
    """
    errors = []
    
    # Pattern 1: ODPS error codes (highest priority)
    # e.g., ODPS-0130071:[11,10] Semantic analysis exception
    odps_pattern = r'ODPS-(\d+):\[(\d+),(\d+)\]\s*([^\n]+)'
    for match in re.finditer(odps_pattern, log_text):
        errors.append({
            'type': 'ODPS',
            'code': f'ODPS-{match.group(1)}',
            'position': f'line {match.group(2)}, col {match.group(3)}',
            'message': match.group(4).strip()[:300],
            'severity': 'high',
        })
    
    # Pattern 2: DataX error codes
    # e.g., Code:[MYSQLErrCode-02], Description:[...]
    datax_pattern = r'Code:\[([^\]]+)\],\s*Description:\[([^\]]+)\]'
    for match in re.finditer(datax_pattern, log_text):
        errors.append({
            'type': 'DataX',
            'code': match.group(1),
            'message': match.group(2).strip()[:300],
            'severity': 'high',
        })
    
    # Pattern 3: Java exceptions
    exception_pattern = r'([a-zA-Z0-9_.]+Exception):\s*([^\n]+)'
    for match in re.finditer(exception_pattern, log_text):
        errors.append({
            'type': 'Exception',
            'code': match.group(1).split('.')[-1],
            'message': match.group(2).strip()[:300],
            'severity': 'medium',
        })
    
    # Remove duplicates (same error code)
    seen_codes = set()
    unique_errors = []
    for error in errors:
        key = f"{error['type']}:{error['code']}"
        if key not in seen_codes:
            seen_codes.add(key)
            unique_errors.append(error)
    
    # Sort by severity
    severity_order = {'high': 0, 'medium': 1, 'low': 2}
    unique_errors.sort(key=lambda x: severity_order.get(x['severity'], 2))
    
    return unique_errors


def analyze_error(error):
    """
    Analyze a single error and provide solutions
    
    Returns analysis dict with causes, solutions, and next steps
    """
    code = error['code']
    msg = error['message'].lower()
    
    analysis = {
        'causes': [],
        'solutions': [],
        'next_step': '',
    }
    
    # ODPS Error Analysis
    if error['type'] == 'ODPS':
        if '0130071' in code or 'semantic' in msg or 'function' in msg:
            analysis['causes'] = [
                f'SQL 中使用了不存在的函数或视图',
            ]
            analysis['solutions'] = [
                f'检查 SQL 第 {error["position"]} 处的函数：{error["message"][:100]}',
                '确认函数名称拼写正确（MaxCompute 函数区分大小写）',
                '在 DataWorks 数据开发中创建临时查询，测试该函数',
            ]
            analysis['next_step'] = '在 DataWorks 数据开发面板试运行 SQL，查看具体哪个函数不存在'
        
        elif '0130141' in code or 'cast' in msg or 'type' in msg:
            analysis['causes'] = [
                '数据类型转换失败',
            ]
            analysis['solutions'] = [
                f'检查错误位置 {error["position"]} 的数据类型',
                '使用 CAST 函数显式转换：CAST(column AS STRING)',
                '检查源数据是否包含无法转换的值',
            ]
            analysis['next_step'] = '查看 SQL 中该位置的字段类型定义'
        
        elif 'table' in msg and ('exist' in msg or 'found' in msg):
            analysis['causes'] = [
                '表不存在',
            ]
            analysis['solutions'] = [
                '检查表名拼写（包括项目空间前缀）',
                '在数据地图中搜索该表',
                '确认环境配置（开发/生产）',
            ]
            analysis['next_step'] = '在 DataWorks 数据地图中搜索表名'
        
        else:
            analysis['causes'] = ['ODPS SQL 执行错误']
            analysis['solutions'] = [
                '查看完整错误信息',
                '检查 SQL 语法',
            ]
            analysis['next_step'] = '复制完整错误信息到 DataWorks 数据开发中测试'
    
    # DataX Error Analysis
    elif error['type'] == 'DataX':
        if 'mysql' in code.lower() or 'jdbc' in msg or 'connection' in msg:
            analysis['causes'] = [
                '数据库连接失败',
            ]
            analysis['solutions'] = [
                '检查 JDBC URL 是否正确',
                '确认 RDS 白名单包含 DataWorks IP',
                '验证账号密码',
            ]
            analysis['next_step'] = '在 RDS 控制台检查白名单配置'
        
        else:
            analysis['causes'] = ['数据同步错误']
            analysis['solutions'] = [
                '检查数据源配置',
                '查看字段映射',
            ]
            analysis['next_step'] = '检查数据同步任务的读写端配置'
    
    # Exception Analysis
    elif error['type'] == 'Exception':
        if 'nullpointer' in code.lower() or 'null' in msg:
            analysis['causes'] = [
                '空指针异常 - 必填参数为空',
            ]
            analysis['solutions'] = [
                '检查任务配置中的必填字段',
                '验证上游节点是否有输出',
            ]
            analysis['next_step'] = '检查节点配置和上游依赖'
        
        else:
            analysis['causes'] = ['程序执行异常']
            analysis['solutions'] = [
                '查看完整堆栈信息',
            ]
            analysis['next_step'] = '收集完整日志联系技术支持'
    
    else:
        analysis['causes'] = ['未知错误']
        analysis['solutions'] = ['收集完整日志']
        analysis['next_step'] = '联系阿里云技术支持'
    
    return analysis


def format_report(errors, analyses, instance_id):
    """Format clean, focused report"""
    output = []
    
    output.append("=" * 70)
    output.append(f"DataWorks 任务诊断报告")
    output.append("=" * 70)
    if instance_id:
        output.append(f"实例 ID: {instance_id}")
    output.append(f"发现错误：{len(errors)} 个\n")
    
    if not errors:
        output.append("✅ 未发现明显错误")
        output.append("\n建议:")
        output.append("  - 检查业务逻辑是否正确")
        output.append("  - 查看完整日志")
    else:
        for i, (error, analysis) in enumerate(zip(errors, analyses), 1):
            output.append("-" * 70)
            output.append(f"错误 {i}: {error['type']} - {error['code']}")
            output.append("")
            
            output.append("错误信息:")
            output.append(f"  {error['message']}")
            if error.get('position'):
                output.append(f"  位置：{error['position']}")
            output.append("")
            
            output.append("可能原因:")
            for cause in analysis['causes']:
                output.append(f"  • {cause}")
            output.append("")
            
            output.append("解决方案:")
            for j, sol in enumerate(analysis['solutions'], 1):
                output.append(f"  {j}. {sol}")
            output.append("")
            
            output.append(f"下一步：{analysis['next_step']}")
            output.append("")
    
    output.append("=" * 70)
    
    return '\n'.join(output)


def main():
    parser = argparse.ArgumentParser(
        description="DataWorks Error Analyzer",
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    
    parser.add_argument("log_file", nargs='?', help="Log file to analyze")
    parser.add_argument("--instance-id", help="Task instance ID")
    parser.add_argument("--json", "-j", action="store_true", help="Output as JSON")
    
    args = parser.parse_args()
    
    # Read log content
    if args.log_file:
        try:
            log_text = Path(args.log_file).read_text()
        except Exception as e:
            print(f"Error reading file: {e}", file=sys.stderr)
            sys.exit(1)
    elif not sys.stdin.isatty():
        log_text = sys.stdin.read()
    else:
        print("Error: No log content provided", file=sys.stderr)
        sys.exit(1)
    
    # Extract errors
    errors = extract_errors(log_text)
    
    # Analyze each error
    analyses = [analyze_error(error) for error in errors]
    
    # Output
    if args.json:
        output = {
            'instance_id': args.instance_id,
            'errors': errors,
            'analyses': analyses,
        }
        print(json.dumps(output, indent=2, ensure_ascii=False))
    else:
        report = format_report(errors, analyses, args.instance_id)
        print(report)


if __name__ == "__main__":
    main()


path: scripts/fetch_instance_log_sdk.py
content: #!/usr/bin/env python3
"""
DataWorks Task Instance Log Fetcher using Alibaba Cloud SDK

Usage:
    python3 fetch_instance_log.py <instance_id> [options]
"""

import argparse
import json
import sys
from pathlib import Path

try:
    from aliyunsdkcore.client import AcsClient
    from aliyunsdkcore.request import CommonRequest
    from aliyunsdkcore.exceptions import ClientException, ServerException
except ImportError:
    print("❌ Alibaba Cloud SDK not installed", file=sys.stderr)
    print("\nInstall with: pip3 install aliyun-python-sdk-core", file=sys.stderr)
    sys.exit(1)


def load_credentials():
    """Load Alibaba Cloud credentials from environment or config file"""
    import os
    
    # Try environment variables first
    access_key = os.getenv('ALIBABA_CLOUD_ACCESS_KEY_ID')
    access_secret = os.getenv('ALIBABA_CLOUD_ACCESS_KEY_SECRET')
    
    if access_key and access_secret:
        return access_key, access_secret
    
    # Try config file
    config_path = Path.home() / '.alibabacloud' / 'credentials'
    if config_path.exists():
        try:
            config = json.loads(config_path.read_text())
            return config.get('access_key_id'), config.get('access_key_secret')
        except Exception:
            pass
    
    # Try current directory credentials.json
    config_path = Path('credentials.json')
    if config_path.exists():
        try:
            config = json.loads(config_path.read_text())
            return config.get('access_key_id'), config.get('access_key_secret')
        except Exception:
            pass
    
    return None, None


def fetch_log(instance_id, region='cn-hangzhou', access_key=None, access_secret=None):
    """
    Fetch task instance log from DataWorks API using SDK
    """
    # Load credentials if not provided
    if not access_key or not access_secret:
        access_key, access_secret = load_credentials()
        if not access_key or not access_secret:
            print("❌ Error: Alibaba Cloud credentials not found", file=sys.stderr)
            print("\nPlease set credentials using one of these methods:", file=sys.stderr)
            print("  1. Environment variables:", file=sys.stderr)
            print("     export ALIBABA_CLOUD_ACCESS_KEY_ID=your_key", file=sys.stderr)
            print("     export ALIBABA_CLOUD_ACCESS_KEY_SECRET=your_secret", file=sys.stderr)
            print("  2. Config file: ~/.alibabacloud/credentials", file=sys.stderr)
            print("  3. Local file: credentials.json", file=sys.stderr)
            return None
    
    try:
        # Create client
        client = AcsClient(access_key, access_secret, region)
        
        # Create request
        request = CommonRequest()
        request.set_method('GET')
        request.set_domain(f'dataworks.{region}.aliyuncs.com')
        request.set_version('2024-05-18')
        request.set_action_name('GetTaskInstanceLog')
        
        request.add_query_param('InstanceId', instance_id)
        request.add_query_param('RegionId', region)
        
        # Send request
        response = client.do_action_with_exception(request)
        return json.loads(response.decode('utf-8'))
        
    except ClientException as e:
        print(f"❌ Client Error: {e}", file=sys.stderr)
        return None
    except ServerException as e:
        print(f"❌ Server Error: {e}", file=sys.stderr)
        return None
    except Exception as e:
        print(f"❌ Error: {e}", file=sys.stderr)
        return None


def format_log_output(response, verbose=False):
    """Format API response for display"""
    if not response:
        return None
    
    data = response.get('Data', {})
    log_content = data.get('LogContent', '')
    instance_status = data.get('InstanceStatus', 'Unknown')
    cycle_time = data.get('CycleTime', 'N/A')
    
    output = []
    output.append("=" * 60)
    output.append("📋 DataWorks Task Instance Log")
    output.append("=" * 60)
    output.append(f"Status: {instance_status}")
    output.append(f"Cycle Time: {cycle_time}")
    output.append("-" * 60)
    output.append("Log Content:")
    output.append("-" * 60)
    
    if log_content:
        if verbose:
            output.append(log_content)
        else:
            lines = log_content.split('\n')
            if len(lines) > 50:
                output.append(f"... ({len(lines) - 50} lines omitted)")
                output.extend(lines[-50:])
            else:
                output.extend(lines)
    else:
        output.append("(No log content available)")
    
    output.append("=" * 60)
    return '\n'.join(output)


def main():
    parser = argparse.ArgumentParser(
        description="Fetch DataWorks task instance log using Alibaba Cloud SDK",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  %(prog)s 12345678
  %(prog)s 12345678 --region cn-shanghai
  %(prog)s 12345678 --json
  %(prog)s 12345678 --verbose
        """
    )
    
    parser.add_argument("instance_id", help="Task instance ID")
    parser.add_argument("--region", "-r", default='cn-hangzhou',
                       help="Alibaba Cloud region (default: cn-hangzhou)")
    parser.add_argument("--access-key", help="Access Key ID")
    parser.add_argument("--access-secret", help="Access Key Secret")
    parser.add_argument("--json", "-j", action="store_true",
                       help="Output as JSON")
    parser.add_argument("--verbose", "-v", action="store_true",
                       help="Show full log (default: last 50 lines)")
    
    args = parser.parse_args()
    
    # Fetch log
    response = fetch_log(
        args.instance_id,
        region=args.region,
        access_key=args.access_key,
        access_secret=args.access_secret
    )
    
    if response is None:
        sys.exit(1)
    
    # Output
    if args.json:
        print(json.dumps(response, indent=2, ensure_ascii=False))
    else:
        formatted = format_log_output(response, args.verbose)
        if formatted:
            print(formatted)


if __name__ == "__main__":
    main()


path: scripts/dataworks_diagnose.py
content: #!/usr/bin/env python3
"""
DataWorks Task Instance Diagnostician - All-in-One Tool

Fetches task instance log and provides diagnostic recommendations.

Usage:
    python3 dataworks_diagnose.py <instance_id> [options]

Examples:
    python3 dataworks_diagnose.py 12345678
    python3 dataworks_diagnose.py 12345678 --region cn-shanghai
    python3 dataworks_diagnose.py 12345678 --auto-fix
"""

import argparse
import json
import os
import subprocess
import sys
from pathlib import Path


def run_script(script_name, args, capture=True, input_text=None):
    """Run a helper script and return output"""
    script_path = Path(__file__).parent / script_name
    
    cmd = [sys.executable, str(script_path)] + args
    
    try:
        result = subprocess.run(
            cmd,
            capture_output=capture,
            text=True,
            timeout=60,
            input=input_text
        )
        
        if result.returncode != 0:
            return None, result.stderr
        
        return result.stdout, None
        
    except subprocess.TimeoutExpired:
        return None, "Script execution timed out"
    except Exception as e:
        return None, str(e)


def main():
    parser = argparse.ArgumentParser(
        description="DataWorks Task Instance Diagnostician",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  %(prog)s 12345678
  %(prog)s 12345678 --region cn-shanghai
  %(prog)s 12345678 --json
  %(prog)s 12345678 --save-report report.txt
        """
    )
    
    parser.add_argument("instance_id", help="Task instance ID")
    parser.add_argument("--region", "-r", default='cn-hangzhou',
                       help="Alibaba Cloud region (default: cn-hangzhou)")
    parser.add_argument("--access-key", help="Access Key ID")
    parser.add_argument("--access-secret", help="Access Key Secret")
    parser.add_argument("--json", "-j", action="store_true",
                       help="Output as JSON")
    parser.add_argument("--verbose", "-v", action="store_true",
                       help="Show full log")
    parser.add_argument("--save-log", metavar="FILE",
                       help="Save raw log to file")
    parser.add_argument("--save-report", metavar="FILE",
                       help="Save diagnostic report to file")
    
    args = parser.parse_args()
    
    print(f"🔍 开始诊断 DataWorks 任务实例：{args.instance_id}")
    print(f"📍 区域：{args.region}")
    print("-" * 60)
    
    # Step 1: Fetch log
    print("\n📥 步骤 1/2: 获取任务日志...")
    
    fetch_args = [
        args.instance_id,
        '--region', args.region,
    ]
    
    if args.access_key:
        fetch_args.extend(['--access-key', args.access_key])
    if args.access_secret:
        fetch_args.extend(['--access-secret', args.access_secret])
    if args.verbose:
        fetch_args.append('--verbose')
    
    log_output, error = run_script('fetch_instance_log.py', fetch_args + ['--json'])
    
    if error:
        print(f"❌ 获取日志失败：{error}")
        sys.exit(1)
    
    print("✅ 日志获取成功")
    
    # Save raw log if requested
    if args.save_log:
        Path(args.save_log).write_text(log_output)
        print(f"💾 原始日志已保存到：{args.save_log}")
    
    # Parse JSON response and extract log content
    try:
        response = json.loads(log_output)
        body = response.get('body', {})
        log_content = body.get('TaskInstanceLog', '')
        request_id = body.get('RequestId', 'N/A')
        
        # Print task info
        print(f"📋 RequestId: {request_id}")
        
        if not log_content:
            print("❌ 日志内容为空")
            sys.exit(1)
        
        # Extract and print task summary
        print("\n📊 任务摘要:")
        found_summary = False
        
        # Try to find DI task summary first
        for line in log_content.split('\n'):
            if 'DI Submit at' in line:
                print(f"   {line.strip()}")
                found_summary = True
            elif 'DI Start at' in line:
                print(f"   {line.strip()}")
            elif 'DI Finish at' in line:
                print(f"   {line.strip()}")
            elif 'DI job state' in line:
                print(f"   {line.strip()}")
                found_summary = True
            elif 'ErrorMessage:' in line:
                print(f"   {line.strip()}")
        
        # If not DI task, try to find ODPS/shell task status
        if not found_summary:
            for line in log_content.split('\n'):
                if 'Current task status' in line:
                    print(f"   {line.strip()}")
                    found_summary = True
                elif 'Exit code of the Shell command' in line:
                    print(f"   {line.strip()}")
                    found_summary = True
                elif 'Shell run failed' in line:
                    print(f"   ❌ {line.strip()}")
                    found_summary = True
                elif 'FAILED:' in line and 'ODPS' in line:
                    print(f"   ❌ {line.strip()[:200]}")
                    found_summary = True
        
        if not found_summary:
            print("   (未找到任务状态信息)")
    except json.JSONDecodeError as e:
        print(f"❌ JSON 解析失败：{e}")
        sys.exit(1)
    
    # Step 2: Diagnose with AI analysis
    print("\n🔬 步骤 2/2: AI 智能分析中...")
    
    # Pass log content via stdin to new analyzer
    analyze_args = ['--instance-id', args.instance_id]
    if args.json:
        analyze_args.append('--json')
    
    report_output, error = run_script('analyze_error.py', analyze_args, input_text=log_content)
    
    if error:
        print(f"❌ 诊断失败：{error}")
        sys.exit(1)
    
    print("✅ 诊断完成")
    print("\n" + "=" * 60)
    print("📋 诊断报告")
    print("=" * 60)
    print(report_output)
    
    # Save report if requested
    if args.save_report:
        Path(args.save_report).write_text(report_output)
        print(f"\n💾 诊断报告已保存到：{args.save_report}")


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)