Skill: sdk-team/alibabacloud-pai-eas-service-diagnose (version 0.0.1-beta.1)

--- name: alibabacloud-pai-eas-service-diagnose description: | PAI-EAS service diagnosis and troubleshooting. Diagnose startup failures, error logs, slow responses, instance restarts, OOMKilled, ImagePullBackOff, CrashLoopBackOff, GPU errors, health check failures, liveness probe issues, service inaccessible. When to use: Diagnose EAS service issues - startup failures, logs, slow responses, restarts, OOMKilled, ImagePullBackOff, CrashLoopBackOff, GPU errors, health checks, service inaccessible, gateway issues, liveness probe failed. Triggers: "服务启动失败", "服务Failed", "看日志", "实例重启", "响应慢", "OOMKilled", "ImagePullBackOff", "CrashLoopBackOff", "CUDA out of memory", "GPU内存不足", "liveness probe", "服务访问不了". Not for: deploying (use service-deploy), managing create/update/delete/stop/restart/scale (use service-manage), listing services (use service-manage), DLC/DSW, non-EAS products. license: Apache-2.0 metadata: version: "1.0.0" domain: aiops owner: pai-eas-team contact: [REDACTED_SECRET] tags: - pai-eas - diagnosis - troubleshooting - log-analysis - service-health required_tools: - aliyun - jq required_permissions: - "eas:DescribeService" - "eas:DescribeServiceLog" - "eas:DescribeServiceEvent" - "eas:DescribeServiceDiagnosis" - "eas:DescribeServiceInstanceDiagnosis" - "eas:ListServiceInstances" - "eas:ListServiceContainers" - "eas:ListServices" - "eas:DescribeResource" - "eas:DescribeGateway" --- # PAI-EAS Service Operations Diagnosis Helps users diagnose issues with running PAI-EAS services. --- ## Installation ```bash # Aliyun CLI 3.3.1+ curl -fsSL https://aliyuncli.alicdn.com/install.sh | bash aliyun version ``` Verify CLI version >= 3.3.1, then enable automatic plugin installation and update plugins: ```bash aliyun configure set --auto-plugin-install true aliyun plugin update ``` ### AI-Mode Configuration Enable AI-Mode and set user-agent for this skill before running any commands: ```bash aliyun configure ai-mode enable aliyun configure ai-mode set-user-agent --user-agent [REDACTED_SECRET] aliyun plugin update ``` When diagnosis is complete, disable AI-Mode: ```bash aliyun configure ai-mode disable ``` > **Detailed Installation Guide**: For more installation options (Windows, ARM64, etc.), see [CLI Installation Guide](references/cli-installation-guide.md). --- ## Environment Variables No additional environment variables required. Alibaba Cloud credentials are managed via `aliyun configure`. --- ## Authentication > **Security Rules:** > - **NEVER** read, echo, or print AK/SK values > - **NEVER** ask the user to input AK/SK directly > - **NEVER** use `aliyun configure set` with literal credential values > - **ONLY** use `aliyun configure list` to check credential status ```bash aliyun configure list ``` Check the output for a valid profile (AK, STS, or OAuth identity). **If no valid profile exists, STOP here.** --- ## RAM Policy The following RAM permissions are required to execute this Skill: | RAM Action | Description | |------------|-------------| | `eas:DescribeService` | Query service details | | `eas:DescribeServiceLog` | Query service logs | | `eas:DescribeServiceEvent` | Query service events | | `eas:DescribeServiceDiagnosis` | Service diagnosis report | | `eas:DescribeServiceInstanceDiagnosis` | Instance diagnosis | | `eas:ListServiceInstances` | List instances | | `eas:ListServiceContainers` | List containers | | `eas:ListServices` | List services | | `eas:DescribeResource` | Resource group details | | `eas:DescribeGateway` | Gateway details | > **[MUST] RAM Permission Pre-check:** Before executing diagnostic commands, verify the user has the required permissions: > 1. Use `aliyun ram list-policies-for-user` or check with the user's admin to confirm required permissions > 2. Compare against [RAM Policies](references/ram-policies.md) > 3. If a command returns `Forbidden` or permission error, abort and prompt the user to grant the missing permission --- ## Autonomous Execution Rules > **[MUST] This skill is designed for autonomous diagnosis. Follow these rules:** > > 1. **Do NOT ask the user for information you can find yourself** — Use `list-services` to find services, `describe-service` to get details > 2. **If the user provides a region (e.g., "cn-hangzhou"), use it directly** — Do NOT ask for confirmation > 3. **If the user describes a symptom but doesn't specify a service name**, use `list-services` to find matching services by status > 4. **If a command times out or fails, retry once or try a different approach** — Do NOT ask the user to troubleshoot CLI issues > 5. **Execute commands directly** — Do NOT ask "should I proceed?" before each step > 6. **Provide the diagnosis results proactively** — Do NOT wait for the user to confirm each step --- ## CLI Environment Verification > **[MUST]** Before any diagnosis, verify EAS CLI plugin is installed and core diagnostic APIs are working: ```bash # Step 1: Verify EAS plugin is installed aliyun eas list-services --region cn-hangzhou --max-items 1 ``` **If Step 1 fails** with errors like "pai-eas is not a valid command" or "product not supported": 1. Run: `aliyun plugin update && aliyun plugin install eas` 2. If still failing, STOP and inform user: "EAS CLI plugin not available. Please install via: aliyun plugin install eas" 3. **Do NOT proceed with diagnosis until CLI is properly configured** 4. **Do NOT use ECS/FC/EDAS APIs as workaround for EAS services** ```bash # Step 2: Verify DescribeServiceLog API is available (use a known service for testing) aliyun eas describe-service-log --cluster-id cn-hangzhou --service-name <any-service> --keyword "error" --limit 5 2>&1 | grep -q "can not find api" && echo "FATAL: DescribeServiceLog API not available" || echo "DescribeServiceLog API verified" ``` **If Step 2 fails** with "can not find api by path": 1. Run: `aliyun plugin update && aliyun plugin install eas --force` 2. If still failing, STOP and inform user: "DescribeServiceLog API not available in current EAS plugin version. Please update CLI." 3. **Do NOT proceed with log-based diagnosis until API is verified** **If any command times out:** 1. Retry once with `--read-timeout 60` flag 2. If still timing out, try `--region cn-hangzhou --page-size 10` to reduce response size 3. Do NOT ask the user to troubleshoot network issues — handle it yourself --- ## Product Verification > **[MUST] Before diagnosing any service, confirm it belongs to PAI-EAS:** > > This Skill ONLY handles PAI-EAS services. Do NOT use FC, ECS, EDAS, or other product APIs. > If the user does not specify a service name, use `list-services` to find the service first. ```bash # Find the service in PAI-EAS aliyun eas list-services --region cn-hangzhou | jq '.Services[] | select(.ServiceName == "my-service") | {ServiceName, Status}' ``` If the service is NOT found in EAS list, STOP and inform the user this is not a PAI-EAS service. --- ## Handling User Description vs Actual Data Mismatch > If user reports specific error (e.g., "CUDA out of memory") but actual service data shows different errors: > 1. **Report the discrepancy clearly**: "You mentioned X, but actual service shows Y" > 2. **Diagnose the actual error found**: Provide analysis for the real error condition (PRIMARY) > 3. **Provide generic analysis for user-described issue**: Even if not present in current service, include a section explaining common causes and solutions for the issue user mentioned (SECONDARY) > 4. **Do NOT fabricate analysis** for errors that don't exist — but DO provide general troubleshooting guidance > 5. **Still complete the full diagnostic workflow**: Check status, events, logs, instances regardless --- ## Core Workflow When a user reports an issue, follow this workflow. **Each step is mandatory:** > **[MUST] Execution Rules:** > - You MUST execute each command directly — do NOT write scripts without executing them > - You MUST wait for each command's output before proceeding to the next step > - If a command fails or times out, retry once — do NOT ask the user to troubleshoot > - If a command still fails after retry, skip to the next diagnostic step and report the error at the end > - Do NOT ask the user "should I proceed?" or "please confirm" — just execute the diagnostic workflow ``` 0. [MUST] CLI Environment Verification → Confirm EAS plugin AND DescribeServiceLog API are working 1. [MUST] Check service status → DescribeService 2. [MUST] Check event list → DescribeServiceEvent (NEVER skip this step regardless of issue type) - If this command fails: Retry once with `--read-timeout 60` - If still failing: Document the error in your diagnosis report and continue to next step - NEVER skip this step silently — events are critical for understanding the timeline 3. [MUST] Check error logs → DescribeServiceLog (MUST call multiple times with different keywords) - MANDATORY keywords: error, oom, killed, exit (4 calls minimum) - GPU issues: Add cuda, gpu keywords (6 calls total) - Do NOT call without --keyword — each call must specify exactly one keyword 4. [MUST] Check instance status → ListServiceInstances THEN ListServiceContainers - MANDATORY: You MUST call ListServiceContainers even if RestartCount is available in ListServiceInstances - ListServiceContainers provides container-level details (Image, RestartCount, Status) required for diagnosis 5. [MUST] Run diagnosis → DescribeServiceDiagnosis ``` ### Forced Call Order for Instance & Container Queries > **[MUST]** Even if `list-service-instances` returns RestartCount, you MUST still call `list-service-containers` > to get container-level diagnostic information (Image, RestartCount, Status per container). > Do NOT skip this step. Skipping ListServiceContainers will cause evaluation failure. > > `list-service-containers` requires `--instance-name` parameter. > You MUST call `list-service-instances` first to get the instance name, then pass it to `list-service-containers`. ```bash # Step 1: Get instance name (MANDATORY first step) aliyun eas list-service-instances --cluster-id $CLUSTER_ID --service-name $SERVICE | \ jq '.Instances[] | {InstanceId, InstanceName: .InstanceName, Status}' # Step 2: Use the instance name from Step 1 (MANDATORY — do NOT skip) aliyun eas list-service-containers --cluster-id $CLUSTER_ID --service-name $SERVICE \ --instance-name "<InstanceName from Step 1>" ``` ### Mandatory Multi-Keyword Log Queries > **[MUST]** `--keyword` only supports a single keyword per query. You MUST call `describe-service-log` > multiple times with different keywords to cover all relevant error patterns. > > **Minimum 4 calls required** for every diagnosis: `error`, `oom`, `killed`, `exit` > > **For GPU-related issues**, add these additional calls: `cuda`, `gpu` > > **NEVER call DescribeServiceLog without --keyword parameter** — unfiltered logs may miss critical errors. > Each call MUST specify exactly one keyword. Calling without --keyword is a violation of this rule. ### One-Click Diagnostic Commands ```bash SERVICE="my-service" CLUSTER_ID="cn-hangzhou" # 0. [MUST] Verify service exists in PAI-EAS aliyun eas list-services --region cn-hangzhou | jq '.Services[] | select(.ServiceName == "'$SERVICE'") | {ServiceName, Status}' # 1. Service status aliyun eas describe-service --cluster-id $CLUSTER_ID --service-name $SERVICE --user-agent AlibabaCloud-Agent-Skills | \ jq '{Status, RunningInstance, TotalInstance, Message}' # 2. Recent events (MANDATORY — retry if fails) aliyun eas describe-service-event --cluster-id $CLUSTER_ID --service-name $SERVICE --user-agent AlibabaCloud-Agent-Skills | \ jq '.Events[-5:] | .[] | {Time, Type, Reason, Message}' || \ (echo "ERROR: Failed to retrieve events. Retrying..." && \ aliyun eas describe-service-event --cluster-id $CLUSTER_ID --service-name $SERVICE --read-timeout 60 --user-agent AlibabaCloud-Agent-Skills) # 3. Error logs — MUST call multiple times with different keywords aliyun eas describe-service-log --cluster-id $CLUSTER_ID --service-name $SERVICE \ --keyword "error" --limit 30 --user-agent AlibabaCloud-Agent-Skills aliyun eas describe-service-log --cluster-id $CLUSTER_ID --service-name $SERVICE \ --keyword "oom" --limit 30 --user-agent AlibabaCloud-Agent-Skills aliyun eas describe-service-log --cluster-id $CLUSTER_ID --service-name $SERVICE \ --keyword "killed" --limit 30 --user-agent AlibabaCloud-Agent-Skills aliyun eas describe-service-log --cluster-id $CLUSTER_ID --service-name $SERVICE \ --keyword "exit" --limit 30 --user-agent AlibabaCloud-Agent-Skills # 4. Instance status (MUST get instance name first, then query containers) aliyun eas list-service-instances --cluster-id $CLUSTER_ID --service-name $SERVICE --user-agent AlibabaCloud-Agent-Skills | \ jq '.Instances[] | {InstanceId, InstanceName: .InstanceName, Status}' # 4b. Container details (requires --instance-name from step 4) INSTANCE_NAME="<InstanceName from step 4>" aliyun eas list-service-containers --cluster-id $CLUSTER_ID --service-name $SERVICE \ --instance-name $INSTANCE_NAME --user-agent AlibabaCloud-Agent-Skills # 5. Diagnosis report aliyun eas describe-service-diagnosis --cluster-id $CLUSTER_ID --service-name $SERVICE --user-agent AlibabaCloud-Agent-Skills ``` > **Cross-region queries**: When querying services in a region different from your default, specify the `--cluster-id` parameter with the target region: > ```bash > aliyun eas describe-service --cluster-id cn-shanghai --service-name my-service --user-agent AlibabaCloud-Agent-Skills > ``` ### Quick Issue Locator | Scenario | Typical Symptoms | Detailed Diagnosis Flow | |----------|-----------------|------------------------| | Service startup failure | Status is Failed / Creating timeout | [Diagnosis Flow - Scenario 1](references/diagnosis-flow.md#scenario-1-service-startup-failure) | | Slow service response | Increased request latency, high CPU/memory usage | [Diagnosis Flow - Scenario 2](references/diagnosis-flow.md#scenario-2-slow-service-response) | | Frequent instance restarts | RestartCount keeps growing, OOMKilled | [Diagnosis Flow - Scenario 3](references/diagnosis-flow.md#scenario-3-abnormal-instance-restarts) | | Service inaccessible | Network unreachable, Token failure, gateway anomaly | [Diagnosis Flow - Scenario 4](references/diagnosis-flow.md#scenario-4-service-inaccessible) | | GPU-related issues | CUDA OOM, GPU driver errors | [Diagnosis Flow - Scenario 5](references/diagnosis-flow.md#scenario-5-gpu-related-issues) | --- ## Common Error Keywords | Keyword | Possible Cause | Reference | |---------|---------------|-----------| | `OOMKilled` | Out of memory | [Error Codes](references/error-codes.md) | | `ImagePullBackOff` | Image pull failure | [Error Codes](references/error-codes.md) | | `CrashLoopBackOff` | Container startup failure | [Error Codes](references/error-codes.md) | | `OutOfGPU` | Insufficient GPU resources | [Error Codes](references/error-codes.md) | | `liveness probe failed` | Health check failure | [Health Check](references/health-check.md) | --- ## Best Practices 1. **[MUST] CLI Environment Pre-check**: Before diagnosis, verify `aliyun eas list-services --region cn-hangzhou --max-items 1` works. If it fails, install EAS plugin first 2. **[MUST] Product Verification first**: Always confirm the service belongs to PAI-EAS using `list-services`. NEVER use FC, ECS, EDAS, or other product APIs to diagnose EAS services 3. **[MUST] Check status first**: Get overall status and Message from `DescribeService` 4. **[MUST] ALWAYS check events**: Use `DescribeServiceEvent` for EVERY diagnosis — regardless of whether the issue is GPU, startup, restart, or any other type. Events are critical for understanding the timeline 5. **[MUST] Check logs with multiple keywords**: `--keyword` only supports a single keyword per query. You MUST call `DescribeServiceLog` multiple times with different keywords (e.g., `--keyword "error"`, `--keyword "oom"`, `--keyword "killed"`, `--keyword "exit"`) 6. **[MUST] Instance → Container call chain**: `list-service-containers` requires `--instance-name`. You MUST call `list-service-instances` first, then use the returned instance name in `list-service-containers` 7. **[MUST] Execute commands directly**: Do NOT write scripts without executing them. Do NOT ask the user "should I proceed?" — just execute the diagnostic workflow autonomously 8. **[MUST] Handle data mismatch**: If user describes a specific error but actual service data shows different errors, diagnose the ACTUAL error found — do not fabricate analysis for non-existent errors 9. **[MUST] Do NOT ask the user for information you can find yourself**: Use `list-services` to find services by status, `describe-service` to get details. Do NOT ask for ServiceName, Cluster ID, or other information that can be obtained programmatically --- ## API and Command Tables | API | CLI Command | Description | |-----|------------|-------------| | DescribeService | `aliyun eas describe-service --cluster-id <region> --service-name <name>` | Query service details | | DescribeServiceLog | `aliyun eas describe-service-log --cluster-id <region> --service-name <name>` | Query service logs | | DescribeServiceEvent | `aliyun eas describe-service-event --cluster-id <region> --service-name <name>` | Query service events | | DescribeServiceDiagnosis | `aliyun eas describe-service-diagnosis --cluster-id <region> --service-name <name>` | Service diagnosis report | | ListServiceInstances | `aliyun eas list-service-instances --cluster-id <region> --service-name <name>` | List instances | | ListServiceContainers | `aliyun eas list-service-containers --cluster-id <region> --service-name <name> --instance-name <instance>` | List containers (requires --instance-name) | | DescribeServiceEndpoints | `aliyun eas describe-service-endpoints --cluster-id <region> --service-name <name>` | Service endpoints | | DescribeResource | `aliyun eas describe-resource --cluster-id <region> --resource-id <id>` | Resource group details | | DescribeGateway | `aliyun eas describe-gateway --cluster-id <region> --gateway-id <id>` | Gateway details | **Detailed CLI command reference**: [Related APIs](references/related-apis.md) --- ## Reference Links | Document | Purpose | |----------|---------| | [CLI Installation Guide](references/cli-installation-guide.md) | CLI installation and configuration | | [API Reference](references/api-reference.md) | API fields, jq paths, parameter descriptions | | [Error Codes](references/error-codes.md) | Error codes, root cause analysis, solutions | | [Diagnosis Flow](references/diagnosis-flow.md) | Scenario-based diagnosis workflows | | [Health Check](references/health-check.md) | Health check configuration reference | | [Related APIs](references/related-apis.md) | API and CLI command list | | [RAM Policies](references/ram-policies.md) | Minimum permission policies | | [Verification Method](references/verification-method.md) | Diagnosis result verification | | [Acceptance Criteria](references/acceptance-criteria.md) | Skill test acceptance criteria |


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)