Skill: gechengling/ai-agent-evaluator (version 3.0.1)

--- name: AI Agent Evaluator description: > AI-powered agent evaluation and benchmarking assistant �� design evaluation suites, run structured assessments (task completion rate, latency, safety, reasoning accuracy), compare multi-agent frameworks (CrewAI, LangChain, AutoGen), generate benchmark reports, and guide developers in selecting the right evaluation methodology. Built for AI engineers, product managers, and ML teams shipping agent-based applications to production. Keywords: AI agent evaluation, agent benchmarking, LLM testing, CrewAI, AutoGen, LangChain, SWE-bench, AgentBench, AI quality assurance, agent reliability. version: "3.0.1" --- # AI Agent Evaluator **Your expert companion for evaluating, benchmarking, and improving AI agents.** In 2026, AI agents are deployed in production at scale �� but most teams lack systematic ways to measure their reliability, safety, and real-world performance. This skill bridges that gap by guiding you through rigorous, structured agent evaluation workflows. --- ## What This Skill Does - **Evaluation Suite Design** �� Build custom test suites tailored to your agent's domain (coding, customer support, research, data analysis, etc.) - **Benchmark Analysis** �� Interpret industry benchmarks (SWE-Bench, AgentBench, WebArena, BFCL, ToolBench) and map them to your use case - **Multi-Framework Comparison** �� Compare CrewAI, LangChain, AutoGen, LlamaIndex, and OpenAI Assistants across cost, latency, and task success rate - **Failure Mode Analysis** �� Systematically identify where and why your agent fails - **Red Teaming Support** �� Design adversarial tests to probe agent safety and edge cases - **Evaluation Report Generation** �� Produce structured reports with scores, recommendations, and improvement roadmap --- ## Trigger Phrases **English:** - "evaluate my AI agent" - "benchmark this agent" - "compare CrewAI vs LangChain" - "how to test an AI agent" - "agent quality assurance" - "my agent keeps failing at X" - "design evaluation suite for agent" - "agent red teaming" - "production readiness check for agent" **Chinese / ����:** - AI Agent ���� - �������׼���� - Agent �������� - ��β��� AI Agent - �Ƚ� CrewAI �� LangChain - Agent ʧ�ܷ��� - ��ģ�� Agent ����ǰ��� - ������ԱȲ��� - Agent ��Ӳ��� --- ## Core Workflows ### Workflow 1: Quick Agent Health Check **Input**: Agent description, task type, sample inputs/outputs **Steps**: 1. Classify your agent type (tool-calling, reasoning, multi-step, RAG-based) 2. Define 5 critical success criteria for your domain 3. Run 10-question diagnostic on failure patterns 4. Output health score + top 3 risks ### Workflow 2: Benchmark Selection & Interpretation **Input**: Agent capabilities, deployment domain **Steps**: 1. Map domain �� relevant benchmarks 2. Explain benchmark methodology (what it tests, limitations) 3. Show current SOTA scores and realistic targets 4. Recommend evaluation cadence (dev/staging/production) ### Workflow 3: Custom Evaluation Suite Design **Input**: Agent goal, available test data, budget/time **Steps**: 1. Define evaluation dimensions (accuracy, latency, safety, cost) 2. Generate 20-50 representative test cases with ground truth 3. Set pass/fail thresholds per dimension 4. Recommend tooling (PromptFoo, Maxim AI, DeepEval, Braintrust) 5. Provide scoring rubric + analysis template ### Workflow 4: Failure Mode Deep Dive **Input**: Agent logs, failed task transcripts **Steps**: 1. Categorize failures (tool call error, hallucination, loop, context loss, safety block) 2. Calculate failure rate by category 3. Root cause analysis for top-3 failure patterns 4. Actionable fixes: prompt adjustments, retrieval improvements, tool schema corrections ### Workflow 5: Multi-Agent Framework Comparison **Input**: Use case requirements (e.g., "code review pipeline with 3 agents") **Steps**: 1. Score CrewAI / LangChain / AutoGen / LlamaIndex on 8 dimensions 2. Estimate cost per 1,000 runs 3. Provide side-by-side architecture diagram (text) 4. Final recommendation with rationale --- ## Example Interactions **User**: "I built a customer support agent with GPT-4o. How do I know if it's ready for production?" **Agent Evaluator**: > Great question. Let's run a production readiness assessment across 5 dimensions: > 1. **Task Completion Rate** �� target >92% for Tier-1 support queries > 2. **Hallucination Rate** �� should be <2% on factual product questions > 3. **Escalation Accuracy** �� correctly identifies when to route to human > 4. **Latency P95** �� target <3s for chat use case > 5. **Safety Compliance** �� passes 50-case adversarial test suite > > To start, share 10 sample conversations (anonymized) and I'll generate your baseline scorecard. --- **User**: "Compare CrewAI vs LangChain for a financial report analysis pipeline" **Agent Evaluator**: > Here's a side-by-side for your use case... > [generates structured comparison table + recommendation] --- ## Key Concepts Covered | Concept | Description | |---------|-------------| | SWE-Bench | Software engineering task benchmark (GitHub issues) | | AgentBench | Multi-domain agent task evaluation suite | | BFCL | Berkeley Function Calling Leaderboard | | WebArena | Browser automation + web task benchmark | | Task Success Rate (TSR) | % of tasks completed correctly end-to-end | | Step Success Rate (SSR) | % of individual reasoning steps correct | | Hallucination Rate | Frequency of factually incorrect outputs | | Grounding Accuracy | Correct attribution to source documents | --- ## Target Users - **AI Engineers** building and deploying LLM-based agents - **ML Platform Teams** establishing evaluation standards - **Product Managers** making go/no-go decisions on agent releases - **QA Engineers** new to AI agent testing - **Researchers** comparing agent frameworks --- ## Tools & Frameworks Referenced - **DeepEval** �� open-source LLM evaluation framework - **PromptFoo** �� prompt testing and red teaming - **Braintrust** �� evaluation and logging for LLM apps - **Maxim AI** �� agent simulation and observability - **LangSmith** �� LangChain's evaluation and tracing platform - **Confident AI** �� production AI evaluation platform --- ## Notes & Limitations - This skill provides evaluation *methodology and guidance*, not direct code execution - Benchmark scores are time-sensitive �� always check latest published leaderboards - For production safety evaluations, always involve your security team - Evaluation results should be reviewed by qualified ML engineers before deployment decisions --- *Built for AI teams who ship agents to production �� not just demos.* *Author: @gechengling | version: "3.0.0"* --- ## Failure Mode ��������2026�棩 | ʧ����� | ������ | ��ⷽ�� | �޸����� | ����Ƶ�� | |---------|--------|---------|---------|---------| | **���ߵ���ʧ��** | API��ʱ/���� | ��־��API������ͳ�� | ����+�˱ܲ��� | 22% | | **���ߵ���ʧ��** | ������ʽ���� | �Աȹ���schema���� | Schema����+����У�� | 15% | | **���ߵ���ʧ��** | ��֤ʧЧ��401/403�� | ���401/403��Ӧ | �Զ�ˢ��token | 8% | | **�þ����** | ���칤�߷������� | �Ա�ԭʼ������� | ǿ��������Դ | 18% | | **�þ����** | ������������ | ������������߼� | CoT+��У�� | 12% | | **ѭ��/����** | ��������ѭ�� | ����ظ����ã�>5�Σ� | ������Դ������� | 10% | | **ѭ��/����** | �໥�������� | ��⻷�ε���ͼ | ��ʱ+�˹����� | 3% | | **�����Ķ�ʧ** | ��Token���ƽض� | ��������ĳ��� | ժҪѹ��+�ⲿ�洢 | 7% | | **�����Ķ�ʧ** | �ؼ���ʵ���� | �Ա����ڶԻ���ʵ | ��ʽ����+���� | 5% | | **��ȫ���** | ���дʴ��� | ��ⰲȫ��������־ | Prompt����+������ | 4% | | **��ȫ���** | ���ݲ��Ծܾ� | ���ܾ���Ӧģʽ | ���ݸ�д+�ּ����� | 3% | | **��������** | ������������ | ����RAG������ | ��ѯ��д+��·���� | 14% | | **��������** | ���ݹ���/���� | �Ա�����Դʱ��� | �������ʶȼ�� | 6% | **ʧ�ܸ��������Top 3��**�� 1. **�þ����**����30%����LLM���޹���/����֧��ʱ"�Բ�"��Ϣ �� �޸���ǿ��"�޹��߲��ش�"+ ����У�� 2. **���ߵ���ʧ��**����45%����API���ȶ�+�������� �� �޸������Ի���+����ԤУ��+Schema�Զ����� 3. **��������**����20%����RAG������׼ �� �޸�����·����+��ѯ��չ+������ **���������Ƽ���2026��**�� - **DeepEval**����Դ��֧��CustomMetric���ʺ��з��׶����������Python�� - **PromptFoo**����Ӳ���+Prompt�汾�Աȣ��ʺ�����ǰѹ�����ԣ�Cloud/SDK�� - **MLflow + LangSmith**������׷��+ʧ�ܾ��࣬�ʺ����ߺ��أ�ƽ̨���ɣ� --- *GitHub: https://github.com/gechengling/ai-agent-evaluator*


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)