Skill: bullkis1/ai-engineer (version 1.0.0)

--- name: ai-engineer description: >- AI/ML engineering specialist for building intelligent features, RAG systems, LLM integrations, data pipelines, vector search, and AI-powered applications. Use when building anything involving: LLMs, embeddings, vector databases, RAG, fine-tuning, prompt engineering, AI agents, ML pipelines, or deploying models to production. NOT for general web dev (use rapid-prototyper) or simple API calls. --- # AI Engineer Build practical AI systems that work in production. Data-driven, systematic, performance-focused. ## Core Capabilities - **LLM Integration**: OpenAI, Anthropic, local models (Ollama, llama.cpp), LiteLLM - **RAG Systems**: Chunking, embeddings, vector search, retrieval, re-ranking - **Vector DBs**: Chroma (local), Pinecone (managed), Weaviate, FAISS, Qdrant - **Agents & Tools**: Tool-calling, multi-step agents, OpenClaw sub-agents - **Data Pipelines**: Ingestion, cleaning, transformation, feature engineering - **MLOps**: Model versioning (MLflow), monitoring, drift detection, A/B testing - **Evaluation**: Benchmark construction, bias testing, performance metrics ## Decision Framework ### Which LLM provider? - **Prototyping/speed**: OpenAI GPT-4o or Anthropic Claude Sonnet - **Local/private**: Ollama + Qwen 2.5 32B or Llama 3.3 70B - **Multi-provider abstraction**: LiteLLM (swap models without code changes) - **Embeddings**: text-embedding-3-small (OpenAI) or nomic-embed-text (local) ### Which vector DB? - **Local/dev**: Chroma (zero setup) - **Production managed**: Pinecone - **Self-hosted production**: Qdrant or Weaviate - **Already in Postgres**: pgvector extension ### RAG or fine-tuning? - **RAG first** — always try RAG before fine-tuning. 90% of cases RAG is enough. - Fine-tune only when: style/tone change needed, domain vocab is highly specialized, latency must be minimal ## RAG Workflow ### 1. Ingest ```python # Chunk documents (rule of thumb: 512 tokens, 50 overlap) from langchain.text_splitter import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50) chunks = splitter.split_documents(docs) ``` ### 2. Embed + store ```python import chromadb from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction client = chromadb.PersistentClient(path="./chroma_db") ef = OpenAIEmbeddingFunction([REDACTED_SECRET]"OPENAI_API_KEY"], model_name="text-embedding-3-small") collection = client.get_or_create_collection("docs", embedding_function=ef) collection.add(documents=[c.page_content for c in chunks], ids=[str(i) for i in range(len(chunks))]) ``` ### 3. Retrieve + generate ```python results = collection.query(query_texts=[user_query], n_results=5) context = "\n\n".join(results["documents"][0]) response = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "system", "content": f"Answer based on this context:\n{context}"}, {"role": "user", "content": user_query}, ] ) ``` See `references/rag-patterns.md` for advanced patterns: re-ranking, hybrid search, HyDE, eval. ## LLM Tool Calling (Agents) ```python tools = [{ "type": "function", "function": { "name": "search_docs", "description": "Search internal documentation", "parameters": { "type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"] } } }] response = openai.chat.completions.create(model="gpt-4o", messages=messages, tools=tools) ``` See `references/agent-patterns.md` for multi-step agent loops, error handling, tool schemas. ## Critical Rules - **Evaluate early** — build an eval set before you build the system - **RAG before fine-tuning** — always - **Log everything** — prompts, completions, latency, token usage from day one - **Test for bias** — especially for user-facing classification or scoring systems - **Never hardcode API keys** — use env vars or secret managers ## References - `references/rag-patterns.md` — Chunking strategies, re-ranking, HyDE, hybrid search, evaluation - `references/agent-patterns.md` — Tool calling, multi-step loops, memory, error handling

path: references/agent-patterns.md
content: # Agent Patterns

## Basic tool-calling loop

```python
import json, openai

def run_agent(user_message: str, tools: list, tool_handlers: dict):
    messages = [{"role": "user", "content": user_message}]
    
    while True:
        response = openai.chat.completions.create(
            model="gpt-4o", messages=messages, tools=tools
        )
        msg = response.choices[0].message
        messages.append(msg)

        if not msg.tool_calls:
            return msg.content  # Final answer

        for call in msg.tool_calls:
            fn_name = call.function.name
            fn_args = json.loads(call.function.arguments)
            result = tool_handlers[fn_name](**fn_args)
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": json.dumps(result),
            })
```

## Multi-step with max iterations

Always cap iterations to avoid infinite loops:

```python
MAX_ITERATIONS = 10

for i in range(MAX_ITERATIONS):
    response = llm.call(messages, tools)
    if response.stop_reason == "end_turn":
        break
    # handle tool calls...
else:
    raise RuntimeError("Agent exceeded max iterations")
```

## Memory patterns

**Short-term (in-context):** Just the message history. Good for single session.

**Long-term (external store):**
```python
# On each turn: retrieve relevant memories
memories = memory_store.search(query=user_message, limit=3)
system_prompt = f"Relevant context:\n{memories}\n\n{base_system_prompt}"
```

**Episodic memory:** Save summaries of completed tasks:
```python
summary = llm.summarize(conversation_history)
memory_store.save(summary, metadata={"timestamp": now, "task": task_type})
```

## Error handling

```python
try:
    result = tool_handler(**args)
except Exception as e:
    # Return error as tool result — let LLM decide what to do
    result = {"error": str(e), "suggestion": "Try with different parameters"}
```

Never crash the agent on tool failure — let it recover gracefully.

## Structured output (Pydantic)

```python
from pydantic import BaseModel
from openai import OpenAI

class TaskPlan(BaseModel):
    steps: list[str]
    estimated_time: str
    risks: list[str]

client = OpenAI()
response = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Plan this project: ..."}],
    response_format=TaskPlan,
)
plan = response.choices[0].message.parsed
```

## OpenClaw sub-agent delegation

When a task is too big for one context, spawn a sub-agent:

```python
# In OpenClaw: use sessions_spawn with runtime="subagent"
# Pass specific task + relevant context
# Poll for completion or let it push-notify
```

## Cost tracking

```python
# Always log usage
usage = response.usage
cost = (usage.prompt_tokens * 0.005 + usage.completion_tokens * 0.015) / 1000
logger.info(f"Turn cost: ${cost:.4f} | Total: ${running_total:.4f}")
```


path: references/rag-patterns.md
content: # RAG Patterns

## Chunking strategies

| Strategy | When to use |
|---|---|
| Fixed size (512 tokens, 50 overlap) | Default — works for most docs |
| Sentence-based | Conversational content, FAQs |
| Semantic chunking | Long docs where paragraphs are coherent units |
| Parent-child | Retrieve small chunks, return larger context window |

## Hybrid search (keyword + semantic)

Combine BM25 (keyword) with vector search for better recall:

```python
# With Qdrant
from qdrant_client import QdrantClient
from qdrant_client.models import SparseVectorParams, VectorParams

# Store both dense and sparse vectors
# Query both and merge with RRF (Reciprocal Rank Fusion)
```

Rule of thumb: hybrid search beats pure vector search for factual retrieval.

## Re-ranking

After initial retrieval (top 20), re-rank with a cross-encoder before passing to LLM:

```python
from sentence_transformers import CrossEncoder
reranker = CrossEncoder([REDACTED_SECRET])
scores = reranker.predict([(query, doc) for doc in retrieved_docs])
top_docs = [doc for _, doc in sorted(zip(scores, retrieved_docs), reverse=True)][:5]
```

## HyDE (Hypothetical Document Embeddings)

Generate a hypothetical answer first, embed that, then search:

```python
hypothetical = llm.complete(f"Write a short answer to: {query}")
results = collection.query(query_texts=[hypothetical], n_results=5)
```

Good for: queries that are very different in style from the documents.

## Evaluation

Build an eval set BEFORE building the system:

```python
eval_set = [
    {"question": "What is the refund policy?", "expected_answer": "30 days"},
    ...
]

# Metrics to track:
# - Retrieval recall: is the right chunk retrieved?
# - Answer faithfulness: does the answer match retrieved context?
# - Answer relevance: does the answer address the question?
```

Tools: RAGAS (automated RAG eval), TruLens, LangSmith.

## Context window management

- Max 4–6 chunks in context (quality > quantity)
- Filter duplicates before passing to LLM
- Add source metadata: `[Source: doc_name, page: 3]`
- If chunks > 6k tokens total → summarize before passing


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)