npm package: @jsoma/piece-opik
Description: Opik piece for ActivePieces - LLM observability, evaluation, and prompt management
Author: jsoma
Latest version: 0.1.0

# Opik Piece for ActivePieces

Integrate [Opik](https://www.comet.com/opik) LLM observability and evaluation platform into your ActivePieces workflows to track, monitor, and improve AI-powered automations.

## Overview

This piece enables you to:
- **Track LLM workflows** with detailed traces and spans
- **Manage prompts** with versioning from Opik's prompt library
- **Apply guardrails** for PII detection and topic moderation
- **Log feedback scores** to measure quality and performance
- **Debug issues** with comprehensive observability

Perfect for newsrooms and content teams building AI workflows that need visibility, quality control, and continuous improvement.

## Installation

Add this piece to your ActivePieces instance through the admin panel or by installing the npm package:

```bash
npm install @activepieces/piece-opik
```

## Configuration

### Authentication

The Opik piece supports both:
- **Opik Cloud**: Use your API key from [Comet.com](https://www.comet.com)
- **Self-hosted Opik**: Connect to your local instance

Configuration fields:
- **Opik URL**: Your Opik instance URL
  - Cloud: `https://www.comet.com/opik/api`
  - Self-hosted: `http://localhost:5173/api`
- **API Key**: Required for Opik Cloud (optional for self-hosted)
- **Workspace**: Optional workspace name

## Available Actions

### 🔍 Tracing Actions

#### Start Trace
Begin tracking a workflow execution.

**Inputs:**
- `name`: Descriptive trace name
- `input`: Input data (JSON)
- `metadata`: Additional context (JSON)
- `tags`: Categories for organization
- `thread_id`: Group related traces
- `project_name`: Opik project

**Output:** `trace_id` for use in subsequent actions

#### End Trace
Complete a trace with final results.

**Inputs:**
- `trace_id`: ID from Start Trace
- `output`: Final output (JSON)
- `metadata`: Additional metadata
- `error`: Error message if failed

#### Log Span
Track individual steps within a trace.

**Inputs:**
- `trace_id`: Parent trace ID
- `name`: Step name
- `type`: LLM/Tool/Agent/General
- `input`/`output`: Step data
- `metadata`: Additional context
- Token usage for LLM calls

**Output:** `span_id` for nested spans or feedback

### 📝 Prompt Management

#### Get Prompt
Retrieve prompt templates from Opik.

**Inputs:**
- `name`: Prompt name
- `version`: Specific version (optional, latest by default)

**Output:**
- `template`: The prompt template
- `metadata`: Associated metadata
- `version`: Version number

### 🛡️ Quality Control

#### Check Guardrails
Screen content for issues.

**Types:**
- **PII Detection**: Find and redact sensitive information
- **Topic Moderation**: Ensure content stays on-topic

**Inputs:**
- `text`: Content to check
- `type`: PII or Topic
- Configuration for allowed/disallowed topics or PII entities

**Output:**
- `passed`: Whether check passed
- `violations`: List of issues found
- `redacted_text`: Cleaned version (for PII)

#### Log Feedback
Score traces or spans for quality metrics.

**Inputs:**
- `entity_type`: Trace or Span
- `entity_id`: ID to score
- `name`: Metric name (accuracy, relevance, etc.)
- `value`: Numeric score
- `reason`: Optional explanation

## Example Workflow: Newsroom Tip Processing

```yaml
1. Trigger: New tip submission via form
2. Opik: Start Trace
   - name: "tip_processing"
   - metadata: {source: "web_form", date: "2024-01-15"}

3. Opik: Check Guardrails (PII)
   - type: "pii"
   - text: [tip content]

4. Opik: Get Prompt
   - name: "tip_evaluator"

5. LLM: Evaluate tip relevance
   - prompt: [from step 4]
   - input: [redacted tip from step 3]

6. Opik: Log Span
   - name: "evaluation"
   - type: "llm"
   - input/output: [LLM data]

7. Branch: If relevant
   a. Opik: Get Prompt ("follow_up_questions")
   b. LLM: Generate questions
   c. Opik: Log Span ("question_generation")
   d. Opik: Get Prompt ("beat_categorizer")
   e. LLM: Categorize beat
   f. Opik: Check Guardrails (Topic - valid beats)
   g. Google Sheets: Add row

8. Opik: Log Feedback
   - name: "tip_quality"
   - value: [0-10 based on relevance]

9. Opik: End Trace
   - output: {processed: true, beat: "politics"}
```

## Use Cases

### Newsroom Workflows
- **Tip evaluation**: Assess reader submissions
- **Content moderation**: Screen for PII and off-topic content
- **Beat categorization**: Route stories to appropriate teams
- **Quality tracking**: Monitor prompt effectiveness

### General AI Workflows
- **Customer support**: Track chatbot interactions
- **Content generation**: Monitor article/report creation
- **Data extraction**: Trace document processing
- **API monitoring**: Track LLM API usage and costs

## Best Practices

1. **Always use Start/End Trace pairs** to capture complete workflow execution
2. **Log Spans for key steps** to identify bottlenecks and failures
3. **Use meaningful names** for traces and spans for easier debugging
4. **Apply guardrails early** in workflows to prevent downstream issues
5. **Version your prompts** in Opik for A/B testing and rollback
6. **Log feedback scores** to track quality over time
7. **Use thread_id** to group related workflows (e.g., all tips from same source)
8. **Include metadata** for filtering and analysis in Opik dashboard

## Debugging

Common issues and solutions:

### Authentication Failed
- Verify API key is correct for Opik Cloud
- Check URL format (include `/api` suffix)
- Ensure self-hosted instance is running

### Trace Not Appearing
- Confirm trace was ended with End Trace action
- Check project_name matches your Opik project
- Verify network connectivity to Opik server

### Guardrails Not Working
- Ensure guardrails are enabled in your Opik instance
- Check configuration for topic lists or PII entities
- Verify text input is not empty

## Advanced Features

### Nested Spans
Create hierarchical traces by using `parent_span_id`:
```javascript
1. Log Span (name: "research_phase") → span_id: "abc123"
2. Log Span (name: "web_search", parent_span_id: "abc123")
3. Log Span (name: "summarization", parent_span_id: "abc123")
```

### Custom Metadata
Add searchable metadata to traces and spans:
```json
{
  "department": "politics",
  "reporter": "jane.doe",
  "priority": "high",
  "word_count": 500
}
```

### Batch Processing
Process multiple items with thread grouping:
```javascript
thread_id: "daily_tips_2024_01_15"
// All tips from the same day share thread_id
```

## Development

### Building from Source
```bash
npm install
npm run build
```

### Testing
```bash
npm test
```

### Contributing
Contributions welcome! Please submit PRs with:
- New actions for Opik features
- Bug fixes
- Documentation improvements
- Example workflows

## Resources

- [Opik Documentation](https://www.comet.com/docs/opik/)
- [ActivePieces Docs](https://www.activepieces.com/docs)
- [Example Workflows](./examples)
- [API Reference](https://www.comet.com/docs/opik/reference/rest-api/)

## License

MIT