npm package: @gulibs/safe-coder
Description: MCP Server for documentation crawling orchestration - calls safe-coder-cli for actual crawling
Latest version: 0.0.30

# Safe Coder MCP Server

> **Version:** 0.0.27  
> **Status:** Development Preview  
> **Architecture:** MCP Server (orchestration layer)

MCP Server for documentation crawling orchestration. Coordinates with `@gulibs/safe-coder-cli` to crawl documentation websites, process content, and provide structured guidance for creating Cursor AI SKILL files.

## ⚠️ Current Version Status

**Version 0.0.27 - Development Preview:**
- ✅ Architecture refactored: Clean separation between MCP Server and CLI
- ✅ Core functionality: Crawling orchestration, content processing, SKILL guidance
- ✅ **NEW: `save_skill` tool** - Automatic SKILL file creation with complete directory structure
- ⚠️ **Not published to npm** - requires manual installation from source

## What This Package Does

**Safe Coder MCP Server is an orchestration layer** that:

1. ✅ **Checks dependencies** - Verifies `safe-coder-cli` is installed
2. ✅ **Spawns CLI process** - Executes crawling via child process
3. ✅ **Parses CLI output** - Reads JSON from stdout, progress from stderr
4. ✅ **Processes content** - Extracts key points, code patterns, topics
5. ✅ **Generates guidance** - Creates structured SKILL generation recommendations
6. ✅ **Manages cache** - Stores results for 7 days in `~/.safe-coder/cache/`
7. ✅ **Saves SKILL files** - Creates complete directory structure (SKILL.md + references/)
8. ✅ **Exposes MCP tools** - Provides 4 tools to Cursor AI

**This package does NOT:**
- ❌ Perform web crawling (delegated to CLI)
- ❌ Run browser automation (CLI responsibility)

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│  Cursor / Claude Desktop (MCP Client)                      │
│  • Calls MCP tools                                         │
│  • Receives data + guidance                                │
│  • Creates SKILL.md based on guidance                      │
└────────────────┬────────────────────────────────────────────┘
                 │ MCP Protocol (JSON-RPC over stdio)
┌────────────────▼────────────────────────────────────────────┐
│  @gulibs/safe-coder (MCP Server) - THIS PACKAGE            │
│  ┌──────────────────────────────────────────────────────┐  │
│  │ 1. Dependency Check (which safe-coder-cli)           │  │
│  │ 2. CLI Executor (spawn child process)                │  │
│  │ 3. Parse JSON Output (stdout → data)                 │  │
│  │ 4. Content Processor (extract key info)              │  │
│  │ 5. Guide Generator (create SKILL guidance)           │  │
│  │ 6. Cache Manager (store/retrieve results)            │  │
│  └──────────────────────────────────────────────────────┘  │
│                                                             │
│  Tools: crawl_documentation, save_skill,                   │
│         list_cached_documentation, clear_cache            │
└────────────────┬────────────────────────────────────────────┘
                 │ spawn('safe-coder-cli', ['crawl', url, ...])
                 │ stdin: commands, stdout: JSON, stderr: progress
┌────────────────▼────────────────────────────────────────────┐
│  @gulibs/safe-coder-cli (Standalone CLI Tool)              │
│  • HTTP + Browser-based web crawling                       │
│  • Content extraction and markup conversion                │
│  • JSON output to stdout (results)                         │
│  • JSON output to stderr (progress messages)               │
└────────────────┬────────────────────────────────────────────┘
                 │ HTTP requests / Puppeteer / Playwright
┌────────────────▼────────────────────────────────────────────┐
│  Documentation Website (e.g., react.dev, vuejs.org)        │
└─────────────────────────────────────────────────────────────┘
```

## Installation

### Step 1: Prerequisites

**Node.js 18+**
```bash
node --version  # Must be v18.0.0 or higher
```

### Step 2: Install CLI Tool (Required)

⚠️ **The MCP server requires `@gulibs/safe-coder-cli` to be globally installed.**

**Option A: From npm (after publication)**
```bash
npm install -g @gulibs/safe-coder-cli
```

**Option B: From source (current development setup)**
```bash
# Clone and build CLI
cd /path/to/safe-coder-cli
npm install
npm run build
npm link  # Makes 'safe-coder-cli' available globally

# Verify
safe-coder-cli --version
# Output: 0.0.1
```

### Step 3: Install MCP Server

⚠️ **Not yet published to npm - use local installation:**

**Development Installation:**
```bash
cd /path/to/safe-coder
npm install
npm run build

# Optional: Link for global access
npm link
```

### Step 4: Configure Cursor

**Edit Cursor MCP configuration:**

File location: `~/Library/Application Support/Cursor/User/globalStorage/mcp.json`

```json
{
  "mcpServers": {
    "safe-coder": {
      "command": "node",
      "args": ["/absolute/path/to/safe-coder/dist/index.js"]
    }
  }
}
```

**Replace `/absolute/path/to/safe-coder`** with actual path, e.g.:
```json
{
  "mcpServers": {
    "safe-coder": {
      "command": "node",
      "args": ["/Users/yourname/projects/mcp/safe-coder/dist/index.js"]
    }
  }
}
```

**Future (after npm publication):**
```json
{
  "mcpServers": {
    "safe-coder": {
      "command": "npx",
      "args": ["-y", "@gulibs/safe-coder@latest"]
    }
  }
}
```

### Step 5: Restart Cursor

Restart Cursor to load the MCP server.

## Usage: SKILL File Generation Workflow

### ⚠️ Important: How SKILL Files Are Created

**The MCP server does NOT create `SKILL.md` files directly.** Here's the actual workflow:

#### What the MCP Server Returns

1. **Crawled content** - Full text from all pages
2. **Processed data** - Key points, code patterns, categories, topics
3. **SKILL generation guide** - Structured recommendations for SKILL.md
4. **Quality assessment** - Scores for content quality, completeness

#### Where SKILL Files Are Saved

**You must explicitly tell Cursor AI where to save the file:**

```
User: "Crawl React docs and create a SKILL file"
      ↓
Cursor AI: [Calls crawl_documentation, receives data + guidance]
      ↓
User: "Save it to ~/.claude/skills/react-docs/SKILL.md"
      ↓
Cursor AI: [Creates file at specified location]
```

**Recommended location pattern:**
```
~/.claude/skills/
├── react-docs/SKILL.md          # React documentation
├── vue-guide/SKILL.md           # Vue.js guide
├── typescript-handbook/SKILL.md # TypeScript handbook
└── [tech-name]/SKILL.md         # Generic pattern
```

**Naming conventions:**
- Directory: `[technology-name]` (lowercase, hyphens)
- File: Always `SKILL.md` (Cursor convention)
- Examples: `react-docs`, `vue-guide`, `next-js-docs`

### Complete Example Workflow

**1. Request crawl:**
```
User: "Crawl the React documentation at https://react.dev and create a SKILL file"
```

**2. Cursor AI calls MCP tool:**
```typescript
crawl_documentation({
  url: "https://react.dev",
  maxPages: 200,  // Default
  depth: 3        // Default
})
```

**3. MCP Server process:**
```
[Dependency Check] ✓ safe-coder-cli found at /usr/local/bin/safe-coder-cli
[Spawning CLI] safe-coder-cli crawl https://react.dev --output-format json ...
[Progress] Crawling... 10/50 pages
[Progress] Crawling... 25/50 pages
[Progress] Crawling... 50/50 pages complete
[Processing] Analyzing content...
[Processing] Generating SKILL guidance...
[Cache] Saving to ~/.safe-coder/cache/aHR0cHM6Ly9yZWFjdC5kZXY=.json
[Complete] Ready to return
```

**4. MCP returns to Cursor:**
```json
{
  "success": true,
  "data": {
    "source": {
      "url": "https://react.dev",
      "crawledAt": "2026-01-15T14:30:00Z",
      "pageCount": 50,
      "depth": 3
    },
    "pages": [
      {
        "url": "https://react.dev/learn",
        "title": "Quick Start",
        "content": "Welcome to React...",
        "keyPoints": [
          "React is a JavaScript library",
          "Components are building blocks",
          "...more..."
        ],
        "codePatterns": [
          {
            "language": "jsx",
            "pattern": "function Component() { return <div>...</div> }",
            "description": "Functional component syntax"
          }
        ],
        "wordCount": 1500,
        "codeBlocks": 12
      }
      // ... 49 more pages
    ],
    "analysis": {
      "mainTopics": ["Components", "Hooks", "State", "Props", "Effects"],
      "difficulty": "intermediate",
      "technicalDensity": 0.65
    }
  },
  "skillGenerationGuide": {
    "suggestedStructure": [
      "Overview",
      "Quick Start", 
      "Core Concepts",
      "Hooks Reference",
      "Best Practices",
      "Common Patterns",
      "Troubleshooting"
    ],
    "keyTopics": [
      "JSX Syntax",
      "Components (Functional & Class)",
      "Props and State",
      "Hooks (useState, useEffect, useContext, etc.)",
      "Event Handling",
      "Conditional Rendering",
      "Lists and Keys"
    ],
    "recommendedSections": {
      "overview": "React is a JavaScript library for building user interfaces...",
      "quickStart": [
        "Install: npx create-react-app my-app",
        "Create first component",
        "Understand JSX"
      ],
      "coreFeatures": [
        "Component-based architecture",
        "Virtual DOM",
        "One-way data flow",
        "Declarative UI"
      ],
      "bestPractices": [
        "Use functional components with Hooks",
        "Keep components small and focused",
        "Lift state up when needed"
      ]
    },
    "tips": [
      "Focus on the Hooks API (modern approach)",
      "Include examples for common Hooks",
      "Explain component lifecycle clearly",
      "Cover both client and server rendering"
    ],
    "quality": {
      "contentQuality": 90,
      "codeExamples": 85,
      "completeness": 88,
      "overall": 87
    }
  },
  "fromCache": false,
  "message": "Successfully crawled 50 pages. Quality score: 87/100"
}
```

**5. Tell Cursor where to save:**
```
User: "Save this as a SKILL file at ~/.claude/skills/react-docs/SKILL.md"
```

**6. Cursor AI generates and saves:**
- Uses `data.pages` for content
- Uses `skillGenerationGuide` for structure
- Creates formatted `SKILL.md`
- Saves to specified location

## MCP Tools Reference

### Tool 1: `crawl_documentation` 

(**Recommended Workflow**: Use with `save_skill` for complete automation)

Crawl documentation website and return processed data with SKILL generation guidance.

**Parameters:**

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `url` | `string` | ✅ Yes | - | Documentation URL to crawl |
| `maxPages` | `number` | No | `200` | Maximum pages to crawl |
| `depth` | `number` | No | `3` | Maximum crawl depth |
| `workers` | `number` | No | `1` | Number of parallel workers |
| `useBrowser` | `boolean` | No | `false` | Force browser automation |
| `browser` | `'puppeteer'` \| `'playwright'` | No | `'puppeteer'` | Browser type |
| `spaStrategy` | `'smart'` \| `'auto'` \| `'manual'` | No | `'smart'` | SPA detection |
| `useCache` | `boolean` | No | `true` | Use cached results |
| `forceRefresh` | `boolean` | No | `false` | Bypass cache |

**⚠️ Parameters NOT Currently Exposed (CLI supports, MCP doesn't):**
- `strategy` (bfs/dfs) - Crawl strategy
- `checkpoint`, `resume` - Checkpoint/resume functionality
- `rateLimit`, `maxRetries`, `retryDelay` - Rate limiting and retry control
- `includePaths`, `excludePaths` - URL path filtering
- `skipLlmsTxt` - llms.txt detection control

**Returns:**
```typescript
{
  success: boolean;
  data: {
    source: { url, crawledAt, pageCount, depth };
    pages: Array<{
      url, title, content, keyPoints, codePatterns,
      category, wordCount, codeBlocks, headings
    }>;
    metadata: { technology, version, documentType, categories };
    statistics: { totalPages, maxDepthReached, errors, linkDiscovery };
    analysis: {
      totalWords, totalCodeBlocks, categories,
      mainTopics, technicalDensity, difficulty
    };
  };
  skillGenerationGuide: {
    suggestedStructure: string[];
    keyTopics: string[];
    recommendedSections: {
      overview, quickStart, coreFeatures, bestPractices, troubleshooting
    };
    tips: string[];
    quality: { contentQuality, codeExamples, completeness, overall };
  };
  fromCache: boolean;
  cacheAge?: string;  // If from cache
  message: string;
}
```

**Example Cursor prompts:**

```bash
# Basic usage
"Crawl https://react.dev and create a SKILL"

# Limit pages
"Crawl Vue.js docs but only 50 pages"

# Force refresh
"Re-crawl React docs, ignore cache"

# Use browser for SPA
"Crawl https://app.example.com with Playwright"
```

### Tool 2: `list_cached_documentation`

List all cached documentation crawl results with metadata.

**Parameters:** None

**Returns:**
```typescript
{
  success: boolean;
  entries: Array<{
    url: string;
    technology?: string;
    pageCount: number;
    cachedAt: string;  // ISO timestamp
    age: string;       // Human-readable, e.g., "2 hours ago"
    size: string;      // Human-readable, e.g., "2.5 MB"
  }>;
  statistics: {
    totalEntries: number;
    totalSize: number;  // Bytes
    oldestEntry?: Date;
    newestEntry?: Date;
  };
  message: string;
}
```

**Example Cursor prompts:**
```bash
"Show me all cached documentation"
"List cached docs"
"What's in the cache?"
```

### Tool 3: `save_skill` ⭐ NEW

**Save complete SKILL directory structure to filesystem.**

Creates a properly formatted Cursor Agent Skill with:
- `SKILL.md` - Main skill file with metadata
- `references/` - Reference documentation organized by category
- Standard directory structure compatible with Cursor

**Parameters:**

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `skillName` | `string` | ✅ Yes | - | Name for skill directory (e.g., "react-docs") |
| `skillContent` | `string` | ✅ Yes | - | Content for SKILL.md file |
| `referenceFiles` | `Record<string, string>` | No | `{}` | Map of category → markdown content |
| `baseDirectory` | `string` | No | `~/.claude/skills` | Base directory for skills |
| `overwrite` | `boolean` | No | `false` | Overwrite existing skill |

**Returns:**
```typescript
{
  success: boolean;
  skillDirectory: string;            // Absolute path to skill directory
  filesCreated: {
    skillMd: string;                 // Path to SKILL.md
    references: string[];            // Paths to reference files
  };
  message: string;
}
```

**Directory Structure Created:**
```
~/.claude/skills/
└── [skillName]/
    ├── SKILL.md                     # Main skill file
    └── references/                  # Reference documentation
        ├── api.md                   # API category
        ├── guides.md                # Guides category
        ├── getting_started.md       # Getting started
        └── ...                      # Other categories
```

**Example Cursor Prompts:**

```bash
# After crawling, extract and save skill data
"Use the crawled React data to create a complete SKILL:
 - skillName: 'react-docs'
 - skillContent: [generated from guidance]
 - referenceFiles: [organized by category]"

# Save with custom location
"Save the Vue skill to ~/my-skills/"

# Overwrite existing
"Update the react-docs skill with new content"
```

**Automated Workflow Example:**

```
User: "Crawl React docs and save as a complete SKILL"

Step 1: Cursor calls crawl_documentation
Step 2: Receives data with pages organized by category
Step 3: Cursor calls save_skill with:
  {
    skillName: "react-docs",
    skillContent: "[Generated SKILL.md from guidance]",
    referenceFiles: {
      "api": "[API pages content]",
      "guides": "[Guide pages content]",
      "getting_started": "[Getting started content]"
    }
  }
Step 4: Tool creates complete directory structure
Step 5: Returns paths to all created files
```

### Tool 4: `clear_cache`

Clear cached documentation results by pattern, all entries, or expired only.

**Parameters:**

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `urlPattern` | `string` | No | - | URL pattern to match (substring) |
| `clearAll` | `boolean` | No | `false` | Clear all cached entries |

**Behavior:**
- If `clearAll: true` → Delete all cached entries
- If `urlPattern` provided → Delete entries matching pattern
- If neither → Delete only expired entries (>7 days old)

**Returns:**
```typescript
{
  success: boolean;
  deletedCount: number;
  message: string;
}
```

**Example Cursor prompts:**
```bash
# Clear all
"Clear all cached documentation"

# Clear by pattern
"Clear cache for react.dev"
"Delete cached Vue docs"

# Clear expired only
"Clear old cached docs"
```

## Internal Workflow

### 1. Dependency Check

```typescript
const checker = new DependencyChecker();
const check = await checker.checkCLIInstalled();

if (!check.installed) {
  throw new Error(`
⚠️  safe-coder-cli is not installed.

Please install globally:
  npm install -g @gulibs/safe-coder-cli

Or from source:
  cd /path/to/safe-coder-cli
  npm install && npm run build && npm link

Then verify:
  safe-coder-cli --version
  `);
}
```

### 2. CLI Execution

```typescript
const executor = new CLIExecutor(checker);
const result = await executor.executeCrawl({
  url: 'https://react.dev',
  maxPages: 200,
  maxDepth: 3,
  // ... other params
}, (progress) => {
  console.error(`[Progress] ${progress.message}`);
});
```

**Under the hood:**
```bash
spawn('safe-coder-cli', [
  'crawl', 
  'https://react.dev',
  '--output-format', 'json',
  '--max-pages', '200',
  '--depth', '3'
])

# stdout → JSON result
# stderr → Progress messages
```

### 3. Content Processing

```typescript
const processor = new ContentProcessor();
const processed = await processor.process(rawResult);

// Extracts:
// - Key points from each page
// - Code patterns and examples
// - Content categories
// - Technical density
// - Difficulty level
```

### 4. SKILL Guidance Generation

```typescript
const guideGenerator = new GuideGenerator();
const guide = guideGenerator.generate(processed);

// Creates:
// - Suggested SKILL structure
// - Key topics to cover
// - Recommended sections
// - Writing tips
// - Quality assessment
```

### 5. Caching

```typescript
const cache = new CacheManager();

// Save (7-day TTL)
await cache.set(url, processed, guide);

// Retrieve
const cached = await cache.get(url);
if (cached && !forceRefresh) {
  return cached;
}
```

**Cache location:** `~/.safe-coder/cache/`  
**Cache format:** JSON files with base64-encoded URL as filename  
**TTL:** 7 days (default)

## Error Handling

### CLI Not Installed

```
Error: safe-coder-cli is not installed

Please install:
  npm install -g @gulibs/safe-coder-cli

Current PATH checked:
  /usr/local/bin
  /usr/bin
  /bin
  ...
```

### CLI Execution Failed

```
Error: CLI process exited with code 1

stderr output:
  [Error] Failed to crawl: Network timeout
  [Error] URL: https://example.com

Possible solutions:
  - Check network connectivity
  - Try with --use-browser flag
  - Increase timeout with --rate-limit
```

### Cache Errors

```
Warning: Failed to cache results
Error: EACCES: permission denied, mkdir '~/.safe-coder/cache'

Continuing without cache...
```

Cache errors are non-fatal - the operation continues without caching.

## Configuration

### Cache Settings

Modify cache behavior by editing `src/cache/cache-manager.ts`:

```typescript
const cache = new CacheManager(
  '/custom/cache/path',  // Default: ~/.safe-coder/cache
  14                      // TTL in days (default: 7)
);
```

### CLI Arguments Mapping

| MCP Parameter | CLI Argument |
|---------------|--------------|
| `url` | `<url>` (positional) |
| `maxPages` | `--max-pages <number>` |
| `depth` | `--max-depth <number>` |
| `workers` | `--workers <number>` |
| `useBrowser` | (implicitly sets `--spa-strategy auto`) |
| `browser` | `--browser <type>` |
| `spaStrategy` | `--spa-strategy <type>` |
| - | `--output-format json` (forced) |

## Troubleshooting

### MCP Server Not Appearing in Cursor

1. **Check configuration file exists:**
   ```bash
   cat ~/Library/Application\ Support/Cursor/User/globalStorage/mcp.json
   ```

2. **Verify JSON syntax is valid**

3. **Check absolute path is correct:**
   ```bash
   node /absolute/path/to/safe-coder/dist/index.js
   # Should not error
   ```

4. **Restart Cursor completely**

### "CLI Not Found" Error

```bash
# Check if CLI is in PATH
which safe-coder-cli

# If not found, reinstall
cd /path/to/safe-coder-cli
npm link

# Verify
safe-coder-cli --version
```

### Slow Crawling

- Reduce `maxPages` (e.g., 50 instead of 200)
- Decrease `depth` (e.g., 2 instead of 3)
- Increase `workers` (e.g., 3 for parallel crawling) - *requires CLI support*

### Cache Not Working

```bash
# Check cache directory
ls -la ~/.safe-coder/cache/

# Permissions issue?
chmod 755 ~/.safe-coder/cache/

# Clear and rebuild
rm -rf ~/.safe-coder/cache/
# Cache will recreate on next crawl
```

## Development

### Project Structure

```
safe-coder/
├── src/
│   ├── index.ts                     # Entry point
│   ├── server/
│   │   └── safe-coder-mcp.ts        # MCP server class
│   ├── executor/
│   │   ├── dependency-checker.ts    # Check CLI installation
│   │   └── cli-executor.ts          # Spawn CLI process
│   ├── processor/
│   │   ├── content-processor.ts     # Analyze content
│   │   └── guide-generator.ts       # Generate SKILL guidance
│   ├── cache/
│   │   └── cache-manager.ts         # Cache management
│   └── tools/
│       ├── crawl-documentation.ts   # Main MCP tool
│       ├── cache-tools.ts           # Cache MCP tools
│       └── index.ts                 # Tool exports
├── dist/                            # Compiled output
├── tsconfig.json
├── package.json
└── README.md
```

### Build and Test

```bash
# Install dependencies
npm install

# Build
npm run build

# Watch mode
npm run build -- --watch

# Test locally
node dist/index.js

# Test with MCP Inspector (if available)
npx @modelcontextprotocol/inspector node dist/index.js
```

### Running with Debug Output

```typescript
// In src/index.ts
console.error('[Debug] MCP Server starting...');
console.error('[Debug] Available tools:', server.listTools());
```

MCP uses stdio, so:
- `stdout` → MCP protocol messages (JSON-RPC)
- `stderr` → Debug/log output (visible in Cursor logs)

## Roadmap / Future Enhancements

### Planned Features

1. **Expose More CLI Parameters**
   - [ ] `strategy` (bfs/dfs)
   - [ ] `checkpoint`, `resume`
   - [ ] `rateLimit`, `maxRetries`
   - [ ] `includePaths`, `excludePaths`

2. **Direct File Creation Tool**
   - [ ] `save_skill` tool to create SKILL.md directly
   - [ ] Automatic filename generation
   - [ ] Default location management

3. **Enhanced Processing**
   - [ ] Semantic analysis of content
   - [ ] Automatic category detection
   - [ ] Code example quality scoring
   - [ ] Cross-reference detection

4. **Better Caching**
   - [ ] Configurable TTL per URL
   - [ ] Cache compression
   - [ ] Partial cache updates
   - [ ] Cache analytics

## Contributing

This package is part of the Safe Coder project. Contributions welcome!

### Development Workflow

1. Fork and clone both `safe-coder` and `safe-coder-cli`
2. Make changes
3. Build and test locally
4. Submit pull request

### Testing

```bash
# Unit tests (when implemented)
npm test

# Integration test
# 1. Build both packages
cd safe-coder-cli && npm run build && npm link
cd ../safe-coder && npm run build && npm link

# 2. Configure in Cursor
# 3. Test end-to-end
```

## License

MIT

## Related Projects

- **@gulibs/safe-coder-cli** - Standalone documentation crawler
- **Cursor** - AI-powered code editor
- **Model Context Protocol (MCP)** - Protocol for AI tool integration

## Support

Issues: https://github.com/gulibs/safe-coder/issues  
Discord: [Link TBD]

---

**Safe Coder** - Bringing documentation intelligence to your AI coding assistant.