Skill: iamadig/x-bookmark-archiver (version 1.0.0)

# X Bookmark Archiver Archive your X (Twitter) bookmarks into categorized markdown files with AI-generated summaries. ## Overview This skill fetches your X bookmarks using the [bird CLI](https://github.com/steipete/bird), categorizes them by URL patterns, generates AI summaries using OpenAI, and saves them as organized markdown files. ## Prerequisites 1. **bird CLI** - Install from [steipete/bird](https://github.com/steipete/bird) 2. **OpenAI API Key** (optional) - Set `OPENAI_API_KEY` for AI-generated summaries 3. **Node.js 18+** ## Installation ```bash # Ensure bird CLI is installed and authenticated bird --version # Set OpenAI API key (optional, for AI summaries) export OPENAI_API_KEY="sk-..." ``` ## Commands ### `run` - Full Pipeline Fetches new bookmarks and processes them: ```bash node skills/x-bookmark-archiver/scripts/run.cjs ``` ### `run --force` - Process Existing Skip fetch, process only pending bookmarks: ```bash node skills/x-bookmark-archiver/scripts/run.cjs --force ``` ### `fetch` - Download Bookmarks Only ```bash node skills/x-bookmark-archiver/scripts/fetch.cjs ``` ### `process` - Archive Pending Only ```bash node skills/x-bookmark-archiver/scripts/process.cjs ``` ## Category Mapping Bookmarks are automatically categorized based on URL patterns: | Category | Domains | |----------|---------| | **tools** | github.com, gitlab.com, github.io, huggingface.co, replicate.com, vercel.com, npmjs.com, pypi.org | | **articles** | medium.com, substack.com, dev.to, hashnode.dev, x.com/i/article, blog.*, towardsdatascience.com | | **videos** | youtube.com, youtu.be, vimeo.com, twitch.tv | | **research** | arxiv.org, paperswithcode.com, semanticscholar.org, researchgate.net, dl.acm.org, ieee.org | | **news** | techcrunch.com, theverge.com, hn.algolia.com, news.ycombinator.com, wired.com, arstechnica.com | | **bookmarks** | *fallback for unmatched URLs* | ## Output Location Markdown files are created in the **OpenClaw workspace** at: **Legacy installs (old name):** ``` ~/clawd/X-knowledge/ ``` **New installs:** ``` ~/.openclaw/workspace/X-knowledge/ ``` **With profile (`OPENCLAW_PROFILE=prod`):** ``` ~/.openclaw/workspace-prod/X-knowledge/ ``` **Override with environment variable:** ```bash export OPENCLAW_WORKSPACE=/custom/path node skills/x-bookmark-archiver/scripts/run.cjs # Creates: /custom/path/X-knowledge/ ``` ## Output Structure ``` ~/.openclaw/workspace/X-knowledge/ ├── tools/ │ ├── awesome-ai-project.md │ └── useful-cli-tool.md ├── articles/ │ ├── how-to-build-x.md │ └── ml-best-practices.md ├── videos/ │ └── conference-talk.md ├── research/ │ └── attention-is-all-you-need.md ├── news/ │ └── latest-tech-announcement.md └── bookmarks/ └── misc-link.md ``` ## Markdown Template Each archived bookmark gets a markdown file with frontmatter: ```markdown --- title: "Awesome AI Project" type: tool date_archived: 2026-01-31 source_tweet: https://x.com/i/web/status/1234567890 link: https://github.com/user/repo tags: ["ai", "machine-learning", "github"] --- This project implements a novel approach to... (AI-generated summary) ``` ## State Management State files track processing progress: ``` /root/clawd/.state/ ├── x-bookmark-pending.json # Bookmarks waiting to be processed └── x-bookmark-processed.json # IDs of already-archived bookmarks ``` ## Environment Variables | Variable | Required | Description | |----------|----------|-------------| | `OPENAI_API_KEY` | No | API key for AI-generated summaries | ## Workflow 1. **Fetch**: Downloads latest 50 bookmarks from X 2. **Filter**: Removes already-processed bookmarks 3. **Expand**: Resolves t.co shortened URLs 4. **Categorize**: Assigns category based on URL domain 5. **Enrich**: Generates title, summary, tags (AI or fallback) 6. **Write**: Saves as markdown in `X-knowledge/{category}/` 7. **Track**: Updates processed IDs, clears pending ## Customization ### Adding Categories Edit `scripts/lib/categorize.cjs`: ```javascript const CATEGORIES = { tools: ['github.com', '...'], your_category: ['example.com', '...'], // ... }; ``` ### Changing Output Directory Edit `scripts/process.cjs`: ```javascript const KNOWLEDGE_DIR = 'your-directory-name'; ``` ### Using Different AI Provider Modify the `generateMetadata()` function in `scripts/process.cjs` to use your preferred API. ## Testing Run the test suite: ```bash # Run all tests cd skills/x-bookmark-archiver/tests node test-all.cjs # Run individual test suites node lib/categorize.test.cjs node lib/state.test.cjs node integration.test.cjs ``` ### Test Coverage - **Unit tests**: `categorize.js` (21 tests) - URL pattern matching - **Unit tests**: `state.js` (9 tests) - JSON read/write operations - **Integration tests** (12 tests) - Full pipeline with mock data ### Manual Testing Without bird CLI, you can test with mock data: ```bash # Create mock pending data cat > /tmp/test-pending.json << 'EOF' [ { "id": "test123", "url": "https://github.com/test/repo", "text": "Test bookmark" } ] EOF # Copy to state directory and process mkdir -p /root/clawd/.state cp /tmp/test-pending.json /root/clawd/.state/x-bookmark-pending.json node skills/x-bookmark-archiver/scripts/process.cjs ``` ## Troubleshooting | Issue | Solution | |-------|----------| | `bird: command not found` | Install bird CLI from GitHub releases | | No bookmarks fetched | Ensure you're logged into X in bird | | AI summaries not generating | Check `OPENAI_API_KEY` is set | | t.co links not expanding | May be network/timeout issues; will use original URL | ## File Structure ``` skills/x-bookmark-archiver/ ├── SKILL.md ├── scripts/ │ ├── fetch.cjs # Download bookmarks from X (CommonJS) │ ├── process.cjs # Generate markdown files (CommonJS) │ ├── run.cjs # Orchestrate fetch → process (CommonJS) │ └── lib/ │ ├── categorize.cjs # URL → category mapping (CommonJS) │ └── state.cjs # JSON state management (CommonJS) └── tests/ ├── test-all.cjs ├── lib/ │ ├── categorize.test.cjs │ │ └── state.test.cjs ├── integration.test.cjs └── fixtures/ └── sample-bookmarks.json ``` ## License MIT

path: scripts/run.cjs
content: #!/usr/bin/env node
/**
 * Main runner script - orchestrates fetch → process pipeline
 */

const { execSync } = require('child_process');
const path = require('path');

const SCRIPTS_DIR = __dirname;

/**
 * Run a script and capture output
 * @param {string} script - Script name to run
 */
function runScript(script) {
  const scriptPath = path.join(SCRIPTS_DIR, script);
  try {
    execSync(`node "${scriptPath}"`, {
      stdio: 'inherit',
      cwd: process.cwd()
    });
  } catch (e) {
    // Script may exit with non-zero, that's ok
    if (e.status !== 0) {
      process.exit(e.status);
    }
  }
}

/**
 * Main function
 */
function main() {
  const args = process.argv.slice(2);
  const force = args.includes('--force');
  
  console.log('═══════════════════════════════════════');
  console.log('   X Bookmark Archiver');
  console.log('═══════════════════════════════════════\n');
  
  // Step 1: Fetch (always run unless --force)
  if (!force) {
    console.log('📥 STEP 1: Fetching bookmarks\n');
    runScript('fetch.cjs');
    console.log('');
  } else {
    console.log('⚡ Force mode: skipping fetch, processing existing pending\n');
  }
  
  // Step 2: Process
  console.log('📤 STEP 2: Processing bookmarks\n');
  runScript('process.cjs');
  
  console.log('\n═══════════════════════════════════════');
  console.log('   Done!');
  console.log('═══════════════════════════════════════');
}

if (require.main === module) {
  main();
}


path: scripts/fetch.cjs
content: #!/usr/bin/env node
/**
 * Fetch bookmarks from X using bird CLI
 * Filters out already-processed bookmarks
 */

const { execSync } = require('child_process');
const { loadProcessed, savePending } = require('./lib/state.cjs');

const FETCH_COUNT = 50;

/**
 * Fetch bookmarks from X using bird CLI
 * @returns {Array} - Array of bookmark objects
 */
function fetchBookmarks() {
  try {
    const output = execSync(`bird bookmarks -n ${FETCH_COUNT} --json`, {
      encoding: 'utf8',
      timeout: 30000
    });
    return JSON.parse(output);
  } catch (e) {
    if (e.stderr?.includes('command not found') || e.message.includes('ENOENT')) {
      console.error('Error: bird CLI not found. Install from https://github.com/smmr-software/bird');
      process.exit(1);
    }
    console.error('Error fetching bookmarks:', e.message);
    return [];
  }
}

/**
 * Main fetch function
 */
function main() {
  console.log('🔍 Fetching bookmarks from X...');
  
  const bookmarks = fetchBookmarks();
  if (bookmarks.length === 0) {
    console.log('No bookmarks found.');
    return;
  }
  
  const processed = loadProcessed();
  
  // Filter out already processed
  const newBookmarks = bookmarks.filter(b => !processed.has(b.id));
  
  if (newBookmarks.length === 0) {
    console.log('✓ No new bookmarks to process');
    return;
  }
  
  // Save to pending
  savePending(newBookmarks);
  
  console.log(`\n✓ Found ${newBookmarks.length} new bookmark${newBookmarks.length === 1 ? '' : 's'}`);
  console.log(`  Total fetched: ${bookmarks.length}`);
  console.log(`  Already processed: ${bookmarks.length - newBookmarks.length}`);
}

if (require.main === module) {
  main();
}

module.exports = { fetchBookmarks };


path: scripts/process.cjs
content: #!/usr/bin/env node
/**
 * Process pending bookmarks into markdown files
 * - Expands t.co links
 * - Categorizes URLs
 * - Generates AI summary
 * - Writes to X-knowledge/{category}/{slug}.md
 */

const fs = require('fs');
const path = require('path');
const https = require('https');
const os = require('os');
const { loadPending, clearPending, markProcessed } = require('./lib/state.cjs');
const { categorize } = require('./lib/categorize.cjs');

// Resolve OpenClaw workspace directory
// Priority: env var > ~/clawd (legacy) > ~/.openclaw/workspace (new default)
function resolveOpenClawWorkspace() {
  // Explicit override
  if (process.env.OPENCLAW_WORKSPACE) {
    return process.env.OPENCLAW_WORKSPACE;
  }
  
  const home = os.homedir();
  
  // Legacy: ~/clawd (old name, no dot)
  const legacyWorkspace = path.join(home, 'clawd');
  if (fs.existsSync(legacyWorkspace)) {
    return legacyWorkspace;
  }
  
  // New default: ~/.openclaw/workspace (or workspace-<profile>)
  const profile = process.env.OPENCLAW_PROFILE?.trim();
  const workspaceName = profile && profile.toLowerCase() !== 'default' 
    ? `workspace-${profile}` 
    : 'workspace';
  return path.join(home, '.openclaw', workspaceName);
}

const WORKSPACE_DIR = resolveOpenClawWorkspace();
const KNOWLEDGE_DIR = path.join(WORKSPACE_DIR, 'X-knowledge');

/**
 * Expand a t.co shortened URL
 * @param {string} url - The URL to expand
 * @returns {Promise<string>} - The expanded URL
 */
function expandUrl(url) {
  return new Promise((resolve) => {
    if (!url.includes('t.co/')) {
      resolve(url);
      return;
    }
    
    const req = https.get(url, { method: 'HEAD' }, (res) => {
      resolve(res.headers.location || url);
    });
    
    req.on('error', () => resolve(url));
    req.setTimeout(10000, () => {
      req.destroy();
      resolve(url);
    });
  });
}

/**
 * Generate slug from URL or title
 * @param {string} url - The URL
 * @param {string} title - The bookmark title
 * @returns {string} - Safe filename slug
 */
function generateSlug(url, title) {
  if (title) {
    return title
      .toLowerCase()
      .replace(/[^a-z0-9]+/g, '-')
      .replace(/^-|-$/g, '')
      .slice(0, 50);
  }
  
  try {
    const urlObj = new URL(url);
    const pathSlug = urlObj.pathname
      .replace(/[^a-z0-9]+/g, '-')
      .replace(/^-|-$/g, '')
      .slice(0, 40);
    return pathSlug || urlObj.hostname.replace(/\./g, '-');
  } catch {
    return 'bookmark-' + Date.now();
  }
}

/**
 * Generate title, summary, and tags using AI (OpenAI-compatible API)
 * @param {string} url - The URL to analyze
 * @param {string} originalText - Original tweet text
 * @returns {Promise<Object>} - { title, summary, tags }
 */
async function generateMetadata(url, originalText) {
  // Check for OpenAI API key
  const [REDACTED_SECRET];
  
  // Extract description from text (remove URL if it's the whole text)
  const description = originalText?.startsWith('http') 
    ? 'Bookmark from X'
    : (originalText || 'Bookmark from X');
  
  if (!apiKey) {
    // Fallback: use URL hostname as title
    try {
      const urlObj = new URL(url);
      return {
        title: urlObj.hostname,
        summary: description,
        tags: []
      };
    } catch {
      return {
        title: 'Untitled Bookmark',
        summary: description,
        tags: []
      };
    }
  }
  
  const prompt = `Analyze this URL and the tweet that shared it. Generate a concise title (max 60 chars), a 2-3 sentence summary, and 3-5 relevant tags.

URL: ${url}
Tweet: ${originalText || 'N/A'}

Respond in JSON format:
{
  "title": "...",
  "summary": "...",
  "tags": ["tag1", "tag2", "tag3"]
}`;

  try {
    const result = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${apiKey}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({
        model: 'gpt-4o-mini',
        messages: [{ role: 'user', content: prompt }],
        response_format: { type: 'json_object' }
      })
    });
    
    const data = await result.json();
    const content = JSON.parse(data.choices[0].message.content);
    
    return {
      title: content.title || 'Untitled',
      summary: content.summary || '',
      tags: content.tags || []
    };
  } catch (e) {
    console.warn('AI generation failed:', e.message);
    return {
      title: 'Untitled Bookmark',
      summary: originalText || `Bookmarked link: ${url}`,
      tags: []
    };
  }
}

/**
 * Write markdown file for a bookmark
 * @param {Object} bookmark - The bookmark object
 * @param {string} category - The category
 * @param {Object} metadata - { title, summary, tags }
 */
function writeMarkdown(bookmark, category, metadata) {
  const categoryDir = path.join(KNOWLEDGE_DIR, category);
  fs.mkdirSync(categoryDir, { recursive: true });
  
  const slug = generateSlug(bookmark.url, metadata.title);
  const filename = path.join(categoryDir, `${slug}.md`);
  
  const dateStr = new Date().toISOString().split('T')[0];
  
  const frontmatter = `---
title: "${metadata.title.replace(/"/g, '\\"')}"
type: ${category}
date_archived: ${dateStr}
source_tweet: https://x.com/i/web/status/${bookmark.id}
link: ${bookmark.url}
tags: [${metadata.tags.map(t => `"${t}"`).join(', ')}]
---

${metadata.summary}
`;
  
  // Handle duplicates by adding number suffix
  let finalFilename = filename;
  let counter = 1;
  while (fs.existsSync(finalFilename)) {
    const ext = path.extname(filename);
    const base = filename.slice(0, -ext.length);
    finalFilename = `${base}-${counter}${ext}`;
    counter++;
  }
  
  fs.writeFileSync(finalFilename, frontmatter);
  return finalFilename;
}

/**
 * Extract URL from bookmark (bird CLI uses 'text' field for URL)
 * @param {Object} bookmark - The bookmark object
 * @returns {string} - The URL
 */
function extractUrl(bookmark) {
  // bird CLI returns URL in 'text' field
  if (bookmark.url) return bookmark.url;
  if (bookmark.text && bookmark.text.startsWith('http')) return bookmark.text;
  // Try to extract URL from text
  const urlMatch = bookmark.text?.match(/(https?:\/\/[^\s]+)/);
  return urlMatch ? urlMatch[1] : '';
}

/**
 * Main processing function
 */
async function main() {
  console.log('📦 Processing bookmarks...\n');
  
  const pending = loadPending();
  if (pending.length === 0) {
    console.log('No pending bookmarks to process.');
    return;
  }
  
  const stats = {
    total: pending.length,
    processed: 0,
    byCategory: {}
  };
  
  const processedIds = [];
  
  for (const bookmark of pending) {
    try {
      // Extract URL from bookmark (bird uses 'text' field)
      const rawUrl = extractUrl(bookmark);
      if (!rawUrl) {
        console.error(`  ✗ No URL found in bookmark ${bookmark.id}`);
        continue;
      }
      
      // Expand t.co links
      const expandedUrl = await expandUrl(rawUrl);
      bookmark.url = expandedUrl;
      
      // Categorize
      const category = categorize(expandedUrl);
      stats.byCategory[category] = (stats.byCategory[category] || 0) + 1;
      
      // Generate metadata with AI
      const metadata = await generateMetadata(expandedUrl, bookmark.text);
      
      // Write markdown
      const filepath = writeMarkdown(bookmark, category, metadata);
      console.log(`  ✓ [${category}] ${metadata.title}`);
      console.log(`    → ${filepath}`);
      
      processedIds.push(bookmark.id);
      stats.processed++;
    } catch (e) {
      console.error(`  ✗ Failed to process bookmark ${bookmark.id}:`, e.message);
    }
  }
  
  // Update state
  markProcessed(processedIds);
  clearPending();
  
  // Output summary
  console.log(`\n✓ Archived ${stats.processed}/${stats.total} bookmarks`);
  
  const categoryBreakdown = Object.entries(stats.byCategory)
    .map(([cat, count]) => `${count} ${cat}`)
    .join(', ');
  
  if (categoryBreakdown) {
    console.log(`  → ${categoryBreakdown}`);
  }
}

if (require.main === module) {
  main().catch(console.error);
}

module.exports = { processBookmarks: main, expandUrl, generateMetadata, writeMarkdown };


path: scripts/lib/state.cjs
content: /**
 * State management for bookmark processing
 * Tracks pending and processed bookmarks
 */

const fs = require('fs');
const path = require('path');

const STATE_DIR = process.env.X_BOOKMARK_STATE_DIR || (require('os').homedir() + '/.clawd/.state');
const PENDING = path.join(STATE_DIR, 'x-bookmark-pending.json');
const PROCESSED = path.join(STATE_DIR, 'x-bookmark-processed.json');

/**
 * Ensure state directory exists
 */
function ensureStateDir() {
  if (!fs.existsSync(STATE_DIR)) {
    fs.mkdirSync(STATE_DIR, { recursive: true });
  }
}

/**
 * Load pending bookmarks
 * @returns {Array} - Array of pending bookmark objects
 */
function loadPending() {
  ensureStateDir();
  if (!fs.existsSync(PENDING)) return [];
  try {
    const data = fs.readFileSync(PENDING, 'utf8');
    return JSON.parse(data);
  } catch (e) {
    console.error('Error loading pending:', e.message);
    return [];
  }
}

/**
 * Save pending bookmarks
 * @param {Array} data - Array of bookmark objects
 */
function savePending(data) {
  ensureStateDir();
  fs.writeFileSync(PENDING, JSON.stringify(data, null, 2));
}

/**
 * Load processed bookmark IDs
 * @returns {Set} - Set of processed bookmark IDs
 */
function loadProcessed() {
  ensureStateDir();
  if (!fs.existsSync(PROCESSED)) return new Set();
  try {
    const data = fs.readFileSync(PROCESSED, 'utf8');
    const arr = JSON.parse(data);
    return new Set(arr);
  } catch (e) {
    console.error('Error loading processed:', e.message);
    return new Set();
  }
}

/**
 * Save processed bookmark IDs
 * @param {Set} set - Set of processed IDs
 */
function saveProcessed(set) {
  ensureStateDir();
  fs.writeFileSync(PROCESSED, JSON.stringify([...set], null, 2));
}

/**
 * Add IDs to processed list
 * @param {string[]} ids - Array of bookmark IDs to add
 */
function markProcessed(ids) {
  const processed = loadProcessed();
  ids.forEach(id => processed.add(id));
  saveProcessed(processed);
}

/**
 * Clear pending list
 */
function clearPending() {
  savePending([]);
}

module.exports = {
  loadPending,
  savePending,
  loadProcessed,
  saveProcessed,
  markProcessed,
  clearPending,
  PENDING,
  PROCESSED
};


path: scripts/lib/categorize.cjs
content: /**
 * URL categorization based on domain patterns
 */

const CATEGORIES = {
  tools: ['github.com', 'gitlab.com', 'github.io', 'huggingface.co', 'replicate.com', 'vercel.com', 'vercel.app', 'npmjs.com', 'pypi.org', 'raw.githubusercontent.com'],
  articles: ['medium.com', 'substack.com', 'dev.to', 'hashnode.dev', 'x.com/i/article', 'blog.', 'towardsdatascience.com'],
  videos: ['youtube.com', 'youtu.be', 'vimeo.com', 'twitch.tv'],
  research: ['arxiv.org', 'paperswithcode.com', 'semanticscholar.org', 'researchgate.net', 'dl.acm.org', 'ieee.org'],
  news: ['techcrunch.com', 'theverge.com', 'hn.algolia.com', 'news.ycombinator.com', 'wired.com', 'arstechnica.com']
};

/**
 * Categorize a URL based on its domain
 * @param {string} url - The URL to categorize
 * @returns {string} - Category name or 'bookmarks' as fallback
 */
function categorize(url) {
  const lowerUrl = url.toLowerCase();
  for (const [cat, domains] of Object.entries(CATEGORIES)) {
    if (domains.some(d => lowerUrl.includes(d))) return cat;
  }
  return 'bookmarks'; // fallback
}

/**
 * Get all available categories
 * @returns {string[]} - List of category names
 */
function getCategories() {
  return [...Object.keys(CATEGORIES), 'bookmarks'];
}

module.exports = { categorize, getCategories, CATEGORIES };


path: tests/integration.test.cjs
content: #!/usr/bin/env node
/**
 * Integration test for full pipeline
 * Uses mock data instead of real bird CLI
 */

const fs = require('fs');
const path = require('path');
const os = require('os');

// Setup test environment
const TEST_DIR = path.join(os.tmpdir(), 'x-bookmark-integration-' + Date.now());
const FIXTURES_DIR = path.join(__dirname, 'fixtures');

console.log('\n🔬 Integration Test');
console.log(`   Test dir: ${TEST_DIR}\n`);

// Create test directories
fs.mkdirSync(TEST_DIR, { recursive: true });
fs.mkdirSync(path.join(TEST_DIR, 'scripts', 'lib'), { recursive: true });

// Copy scripts to test dir for isolation
const scriptsToCopy = ['lib/categorize.cjs', 'lib/state.cjs', 'process.cjs'];
for (const script of scriptsToCopy) {
  const src = path.join(__dirname, '..', 'scripts', script);
  const dest = path.join(TEST_DIR, 'scripts', script);
  fs.copyFileSync(src, dest);
}

// Mock state module paths
const stateCode = fs.readFileSync(path.join(TEST_DIR, 'scripts', 'lib', 'state.cjs'), 'utf8');
const mockedState = stateCode.replace(
  "const STATE_DIR = process.env.X_BOOKMARK_STATE_DIR || (require('os').homedir() + '/.clawd/.state');",
  `const STATE_DIR = '${TEST_DIR}/.state';`
);
fs.writeFileSync(path.join(TEST_DIR, 'scripts', 'lib', 'state.cjs'), mockedState);

// Mock process.js output path
const processCode = fs.readFileSync(path.join(TEST_DIR, 'scripts', 'process.cjs'), 'utf8');
const mockedProcess = processCode.replace(
  "const KNOWLEDGE_DIR = 'X-knowledge';",
  `const KNOWLEDGE_DIR = '${TEST_DIR}/X-knowledge';`
);
fs.writeFileSync(path.join(TEST_DIR, 'scripts', 'process.cjs'), mockedProcess);

// Load modules from test dir
const { categorize } = require(path.join(TEST_DIR, 'scripts', 'lib', 'categorize.cjs'));
const state = require(path.join(TEST_DIR, 'scripts', 'lib', 'state.cjs'));

// Load sample bookmarks
const sampleBookmarks = JSON.parse(fs.readFileSync(path.join(FIXTURES_DIR, 'sample-bookmarks.json'), 'utf8'));

function test(name, fn) {
  try {
    fn();
    console.log(`✓ ${name}`);
  } catch (e) {
    console.error(`✗ ${name}`);
    console.error(`  ${e.message}`);
    process.exitCode = 1;
  }
}

// Test 1: Load and categorize bookmarks
console.log('📋 Test: Load and categorize sample bookmarks');
test('Sample data has 5 bookmarks', () => {
  if (sampleBookmarks.length !== 5) throw new Error(`Expected 5, got ${sampleBookmarks.length}`);
});

test('GitHub bookmark categorized as tools', () => {
  const gh = sampleBookmarks.find(b => b.url.includes('github.com'));
  if (!gh) throw new Error('GitHub bookmark not found');
  const cat = categorize(gh.url);
  if (cat !== 'tools') throw new Error(`Expected tools, got ${cat}`);
});

test('Medium bookmark categorized as articles', () => {
  const med = sampleBookmarks.find(b => b.url.includes('medium.com'));
  if (!med) throw new Error('Medium bookmark not found');
  const cat = categorize(med.url);
  if (cat !== 'articles') throw new Error(`Expected articles, got ${cat}`);
});

test('arXiv bookmark categorized as research', () => {
  const arx = sampleBookmarks.find(b => b.url.includes('arxiv.org'));
  if (!arx) throw new Error('arXiv bookmark not found');
  const cat = categorize(arx.url);
  if (cat !== 'research') throw new Error(`Expected research, got ${cat}`);
});

test('YouTube bookmark categorized as videos', () => {
  const yt = sampleBookmarks.find(b => b.url.includes('youtube.com'));
  if (!yt) throw new Error('YouTube bookmark not found');
  const cat = categorize(yt.url);
  if (cat !== 'videos') throw new Error(`Expected videos, got ${cat}`);
});

// Test 2: State operations
console.log('\n💾 Test: State save/load cycle');
test('Save pending bookmarks', () => {
  state.savePending(sampleBookmarks);
  const loaded = state.loadPending();
  if (loaded.length !== 5) throw new Error(`Expected 5, got ${loaded.length}`);
});

test('Mark some as processed', () => {
  state.markProcessed(['1234567890', '1234567891']);
  const processed = state.loadProcessed();
  if (!processed.has('1234567890')) throw new Error('ID 1234567890 not in processed');
  if (!processed.has('1234567891')) throw new Error('ID 1234567891 not in processed');
  if (processed.size !== 2) throw new Error(`Expected 2 processed, got ${processed.size}`);
});

test('Clear pending', () => {
  state.clearPending();
  const pending = state.loadPending();
  if (pending.length !== 0) throw new Error(`Expected empty, got ${pending.length}`);
});

// Test 3: Markdown generation (without AI)
console.log('\n📝 Test: Markdown generation');
test('Generate slug from title', () => {
  // Using the process.js logic
  function generateSlug(url, title) {
    if (title) {
      return title
        .toLowerCase()
        .replace(/[^a-z0-9]+/g, '-')
        .replace(/^-|-$/g, '')
        .slice(0, 50);
    }
    try {
      const urlObj = new URL(url);
      return urlObj.hostname.replace(/\./g, '-');
    } catch {
      return 'bookmark-' + Date.now();
    }
  }
  
  const slug = generateSlug('https://example.com', 'Hello World Test');
  if (slug !== 'hello-world-test') throw new Error(`Got: ${slug}`);
});

test('Generate slug from URL when no title', () => {
  function generateSlug(url) {
    try {
      const urlObj = new URL(url);
      return urlObj.hostname.replace(/\./g, '-');
    } catch {
      return 'bookmark';
    }
  }
  
  const slug = generateSlug('https://github.com/user/repo');
  if (slug !== 'github-com') throw new Error(`Got: ${slug}`);
});

// Test 4: Directory structure
console.log('\n📁 Test: Output directory structure');
const knowledgeDir = path.join(TEST_DIR, 'X-knowledge');
fs.mkdirSync(path.join(knowledgeDir, 'tools'), { recursive: true });
fs.mkdirSync(path.join(knowledgeDir, 'articles'), { recursive: true });

test('Create category directories', () => {
  if (!fs.existsSync(path.join(knowledgeDir, 'tools'))) {
    throw new Error('tools dir not created');
  }
  if (!fs.existsSync(path.join(knowledgeDir, 'articles'))) {
    throw new Error('articles dir not created');
  }
});

test('Write sample markdown file', () => {
  const mdContent = `---
title: "Test Article"
type: articles
date_archived: 2026-01-31
source_tweet: https://x.com/i/web/status/1234567892
link: https://medium.com/@author/test
tags: ["test", "article"]
---

This is a test summary.
`;
  const filepath = path.join(knowledgeDir, 'articles', 'test-article.md');
  fs.writeFileSync(filepath, mdContent);
  
  if (!fs.existsSync(filepath)) throw new Error('File not created');
  const content = fs.readFileSync(filepath, 'utf8');
  if (!content.includes('title:')) throw new Error('Frontmatter missing');
});

// Cleanup
console.log('\n🧹 Cleaning up...');
fs.rmSync(TEST_DIR, { recursive: true, force: true });

console.log('\n' + (process.exitCode ? '❌ Integration tests failed' : '✅ Integration tests passed'));


path: tests/test-all.cjs
content: #!/usr/bin/env node
/**
 * Test runner - runs all tests
 */

const { execSync } = require('child_process');
const path = require('path');

const tests = [
  { name: 'Categorize Unit Tests', file: 'lib/categorize.test.cjs' },
  { name: 'State Unit Tests', file: 'lib/state.test.cjs' },
  { name: 'Integration Tests', file: 'integration.test.cjs' }
];

console.log('═══════════════════════════════════════════');
console.log('   X-Bookmark-Archiver Test Suite');
console.log('═══════════════════════════════════════════\n');

let failed = 0;

for (const test of tests) {
  console.log(`\n▶ Running: ${test.name}`);
  console.log('─'.repeat(40));
  
  try {
    execSync(`node "${path.join(__dirname, test.file)}"`, {
      stdio: 'inherit',
      cwd: __dirname
    });
  } catch (e) {
    failed++;
  }
}

console.log('\n═══════════════════════════════════════════');
if (failed === 0) {
  console.log('   ✅ All test suites passed!');
} else {
  console.log(`   ❌ ${failed} test suite(s) failed`);
}
console.log('═══════════════════════════════════════════\n');

process.exit(failed > 0 ? 1 : 0);


path: tests/fixtures/sample-bookmarks.json
content: id: 1234567890
url: https://t.co/abc123
text: Check out this awesome AI tool!
created_at: 2026-01-30T12:00:00Z

id: 1234567891
url: https://github.com/user/awesome-project
text: Great open source project for ML pipelines
created_at: 2026-01-29T15:30:00Z

id: 1234567892
url: https://medium.com/@author/building-scalable-systems
text: Excellent article on system design
created_at: 2026-01-28T09:15:00Z

id: 1234567893
url: https://arxiv.org/abs/2401.12345
text: New paper on transformer architectures
created_at: 2026-01-27T18:45:00Z

id: 1234567894
url: https://youtube.com/watch?v=dQw4w9WgXcQ
text: Must-watch tutorial on React patterns
created_at: 2026-01-26T11:20:00Z

path: tests/lib/categorize.test.cjs
content: /**
 * Tests for categorize.js
 */

const { categorize, getCategories, CATEGORIES } = require('../../scripts/lib/categorize.cjs');

function test(name, fn) {
  try {
    fn();
    console.log(`✓ ${name}`);
  } catch (e) {
    console.error(`✗ ${name}`);
    console.error(`  ${e.message}`);
    process.exitCode = 1;
  }
}

function assertEqual(actual, expected, msg) {
  if (actual !== expected) {
    throw new Error(`${msg || 'Assertion failed'}: expected "${expected}", got "${actual}"`);
  }
}

// Test: Tools categorization
console.log('\n📁 Tools Category Tests');
test('GitHub repo URL', () => {
  assertEqual(categorize('https://github.com/user/repo'), 'tools');
});
test('GitHub raw URL', () => {
  assertEqual(categorize('https://raw.githubusercontent.com/...'), 'tools');
});
test('HuggingFace', () => {
  assertEqual(categorize('https://huggingface.co/spaces/demo'), 'tools');
});
test('Vercel deployment', () => {
  assertEqual(categorize('https://myapp.vercel.app'), 'tools');
});

// Test: Articles categorization
console.log('\n📄 Articles Category Tests');
test('Medium article', () => {
  assertEqual(categorize('https://medium.com/@user/article'), 'articles');
});
test('Substack newsletter', () => {
  assertEqual(categorize('https://newsletter.substack.com/p/post'), 'articles');
});
test('Dev.to post', () => {
  assertEqual(categorize('https://dev.to/user/post-title'), 'articles');
});
test('X article', () => {
  assertEqual(categorize('https://x.com/i/article/12345'), 'articles');
});
test('Blog subdomain', () => {
  assertEqual(categorize('https://blog.example.com/post'), 'articles');
});

// Test: Videos categorization
console.log('\n🎬 Videos Category Tests');
test('YouTube video', () => {
  assertEqual(categorize('https://youtube.com/watch?v=123'), 'videos');
});
test('YouTube short URL', () => {
  assertEqual(categorize('https://youtu.be/abc123'), 'videos');
});
test('Vimeo', () => {
  assertEqual(categorize('https://vimeo.com/123456'), 'videos');
});

// Test: Research categorization
console.log('\n🔬 Research Category Tests');
test('arXiv paper', () => {
  assertEqual(categorize('https://arxiv.org/abs/1234.5678'), 'research');
});
test('Papers With Code', () => {
  assertEqual(categorize('https://paperswithcode.com/paper/abc'), 'research');
});

// Test: News categorization
console.log('\n📰 News Category Tests');
test('TechCrunch', () => {
  assertEqual(categorize('https://techcrunch.com/2024/01/01/article'), 'news');
});
test('Hacker News', () => {
  assertEqual(categorize('https://news.ycombinator.com/item?id=123'), 'news');
});

// Test: Fallback
console.log('\n📌 Fallback Tests');
test('Unknown domain falls back to bookmarks', () => {
  assertEqual(categorize('https://random-unknown-site.com/page'), 'bookmarks');
});
test('Empty string falls back', () => {
  assertEqual(categorize(''), 'bookmarks');
});

// Test: Case insensitivity
console.log('\n🔤 Case Insensitivity Tests');
test('Uppercase GitHub', () => {
  assertEqual(categorize('https://GITHUB.COM/user/repo'), 'tools');
});
test('Mixed case YouTube', () => {
  assertEqual(categorize('https://YouTube.Com/watch'), 'videos');
});

// Test: getCategories
console.log('\n📋 Category List Test');
test('getCategories returns all categories', () => {
  const cats = getCategories();
  if (!cats.includes('tools')) throw new Error('Missing tools');
  if (!cats.includes('articles')) throw new Error('Missing articles');
  if (!cats.includes('bookmarks')) throw new Error('Missing bookmarks');
});

console.log('\n' + (process.exitCode ? '❌ Some tests failed' : '✅ All tests passed'));


path: tests/lib/state.test.cjs
content: /**
 * Tests for state.js
 */

const fs = require('fs');
const path = require('path');
const os = require('os');

// Create temp state dir for testing
const TEST_STATE_DIR = path.join(os.tmpdir(), 'x-bookmark-test-' + Date.now());

// Create wrapper that uses test directory
const state = {
  _ensureDir: () => {
    if (!fs.existsSync(TEST_STATE_DIR)) {
      fs.mkdirSync(TEST_STATE_DIR, { recursive: true });
    }
  },
  loadPending: function() {
    this._ensureDir();
    const file = path.join(TEST_STATE_DIR, 'pending.json');
    if (!fs.existsSync(file)) return [];
    return JSON.parse(fs.readFileSync(file, 'utf8'));
  },
  savePending: function(data) {
    this._ensureDir();
    const file = path.join(TEST_STATE_DIR, 'pending.json');
    fs.writeFileSync(file, JSON.stringify(data, null, 2));
  },
  loadProcessed: function() {
    this._ensureDir();
    const file = path.join(TEST_STATE_DIR, 'processed.json');
    if (!fs.existsSync(file)) return new Set();
    return new Set(JSON.parse(fs.readFileSync(file, 'utf8')));
  },
  saveProcessed: function(set) {
    this._ensureDir();
    const file = path.join(TEST_STATE_DIR, 'processed.json');
    fs.writeFileSync(file, JSON.stringify([...set], null, 2));
  },
  markProcessed: function(ids) {
    const processed = this.loadProcessed();
    ids.forEach(id => processed.add(id));
    this.saveProcessed(processed);
  },
  clearPending: function() {
    this.savePending([]);
  }
};

function test(name, fn) {
  try {
    fn();
    console.log(`✓ ${name}`);
  } catch (e) {
    console.error(`✗ ${name}`);
    console.error(`  ${e.message}`);
    process.exitCode = 1;
  }
}

function assertEqual(actual, expected, msg) {
  if (JSON.stringify(actual) !== JSON.stringify(expected)) {
    throw new Error(`${msg || 'Assertion failed'}: expected ${JSON.stringify(expected)}, got ${JSON.stringify(actual)}`);
  }
}

// Setup: Clean test dir before tests
if (fs.existsSync(TEST_STATE_DIR)) {
  fs.rmSync(TEST_STATE_DIR, { recursive: true });
}

console.log('\n📦 State Management Tests');
console.log(`   Using: ${TEST_STATE_DIR}\n`);

// Test: Empty state
test('loadPending returns empty array when no file', () => {
  assertEqual(state.loadPending(), []);
});

test('loadProcessed returns empty set when no file', () => {
  assertEqual([...state.loadProcessed()], []);
});

// Test: Save and load pending
test('savePending writes JSON file', () => {
  const data = [{ id: '123', url: 'https://example.com' }];
  state.savePending(data);
  assertEqual(state.loadPending(), data);
});

test('savePending preserves array of multiple items', () => {
  const data = [
    { id: '1', url: 'https://a.com' },
    { id: '2', url: 'https://b.com' },
    { id: '3', url: 'https://c.com' }
  ];
  state.savePending(data);
  assertEqual(state.loadPending().length, 3);
  assertEqual(state.loadPending()[1].id, '2');
});

// Test: Save and load processed
test('saveProcessed writes set as array', () => {
  const ids = new Set(['abc', 'def', 'ghi']);
  state.saveProcessed(ids);
  const loaded = state.loadProcessed();
  assertEqual(loaded.has('abc'), true);
  assertEqual(loaded.has('def'), true);
  assertEqual(loaded.has('xyz'), false);
});

// Test: markProcessed
test('markProcessed adds IDs to processed set', () => {
  state.clearPending();
  state.saveProcessed(new Set());
  state.markProcessed(['id1', 'id2']);
  const processed = state.loadProcessed();
  assertEqual(processed.has('id1'), true);
  assertEqual(processed.has('id2'), true);
  assertEqual(processed.size, 2);
});

test('markProcessed appends to existing set', () => {
  state.saveProcessed(new Set(['existing']));
  state.markProcessed(['new1', 'new2']);
  const processed = state.loadProcessed();
  assertEqual(processed.has('existing'), true);
  assertEqual(processed.has('new1'), true);
  assertEqual(processed.has('new2'), true);
  assertEqual(processed.size, 3);
});

// Test: clearPending
test('clearPending empties pending array', () => {
  state.savePending([{ id: '1' }, { id: '2' }]);
  state.clearPending();
  assertEqual(state.loadPending(), []);
});

// Test: File format
test('JSON files are pretty-printed', () => {
  state.savePending([{ id: '1', url: 'test' }]);
  const file = path.join(TEST_STATE_DIR, 'pending.json');
  const content = fs.readFileSync(file, 'utf8');
  if (!content.includes('\n')) {
    throw new Error('JSON not pretty-printed');
  }
});

// Cleanup
console.log('\n🧹 Cleaning up test directory...');
fs.rmSync(TEST_STATE_DIR, { recursive: true, force: true });

console.log('\n' + (process.exitCode ? '❌ Some tests failed' : '✅ All tests passed'));


Security scan verdicts — ClawScan: clean | static analysis: suspicious | VirusTotal: clean | SkillSpector: suspicious (HIGH)