Skill: rodionkertling/tg-news-digest-lite (version 1.0.0)

--- id: tg-news-digest-lite name: "Telegram News Digest (Lite)" version: "1.0.0" description: | Monitors public Telegram channels via web scraping (t.me/s/*), extracts new messages, generates AI-powered summaries, and delivers structured digests to your configured OpenClaw notification channel. Zero authentication required — just provide channel names. author: "your-github-handle" license: "MIT" tags: ["telegram", "news-monitoring", "web-scraping", "ai-summarization", "digest", "russian", "english"] category: "information-retrieval" minOpenClawVersion: "2026.4.0" runtime: "node20" permissions: network: ["t.me", "*.t.me", "api.openai.com", "*.openrouter.ai", "*.deepinfra.com"] filesystem: ["./state", "./config.yaml"] env: ["OPENCLAW_GATEWAY_URL", "OPENCLAW_GATEWAY_TOKEN"] --- # Telegram News Digest (Lite) > 🤖 **Autonomous AI agent for monitoring public Telegram channels** > ✅ No Telegram API keys • ✅ No authentication • ✅ Zero-config startup • ✅ Works with any LLM provider ## 🎯 What Problem Does This Solve? ### Real-world scenario You're a product manager tracking competitor announcements, a journalist monitoring breaking news, or a crypto trader watching market signals. Manually checking 10+ Telegram channels every hour is unsustainable. This skill automates the entire workflow: ``` [10 Telegram channels] ↓ [Fetch new messages every 30 min] ↓ [Filter duplicates + detect novelty] ↓ [Generate AI summary: "Key facts in 3 bullets"] ↓ [Send digest to your Telegram/Discord/Email] ``` ### Before vs After | Task | Manual Approach | With tg-news-digest-lite | |------|----------------|-------------------------| | Check 10 channels | ~15 min/hour | 0 min (fully automated) | | Detect new posts | Scroll + mental dedup | SHA256 hash comparison | | Summarize content | Read + mentally condense | LLM generates structured bullets | | Notify yourself | Copy-paste to chat | Auto-deliver to configured channel | | Missed updates | High risk (human error) | Guaranteed (persistent state) | ## 🏗 How It Works: Architecture Deep Dive ### Data Flow Diagram ```mermaid graph LR A[OpenClaw Cron/Heartbeat] --> B[run_digest_cycle] B --> C[Load config.yaml] B --> D[Initialize TelegramWebScraper] B --> E[Initialize StateManager] B --> F[Initialize Summarizer] C --> G[Channel list: durov, tjournal, roem...] D --> H[Fetch https://t.me/s/{channel}] H --> I[Parse HTML with Cheerio] I --> J[Extract message text + timestamp] J --> K[Generate SHA256 hash per message] E --> L[Load seen_hashes from state/seen_messages.json] K --> M[Filter: hash NOT IN seen_hashes] M --> N{New messages?} N -->|Yes| O[Send to Summarizer] N -->|No| P[Skip channel] O --> Q[LLM prompt: "Summarize these messages"] Q --> R[Parse JSON response: {summary, tags, sentiment}] R --> S[Format digest with markdown] S --> T[Send via ctx.send to notify_channel] K --> U[Update seen_hashes cache] U --> V[Save to state/seen_messages.json] ``` ### Component Responsibilities | Component | Responsibility | Key Design Decisions | |-----------|---------------|---------------------| | `TelegramWebScraper` | Fetch + parse t.me/s/* pages | • Uses `axios` with custom User-Agent to avoid bot detection<br>• Resilient HTML parsing: fallback selectors if Telegram changes structure<br>• Rate limiting: configurable delay between requests (default: 2000ms) | | `StateManager` | Track which messages were already processed | • SHA256 hash of normalized text (case-insensitive, whitespace-trimmed)<br>• Rolling cache: keeps last 100 hashes per channel to bound memory<br>• Atomic JSON writes to prevent corruption on crash | | `Summarizer` | Generate concise summaries via LLM | • OpenAI-compatible API (works with OpenRouter, DeepInfra, local Ollama)<br>• JSON-mode enforcement for structured output<br>• Fallback: returns raw messages if LLM is unavailable | | `index.js` (tools) | Orchestrate the pipeline + expose CLI tools | • Three tools: `run_digest_cycle`, `configure_channels`, `get_status`<br>• Graceful error handling: continues processing other channels if one fails<br>• Token usage reporting for cost tracking | ## ⚙️ Configuration: Real Examples ### Basic Setup (30 seconds) ```yaml # config.yaml monitoring: channels: - "durov" # Pavel Durov's channel - "telegram" # Official Telegram News - "tjournal" # Russian tech media - "https://t.me/s/roem" # Full URL format also supported - "@coindesk" # With @ prefix — also works fetch_window: 20 # Check last 20 messages per channel (catch up on missed updates) rate_limit_ms: 2000 # Delay between requests to avoid IP throttling digest: language: "en" # Summary language: "en", "ru", "es", etc. max_tokens_per_channel: 300 # Limit summary length (~2-3 short paragraphs) format: "markdown" # Output format: "markdown", "html", or "text" group_by: "time" # Organize digest: "time" (chronological), "channel", or "none" notify_channel: "default" # Where to send: "default", "telegram", "discord", "webhook" min_messages_for_summary: 2 # Don't summarize if <2 new messages (avoid noise) ``` ### Advanced: Custom LLM Provider ```yaml # If you use OpenRouter instead of default gateway: # Set in ~/.openclaw/openclaw.json: { "models": { "providers": { "openrouter": { "baseUrl": "https://openrouter.ai/api/v1", "apiKey": "${OPENROUTER_KEY}" } } }, "agents": { "defaults": { "model": { "primary": "openrouter/qwen/qwen3.5-35b" } } } } ``` Then in `config.yaml`: ```yaml digest: language: "en" # Skill automatically uses context.model.primary from OpenClaw ``` ### Use Case: Crypto Trader Monitoring ```yaml monitoring: channels: - "coindesk" - "theblock__" - "cryptobriefing" - "binance" - "elonmusk" # For meme-coin signals 😅 fetch_window: 50 # Deeper history for volatile markets rate_limit_ms: 3000 # Conservative to avoid blocks digest: language: "en" max_tokens_per_channel: 200 # Ultra-concise for quick scanning format: "text" # Plain text for SMS/Telegram compatibility group_by: "channel" # Group by source for attribution min_messages_for_summary: 1 # Even single important message triggers alert ``` ## ▶️ Usage Examples ### Run a Manual Check ```bash # Check all configured channels openclaw skills exec tg-news-digest-lite/run_digest_cycle # Override channels for one-off check openclaw skills exec tg-news-digest-lite/run_digest_cycle \ --args '{"channels": ["durov", "https://t.me/s/roem"]}' # Force re-fetch even if messages seem "seen" (debugging) openclaw skills exec tg-news-digest-lite/run_digest_cycle \ --args '{"force": true}' ``` ### Manage Channel List Dynamically ```bash # Add new channels openclaw skills exec tg-news-digest-lite/configure_channels \ --args '{"add": ["techcrunch", "wired"]}' # Remove channels you no longer care about openclaw skills exec tg-news-digest-lite/configure_channels \ --args '{"remove": ["old-channel"]}' # View current configuration openclaw skills exec tg-news-digest-lite/get_status ``` **Example output of `get_status`:** ```json { "tracked_channels": 7, "channels": ["durov", "telegram", "tjournal", "roem", "coindesk", "techcrunch", "wired"], "cache_entries": 342, "rate_limit_ms": 2000 } ``` ### Sample Digest Output When new messages are detected, you'll receive a formatted digest like: ```markdown 📰 **Telegram News Digest** 🕒 2026-04-28 14:30:00 UTC 🔹 **@durov** (3 msg) • Telegram announces new bot API features for group management • Privacy update: end-to-end encryption now available for group calls • Q2 2026 roadmap teaser: "Something big coming in July" 🏷 privacy, api, roadmap 🔹 **@coindesk** (5 msg) • Bitcoin holds $67K amid ETF inflow reports • SEC delays decision on Ethereum futures ETF again • New DeFi protocol raises $12M seed round 🏷 bitcoin, etf, defi, regulation 🔹 **@tjournal** (2 msg) • Russian tech startups face new compliance requirements • Yandex announces AI assistant integration across products 🏷 russia, startups, ai ``` ## 🔄 Automation: Run on Schedule ### Option 1: OpenClaw Cron (Recommended) Add to `~/.openclaw/openclaw.json`: ```json { "cron": [ { "id": "tg-digest-morning", "schedule": "0 9 * * *", "tool": [REDACTED_SECRET], "isolate": true, "description": "Morning news digest at 9 AM" }, { "id": "tg-digest-evening", "schedule": "0 18 * * *", "tool": [REDACTED_SECRET], "isolate": true, "description": "Evening recap at 6 PM" } ] } ``` ### Option 2: Heartbeat Integration Create `~/workspace/HEARTBEAT.md`: ```markdown ## Telegram News Check - Run [REDACTED_SECRET] every 30 minutes - Only send digest if new messages detected - Use `qwen3.5-35b` for summarization, `gpt-4o-mini` for routing ``` Then configure heartbeat in `openclaw.json`: ```json { "agents": { "defaults": { "heartbeat": { "every": "30m", "lightContext": true, "isolatedSession": true } } } } ``` ## ⚠️ Limitations & Edge Cases (Be Aware) ### What Works ✅ - Public channels with `https://t.me/s/{username}` accessible in browser - Text-only messages (including formatted text: bold, italic, links) - Channels with 1–1000+ messages (pagination handled internally) - Multi-language content (LLM handles translation if configured) ### What Doesn't Work ❌ - Private channels or groups (require authentication — use `tg-news-digest` Pro version) - Messages with only media (no text) — skipped silently - Channels that block scrapers via Cloudflare (rare, but possible) - Very high-frequency channels (>100 msgs/hour) — may hit rate limits ### Known Quirks 🔧 | Symptom | Likely Cause | Workaround | |---------|-------------|------------| | "Канал не найден" (Channel not found) | Channel is private or username changed | Verify URL works in browser: `https://t.me/s/username` | | Empty digest despite new messages | `min_messages_for_summary` too high | Lower to `1` or check `fetch_window` size | | LLM returns malformed JSON | Model doesn't support `response_format: json_object` | Switch to `qwen3.5-35b`, `claude-3.5`, or `gpt-4o` | | Repeated messages in digest | State file corrupted or deleted | Delete `state/seen_messages.json` to reset cache | | Slow execution (>30s) | Many channels + high `fetch_window` | Reduce `fetch_window` or increase `rate_limit_ms` | ## 🛡️ Security & Privacy Considerations ### Data Flow Transparency ``` Your machine → HTTPS GET to t.me/s/{channel} (public webpage) → Local text extraction + hashing → LLM API call (only message text, no metadata) → Digest sent to your configured channel ``` **No data leaves your control except:** 1. Public Telegram webpage requests (same as opening in browser) 2. Message text sent to your chosen LLM provider (configurable) ### Best Practices - ✅ Run on trusted infrastructure (your laptop, private server) - ✅ Use `isolate: true` in cron jobs to contain context growth - ✅ Rotate `OPENCLAW_GATEWAY_TOKEN` periodically if using remote gateway - ✅ Monitor `state/seen_messages.json` size (auto-truncated to 100 hashes/channel) ### What This Skill Does NOT Do - ❌ Does not store Telegram credentials (none required) - ❌ Does not forward raw messages to third parties - ❌ Does not modify Telegram content in any way - ❌ Does not interact with Telegram APIs (only public web pages) ## 📊 Performance Characteristics ### Resource Usage (Typical Run) | Metric | Value | Notes | |--------|-------|-------| | Execution time | 15–45 seconds | Depends on #channels and network latency | | Memory footprint | ~50–150 MB | Node.js + Cheerio + LLM response buffering | | Network requests | 1 per channel + 1 LLM call | With 2s rate limit between Telegram requests | | Token consumption | ~200–800 tokens/channel | For summarization only (not raw messages) | ### Cost Estimate (Using OpenRouter) Assuming `qwen/qwen3.5-35b` at $1.30/1M output tokens: ``` 10 channels × 300 tokens × $1.30/1M = $0.0039 per run Hourly execution (24×) = ~$0.09/day = ~$2.70/month ``` > 💡 Tip: Use cheaper models (`gpt-4o-mini`, `qwen2.5-7b`) for summarization if cost-sensitive. ## 🐛 Troubleshooting Guide ### Error: `Skill execution failed: Network timeout` ```bash # Check your internet connection ping t.me # Increase timeout in config.yaml monitoring: rate_limit_ms: 5000 # More conservative # Test scraper directly node -e " import('./src/scraper.js').then(m => { const s = new m.TelegramWebScraper({rate_limit_ms: 1000}, console); s.fetchChannelMessages('durov', 5).then(console.log); }); " ``` ### Error: `LLM response parse failed` ```bash # Check which model is being used openclaw config get agents.defaults.model.primary # Test LLM connectivity manually curl -H "Authorization: Bearer $OPENCLAW_GATEWAY_TOKEN" \ -H "Content-Type: application/json" \ -d '{"model":"qwen/qwen3.5-35b","messages":[{"role":"user","content":"test"}]}' \ https://your-gateway/v1/chat/completions # Fallback: switch to a model known to support JSON mode # Update ~/.openclaw/openclaw.json: { "agents": { "defaults": { "model": { "primary": "anthropic/claude-3.5-sonnet" } } } } ``` ### Debug Mode: See Internal Logs ```bash # Enable verbose logging export DEBUG=tg-news:* openclaw skills exec tg-news-digest-lite/run_digest_cycle --args '{"channels":["durov"]}' # Or check OpenClaw logs openclaw logs --grep "tg-news" --tail 50 ``` ### Reset State (Start Fresh) ```bash # Backup then delete state mv ~/.openclaw/skills/tg-news-digest-lite/state/seen_messages.json \ ~/.openclaw/skills/tg-news-digest-lite/state/seen_messages.json.bak # Next run will treat all messages as "new" ``` ## 🔄 Upgrading from Older Versions ### v0.x → v1.0.0 Breaking Changes - State file format changed: now uses SHA256 hashes instead of raw text - Migration: automatic on first run (old entries marked as "seen") - Config key renamed: `telegram_channels` → `monitoring.channels` ### Upgrade Steps ```bash # 1. Backup your config cp ~/.openclaw/skills/tg-news-digest-lite/config.yaml ~/backup-config.yaml # 2. Update the skill openclaw skills update tg-news-digest-lite # 3. Verify config structure openclaw skills exec tg-news-digest-lite/get_status # 4. (Optional) Migrate channel list format if needed ``` ## 🤝 Contributing & Feedback Found a bug? Have a feature request? 1. Check existing issues: https://github.com/your-org/tg-news-digest-lite/issues 2. For HTML parsing issues: include the raw HTML snippet (use `--debug` flag) 3. For LLM issues: specify your model provider and version 4. Submit PRs with tests for new channel formats ## 📜 License MIT License — use freely in personal and commercial projects. See [LICENSE](./LICENSE) for details. --- *Built with ❤️ for the OpenClaw community. Questions? Open an issue or ping @your-handle on Discord.* ```

path: README.md
content: # tg-news-digest-lite

> 🤖 Monitor public Telegram channels, auto-summarize news, get digests — zero setup.

✅ No Telegram API keys • ✅ No authentication • ✅ Works out of the box

## 🚀 Install in 30 seconds
```bash
openclaw skills install tg-news-digest-lite
```

## ⚙️ Configure (only channels!)
Edit `~/.openclaw/skills/tg-news-digest-lite/config.yaml`:
```yaml
monitoring:
  channels:
    - "durov"
    - "telegram"
    - "https://t.me/s/roem"
```

## ▶️ Run
```bash
# Manual check
openclaw skills exec tg-news-digest-lite/run_digest_cycle

# Add channels dynamically
openclaw skills exec tg-news-digest-lite/configure_channels --add "coindesk"
```

## 📚 Full documentation
See [SKILL.md](./SKILL.md) for:
- Architecture deep dive
- Real-world examples
- Troubleshooting guide
- Security considerations
- Performance tuning

---

*Built for the OpenClaw community. MIT License.*

path: package.json
content: {
  "name": "tg-news-digest-lite",
  "version": "1.0.0",
  "type": "module",
  "description": "Autonomous AI agent for monitoring public Telegram channels via web scraping, with zero authentication required",
  "main": "src/index.js",
  "scripts": {
    "test": "node --test test/*.test.js",
    "lint": "eslint src/**/*.js"
  },
  "keywords": [
    "telegram",
    "news-monitoring",
    "web-scraping",
    "ai-summarization",
    "openclaw-skill",
    "digest",
    "automation"
  ],
  "author": "your-name <[REDACTED_SECRET]>",
  "license": "MIT",
  "dependencies": {
    "axios": "^1.6.8",
    "cheerio": "^1.0.0-rc.12",
    "fs-extra": "^11.2.0",
    "openai": "^4.28.0",
    "yaml": "^2.4.1"
  },
  "engines": {
    "node": ">=20.0.0"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/your-org/tg-news-digest-lite.git"
  },
  "bugs": {
    "url": "https://github.com/your-org/tg-news-digest-lite/issues"
  },
  "homepage": "https://clawhub.ai/skills/tg-news-digest-lite"
}

path: config.example.yaml
content: # tg-news-digest-lite configuration template
# Copy to config.yaml and customize

monitoring:
  # List of public Telegram channels to monitor
  # Supported formats:
  #   - "username"          (e.g., "durov")
  #   - "@username"         (e.g., "@telegram")
  #   - "https://t.me/s/username"  (full URL)
  #   - "https://t.me/username"    (also works)
  channels:
    - "durov"
    - "telegram"
    - "tjournal"
  
  # How many recent messages to check per channel (for catching up)
  fetch_window: 20
  
  # Delay between HTTP requests to avoid rate limiting (milliseconds)
  # Increase if you get "429 Too Many Requests" errors
  rate_limit_ms: 2000
  
  # User-Agent string for HTTP requests
  # Change if Telegram blocks the default one
  user_agent: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"

digest:
  # Language for AI-generated summaries
  # Supported: "en", "ru", "es", "fr", "de", etc. (depends on your LLM)
  language: "en"
  
  # Maximum tokens for the summary per channel
  # ~100 tokens = 2-3 short paragraphs
  max_tokens_per_channel: 300
  
  # Output format for the digest
  # Options: "markdown" (rich formatting), "html", "text" (plain)
  format: "markdown"
  
  # How to group messages in the digest
  # "time" = chronological order across channels
  # "channel" = group by source channel
  # "none" = flat list
  group_by: "time"
  
  # Which OpenClaw channel to send the digest to
  # "default" = use the current session's output channel
  # "telegram", "discord", "webhook" = explicit routing
  notify_channel: "default"
  
  # Minimum number of new messages required to trigger summarization
  # Prevents noise from single-message updates
  min_messages_for_summary: 2

path: skill.yaml
content: id: tg-news-digest-lite
name: "Telegram News Digest (Lite)"
version: "1.0.0"
description: |
  Monitors public Telegram channels via web scraping (t.me/s/*), extracts new messages,
  generates AI-powered summaries, and delivers structured digests to your configured
  OpenClaw notification channel. Zero authentication required — just provide channel names.
entrypoint: "src/index.js"
runtime: "node20"
permissions:
  network: ["t.me", "*.t.me", "api.openai.com", "*.openrouter.ai", "*.deepinfra.com"]
  filesystem: ["./state", "./config.yaml"]
  env: ["OPENCLAW_GATEWAY_URL", "OPENCLAW_GATEWAY_TOKEN"]
tools:
  - name: run_digest_cycle
    description: "Executes the full monitoring cycle: fetch channel messages → detect new content → generate AI summary → deliver digest to configured channel"
    parameters:
      type: object
      properties:
        channels:
          type: array
          items: { type: string }
          description: "List of channels to check (username, @username, or full t.me URL). Overrides config.yaml if provided."
        force:
          type: boolean
          default: false
          description: "Ignore seen-messages cache and re-fetch all messages (useful for debugging or cache reset)"
  - name: configure_channels
    description: "Dynamically add or remove channels from the monitoring list without editing config files"
    parameters:
      type: object
      properties:
        add: { type: array, items: { type: string }, description: "Channel identifiers to add" }
        remove: { type: array, items: { type: string }, description: "Channel identifiers to remove" }
  - name: get_status
    description: "Return current operational status: tracked channels, cache size, last update timestamp"
    parameters: { type: object, properties: {} }

path: src/scraper.js
content: import axios from "axios";
import * as cheerio from "cheerio";
import crypto from "crypto";

export class TelegramWebScraper {
  constructor(config, logger) {
    this.config = config;
    this.logger = logger;
    this.client = axios.create({
      timeout: 15000,
      headers: {
        "User-Agent": config.user_agent || "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"
      },
      validateStatus: status => status < 500 // Handle 4xx as errors, not exceptions
    });
  }

  /**
   * Fetch and parse messages from a public Telegram channel webpage
   * @param {string} channelIdentifier - username, @username, or full URL
   * @param {number} limit - max messages to extract
   * @returns {Promise<{messages: Array, error?: string}>}
   */
  async fetchChannelMessages(channelIdentifier, limit = 20) {
    const username = this._normalizeChannel(channelIdentifier);
    if (!username) {
      return { error: `Invalid channel identifier: ${channelIdentifier}`, messages: [] };
    }
    
    const url = `https://t.me/s/${username}`;

    try {
      this.logger.debug(`Fetching ${url}`);
      const { data: html, status } = await this.client.get(url);
      
      if (status === 404) {
        return { 
          error: `Channel not found or private: @${username}. Verify it's public at ${url}`, 
          messages: [] 
        };
      }
      if (status >= 400) {
        return { 
          error: `HTTP ${status} from ${url}. Telegram may be rate-limiting requests.`, 
          messages: [] 
        };
      }

      return { messages: this._parseMessages(html, username, limit), error: null };
    } catch (err) {
      return { 
        error: `Network error fetching ${username}: ${err.message}`, 
        messages: [] 
      };
    }
  }

  /**
   * Normalize various channel identifier formats to plain username
   * Supports: "durov", "@durov", "https://t.me/s/durov", "https://t.me/durov"
   */
  _normalizeChannel(identifier) {
    if (!identifier || typeof identifier !== "string") return null;
    
    // Extract username from URL or raw identifier
    const urlMatch = identifier.match(/t\.me\/(?:s\/)?@?([^/?#]+)/i);
    const rawMatch = identifier.match(/^@?([^/?#]+)$/);
    
    const username = (urlMatch || rawMatch)?.[1]?.toLowerCase();
    
    // Basic validation: alphanumeric + underscores, 3-32 chars
    if (username && /^[a-z0-9_]{3,32}$/.test(username)) {
      return username;
    }
    return null;
  }

  /**
   * Parse HTML from t.me/s/* page and extract message objects
   * Resilient to minor Telegram HTML structure changes
   */
  _parseMessages(html, channel, limit) {
    const $ = cheerio.load(html);
    const messages = [];

    // Primary selector: Telegram's standard message text container
    // Fallback: if structure changes, we'd need to update this
    $('.tgme_widget_message_text').each((i, el) => {
      if (i >= limit) return false; // Respect limit
      
      // Extract and clean text content
      let text = $(el).text()
        .replace(/\s+/g, ' ')           // Normalize whitespace
        .replace(/[\u200B-\u200D\uFEFF]/g, '') // Remove zero-width chars
        .replace(/\n{3,}/g, '\n\n')     // Limit consecutive newlines
        .trim();
      
      // Filter out noise: empty, too short, or boilerplate
      if (text && text.length > 15 && !this._isBoilerplate(text)) {
        const hash = this._hashMessage(text);
        const timestamp = this._extractTimestamp($(el)) || Date.now();
        
        messages.push({
          id: hash,      // Unique identifier for deduplication
          text,          // Cleaned message content
          timestamp,     // Approximate message time (ms since epoch)
          channel        // Source channel username
        });
      }
    });

    // Return in chronological order (oldest → newest) for coherent summarization
    return messages.reverse();
  }

  /**
   * Attempt to extract message timestamp from HTML
   * Returns milliseconds since epoch, or null if unavailable
   */
  _extractTimestamp($el) {
    // Try <time datetime="..."> first (most reliable)
    const timeEl = $el.closest('.tgme_widget_message').find('time[datetime]');
    if (timeEl.length) {
      const dt = new Date(timeEl.attr('datetime'));
      if (!isNaN(dt.getTime())) return dt.getTime();
    }
    
    // Fallback: data-post attribute contains channel/message_id
    // We can't get exact time, but can use as secondary dedup key
    const postAttr = $el.closest('.tgme_widget_message').attr('data-post');
    if (postAttr) {
      // Format: "channel/12345" — we already have channel, so just log
      this.logger.debug(`Found data-post: ${postAttr} (using fallback timestamp)`);
    }
    
    return null;
  }

  /**
   * Filter out common boilerplate text that isn't user content
   */
  _isBoilerplate(text) {
    const patterns = [
      /forwarded from/i,
      /edited\b/i,
      /pinned message/i,
      /this channel is/i,
      /join chat/i
    ];
    return patterns.some(re => re.test(text));
  }

  /**
   * Generate deterministic hash for message deduplication
   * Uses SHA256 of normalized text (case-insensitive, trimmed)
   */
  _hashMessage(text) {
    const normalized = text.trim().toLowerCase();
    return crypto
      .createHash('sha256')
      .update(normalized)
      .digest('hex')
      .slice(0, 16); // 16-char hex = 64 bits, sufficient for dedup
  }

  /**
   * Simple delay utility for rate limiting
   */
  async delay(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

path: src/index.js
content: import fs from "fs-extra";
import yaml from "yaml";
import { TelegramWebScraper } from "./scraper.js";
import { StateManager } from "./storage.js";
import { Summarizer } from "./summarizer.js";

// Minimal logger for skill runtime
const logger = {
  info: (...a) => console.log("[INFO]", ...a),
  warn: (...a) => console.warn("[WARN]", ...a),
  error: (...a) => console.error("[ERROR]", ...a),
  debug: (...a) => process.env.DEBUG && console.debug("[DEBUG]", ...a)
};

export const tools = {
  /**
   * Main tool: execute the full news digest pipeline
   * 
   * Flow:
   * 1. Load configuration (config.yaml or args)
   * 2. For each channel:
   *    a. Fetch public webpage via TelegramWebScraper
   *    b. Parse messages, generate SHA256 hashes
   *    c. Filter out already-seen messages using StateManager
   *    d. If new messages exist and meet threshold, send to Summarizer
   *    e. Update seen-messages cache
   * 3. Format all summaries into a single digest
   * 4. Deliver via ctx.send to configured notify_channel
   * 
   * @param {Object} params
   * @param {Object} params.context - OpenClaw runtime context (model, send, etc.)
   * @param {Object} params.args - Tool arguments from user/cli
   * @returns {Promise<Object>} Execution result with status and metrics
   */
  async run_digest_cycle({ context, args }) {
    const config = await this._loadConfig();
    const scraper = new TelegramWebScraper(config.monitoring, logger);
    const storage = new StateManager();
    const llm = new Summarizer(
      process.env.OPENCLAW_GATEWAY_URL || "http://127.0.0.1:18789",
      context.model?.primary
    );

    // Resolve channel list: args override config
    const rawChannels = args.channels?.length 
      ? args.channels 
      : config.monitoring.channels;
    
    const channels = rawChannels
      .map(c => scraper._normalizeChannel(c))
      .filter(Boolean); // Remove invalid entries

    if (!channels.length) {
      return { 
        status: "error", 
        message: "No valid channels provided. Use --args '{\"channels\":[\"durov\"]}' or update config.yaml" 
      };
    }

    let digest = [];
    let totalTokens = 0;
    let errors = [];

    // Process each channel sequentially (with rate limiting)
    for (const channel of channels) {
      try {
        // Respect rate limit to avoid IP throttling
        await scraper.delay(config.monitoring.rate_limit_ms);

        // Fetch and parse messages from public webpage
        const result = await scraper.fetchChannelMessages(
          channel, 
          config.monitoring.fetch_window
        );
        
        if (result.error) {
          errors.push(`${channel}: ${result.error}`);
          continue; // Continue with other channels
        }

        // Load previously seen message hashes for deduplication
        const seen = await storage.getSeenHashes(channel);
        
        // Filter to only new messages (by hash)
        const newMessages = result.messages.filter(m => !seen.has(m.id));

        // Skip if no new content (and not forced)
        if (!newMessages.length && !args.force) {
          logger.debug(`No new messages in @${channel}`);
          continue;
        }

        // Generate AI summary if enough new messages
        if (newMessages.length >= (config.digest.min_messages_for_summary || 1)) {
          const summary = await llm.generate(newMessages, config.digest);
          digest.push({
            channel,
            count: newMessages.length,
            ...summary
          });
          totalTokens += summary.token_count;
        }

        // Update cache with ALL fetched messages (not just new ones)
        // This ensures we don't re-process the same messages next time
        const allHashes = result.messages.map(m => m.id);
        await storage.updateSeenHashes(channel, allHashes);

      } catch (err) {
        errors.push(`${channel}: ${err.message}`);
        logger.error(`Pipeline error for ${channel}:`, err);
        // Continue processing other channels (fault isolation)
      }
    }

    // Deliver digest if we have content to report
    if (digest.length > 0) {
      const formatted = this._formatDigest(digest, config.digest);
      await this._notify(formatted, config.digest.notify_channel, context);
    }

    // Return structured result for logging/auditing
    return {
      status: digest.length ? "success" : "no_new_content",
      channels_checked: channels.length,
      channels_with_updates: digest.length,
      consumed_tokens: totalTokens,
      errors: errors.length ? errors : undefined,
      timestamp: new Date().toISOString()
    };
  },

  /**
   * Dynamically manage the list of monitored channels
   * Updates config.yaml and clears cache for removed channels
   */
  async configure_channels({ args }) {
    const configPath = "./config.yaml";
    const config = yaml.parse(await fs.readFile(configPath, "utf8"));
    const current = new Set(config.monitoring.channels);

    // Add new channels (normalize format)
    if (args.add?.length) {
      args.add.forEach(c => {
        const normalized = c.match(/t\.me\/(?:s\/)?@?([^/?#]+)/i)?.[1] || c.replace(/^@/, "");
        if (normalized) current.add(normalized);
      });
    }
    
    // Remove channels and clear their cache
    if (args.remove?.length) {
      args.remove.forEach(c => {
        const normalized = c.match(/t\.me\/(?:s\/)?@?([^/?#]+)/i)?.[1] || c.replace(/^@/, "");
        if (normalized) {
          current.delete(normalized);
          // Clear cache for removed channel
          const storage = new StateManager();
          await storage.clearChannel(normalized);
        }
      });
    }

    // Persist updated config
    config.monitoring.channels = Array.from(current);
    await fs.writeFile(configPath, yaml.stringify(config), "utf8");

    return { 
      status: "updated", 
      channels: Array.from(current),
      message: `Now monitoring ${current.size} channel(s)` 
    };
  },

  /**
   * Return operational status for monitoring/health checks
   */
  async get_status() {
    const config = await this._loadConfig();
    const storage = new StateManager();
    const state = await storage.init();
    
    return {
      tracked_channels: config.monitoring.channels.length,
      channels: config.monitoring.channels,
      cache_entries: Object.keys(state.channels).reduce(
        (sum, ch) => sum + (state.channels[ch]?.length || 0), 0
      ),
      rate_limit_ms: config.monitoring.rate_limit_ms,
      last_check: state.meta?.last_run || "never"
    };
  },

  // ===== Internal helpers =====

  async _loadConfig() {
    try {
      return yaml.parse(await fs.readFile("./config.yaml", "utf8"));
    } catch (err) {
      logger.warn("Config not found, using defaults");
      return {
        monitoring: { 
          channels: [], 
          fetch_window: 20, 
          rate_limit_ms: 2000,
          user_agent: "Mozilla/5.0 (X11; Linux x86_64)"
        },
        digest: { 
          language: "en", 
          max_tokens_per_channel: 300, 
          format: "markdown", 
          notify_channel: "default",
          min_messages_for_summary: 2
        }
      };
    }
  },

  _formatDigest(items, cfg) {
    const header = `📰 **Telegram News Digest**\n🕒 ${new Date().toLocaleString('en-US', { timeZone: 'UTC' })} UTC\n\n`;
    
    let body = "";
    const sorted = cfg.group_by === "time" 
      ? items.sort((a, b) => b.channel.localeCompare(a.channel)) // placeholder: real time sort would need timestamps
      : items;

    for (const item of sorted) {
      body += `🔹 **@${item.channel}** (${item.count} new message${item.count > 1 ? 's' : ''})\n`;
      body += `${item.summary}\n`;
      if (item.tags?.length) body += `🏷 ${item.tags.join(", ")}\n`;
      body += `\n`;
    }
    
    const footer = cfg.format === "markdown" 
      ? `\n*Generated by tg-news-digest-lite • [Report issues](https://github.com/your-org/tg-news-digest-lite)*`
      : "";

    return header + body + footer;
  },

  async _notify(text, channel, ctx) {
    // Preferred: use OpenClaw's session.send for integrated delivery
    if (typeof ctx?.send === "function") {
      await ctx.send({ channel, content: text });
      logger.info(`Digest sent to channel: ${channel}`);
    } else {
      // Fallback: log to stdout (for cron/heartbeat mode)
      console.log(`\n[NOTIFY → ${channel}]\n${text}\n`);
      logger.info(`Digest printed to stdout (configure notify_channel for integrated delivery)`);
    }
  }
};

export default { tools };

path: src/summarizer.js
content: import { createOpenAI } from "openai";

export class Summarizer {
  constructor(gatewayUrl, defaultModel) {
    this.client = createOpenAI({
      baseURL: `${gatewayUrl}/v1`,
      [REDACTED_SECRET] || "openclaw",
      timeout: 30000
    });
    this.model = defaultModel || "qwen/qwen3.5-35b";
  }

  /**
   * Generate structured summary from message list using LLM
   * @param {Array} messages - Array of {text, timestamp, channel}
   * @param {Object} config - Digest configuration (language, tokens, etc.)
   * @returns {Promise<Object>} {summary, tags, sentiment, token_count, status}
   */
  async generate(messages, config) {
    if (!messages?.length) {
      return { summary: "", token_count: 0, status: "empty", tags: [], sentiment: "neutral" };
    }

    const prompt = this._buildPrompt(messages, config);
    
    try {
      const res = await this.client.chat.completions.create({
        model: this.model,
        messages: [{ role: "user", content: prompt }],
        temperature: 0.3, // Low temperature for factual summaries
        max_tokens: config.max_tokens_per_channel || 300,
        response_format: { type: "json_object" } // Enforce JSON output
      });

      const raw = res.choices[0]?.message?.content || "{}";
      const parsed = JSON.parse(raw);
      
      return {
        summary: parsed.summary || "No significant updates detected.",
        tags: Array.isArray(parsed.tags) ? parsed.tags : [],
        sentiment: ["positive", "neutral", "negative"].includes(parsed.sentiment) 
          ? parsed.sentiment 
          : "neutral",
        token_count: res.usage?.completion_tokens || 0,
        status: "success"
      };
    } catch (err) {
      // Graceful fallback: return raw messages if LLM fails
      const fallbackText = messages
        .slice(-3) // Last 3 messages as representative sample
        .map(m => `• ${m.text}`)
        .join("\n");
      
      return {
        summary: `[LLM unavailable] Latest messages:\n${fallbackText}`,
        tags: [],
        sentiment: "unknown",
        token_count: 0,
        status: "fallback"
      };
    }
  }

  /**
   * Construct prompt that guides LLM to produce structured, concise output
   */
  _buildPrompt(messages, config) {
    const messageList = messages
      .map(m => `• ${m.text}`)
      .join("\n");
    
    return `You are a professional news analyst. Review these Telegram messages and return ONLY valid JSON:

{
  "summary": "Concise summary in ${config.language} (max ${config.max_tokens_per_channel} tokens). Focus on factual updates, ignore spam/repeats/ads.",
  "tags": ["topic1", "topic2"],
  "sentiment": "positive|neutral|negative"
}

Messages from channel:
${messageList}

Rules:
- Output MUST be valid JSON, no markdown, no explanations
- Summary should be 2-4 bullet points worth of information
- Tags should be 2-4 relevant keywords
- If no meaningful content, set summary to "No significant updates.`;
  }
}

path: src/storage.js
content: import fs from "fs-extra";
import path from "path";

export class StateManager {
  constructor(dir = "./state") {
    this.path = path.join(dir, "seen_messages.json");
  }

  /**
   * Initialize state file (create if missing) and return parsed content
   */
  async init() {
    await fs.ensureFile(this.path);
    try {
      return await fs.readJson(this.path);
    } catch (err) {
      // Corrupted or unreadable file — start fresh
      return { channels: {}, meta: { created_at: new Date().toISOString() } };
    }
  }

  /**
   * Get set of seen message hashes for a channel
   * @param {string} channel - normalized username
   * @returns {Promise<Set<string>>}
   */
  async getSeenHashes(channel) {
    const state = await this.init();
    const hashes = state.channels?.[channel] || [];
    return new Set(hashes);
  }

  /**
   * Update seen hashes for a channel with rolling window
   * @param {string} channel - normalized username
   * @param {string[]} newHashes - hashes to add
   * @param {number} maxKeep - max hashes to retain per channel (default: 100)
   */
  async updateSeenHashes(channel, newHashes, maxKeep = 100) {
    const state = await this.init();
    if (!state.channels) state.channels = {};
    
    const existing = state.channels[channel] || [];
    // Merge + deduplicate + limit size
    const merged = [...new Set([...existing, ...newHashes])];
    if (merged.length > maxKeep) {
      merged.splice(0, merged.length - maxKeep);
    }
    
    state.channels[channel] = merged;
    state.meta = state.meta || {};
    state.meta.last_updated = new Date().toISOString();
    
    await fs.outputJson(this.path, state, { spaces: 2 });
  }

  /**
   * Clear all cached hashes for a channel (e.g., on removal)
   */
  async clearChannel(channel) {
    const state = await this.init();
    if (state.channels?.[channel]) {
      delete state.channels[channel];
      await fs.outputJson(this.path, state, { spaces: 2 });
    }
  }
}

Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)