Skill: hegghammer/newsapi-search (version 1.0.0)

--- name: newsapi-search version: 1.0.0 description: Search news articles via NewsAPI with filtering by time windows, sources, domains, and languages. --- # NewsAPI Search Search 5,000+ news sources via [NewsAPI](https://newsapi.org). Supports comprehensive article discovery (/everything) and breaking headlines (/top-headlines). ## Quick Start ```bash # Basic search node scripts/search.js "technology" --days 7 # Filter by quality sources node scripts/search.js "technology" --sources bbc-news,reuters,al-jazeera-english # Exclude low-quality domains node scripts/search.js "technology" --exclude tmz.com,radaronline.com # Breaking headlines node scripts/search.js "technology" --headlines --country us # List available sources node scripts/sources.js --country us --category general ``` ## Setup Add API key to `~/.openclaw/.env`: ``` NEWSAPI_KEY=your_api_key ``` Get key from https://newsapi.org (free tier: 100 requests/day) ## Endpoints ### Everything Search Comprehensive search across millions of articles. **Time Windows:** ```bash node scripts/search.js "query" --hours 24 node scripts/search.js "query" --days 7 # default node scripts/search.js "query" --weeks 2 node scripts/search.js "query" --months 1 node scripts/search.js "query" --from 2026-01-01 --to 2026-01-31 ``` **Filters:** ```bash node scripts/search.js "query" --sources bbc-news,cnn # max 20 node scripts/search.js "query" --domains nytimes.com,bbc.co.uk node scripts/search.js "query" --exclude gossip-site.com node scripts/search.js "query" --lang en # or 'any' ``` **Search Fields:** ```bash node scripts/search.js "query" --title-only # title only node scripts/search.js "query" --in title,description # specific fields ``` **Advanced Query Syntax:** - `"exact phrase"` — exact match - `+musthave` — required word - `-exclude` — excluded word - `word1 AND word2` — both required - `word1 OR word2` — either accepted - `(word1 OR word2) AND word3` — grouping **Pagination & Sorting:** ```bash node scripts/search.js "query" --page 2 --limit 20 node scripts/search.js "query" --sort relevancy # default node scripts/search.js "query" --sort date # newest first node scripts/search.js "query" --sort popularity ``` ### Top Headlines Live breaking news by country or category. ```bash # By country node scripts/search.js "query" --headlines --country us # By category node scripts/search.js --headlines --country us --category business # By source node scripts/search.js --headlines --sources bbc-news,cnn ``` Categories: `business`, `entertainment`, `general`, `health`, `science`, `sports`, `technology` **Note:** Cannot mix `--country`/`--category` with `--sources` in headlines mode. ### List Sources ```bash node scripts/sources.js # all sources node scripts/sources.js --country us # filter by country node scripts/sources.js --category business node scripts/sources.js --lang en node scripts/sources.js --json # JSON output ``` ## Advanced Usage For complete parameter reference, see [references/api-reference.md](references/api-reference.md). For common workflows and search patterns, see [references/examples.md](references/examples.md). ## Programmatic API ```javascript const { searchEverything, searchHeadlines, getSources } = require('./scripts/search.js'); const results = await searchEverything('climate change', { timeWindow: { type: 'days', value: 7 }, sources: 'bbc-news,reuters', excludeDomains: 'tmz.com', limit: 20 }); const headlines = await searchHeadlines('business', { country: 'us', category: 'business' }); ``` ## Free Tier Limits - 100 requests/day - 100 results per request (max) - 1-month delay on archived content ## Output Format Returns structured JSON: ```json { "query": "technology", "endpoint": "everything", "totalResults": 64, "returnedResults": 10, "page": 1, "results": [ { "title": "...", "url": "...", "source": "BBC News", "publishedAt": "2026-02-05T14:30:00Z", "description": "...", "content": "..." } ] } ```

path: references/api-reference.md
content: # NewsAPI Parameter Reference

Complete reference for all NewsAPI endpoints and parameters.

## Table of Contents

- [Everything Endpoint](#everything-endpoint)
- [Top Headlines Endpoint](#top-headlines-endpoint)
- [Sources Endpoint](#sources-endpoint)
- [Query Syntax](#query-syntax)
- [Language Codes](#language-codes)
- [Country Codes](#country-codes)
- [Category Codes](#category-codes)

---

## Everything Endpoint

`GET https://newsapi.org/v2/everything`

Search through millions of articles from 5,000+ sources.

### Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `q` | string | No* | Keywords or phrases (max 500 chars). Supports advanced search. Required unless using `qInTitle`. |
| `qInTitle` | string | No* | Keywords/phrases to search in article titles only. |
| `sources` | string | No | Comma-separated source IDs (max 20). Cannot mix with `domains`. |
| `domains` | string | No | Comma-separated domains to restrict search (e.g., `bbc.co.uk,nytimes.com`). |
| `excludeDomains` | string | No | Comma-separated domains to exclude. |
| `from` | string | No | Start date (ISO 8601: `2026-02-05` or `2026-02-05T21:13:17`). |
| `to` | string | No | End date (ISO 8601). |
| `language` | string | No | 2-letter ISO-639-1 code. Default: all languages. |
| `sortBy` | string | No | `relevancy` (default), `publishedAt`, `popularity`. |
| `pageSize` | int | No | Results per page. Default: 100. Max: 100. |
| `page` | int | No | Page number. Default: 1. |
| `searchIn` | string | No | Fields to search: `title`, `description`, `content` (comma-separated). |
| `apiKey` | string | **Yes** | Your API key. |

### CLI Mapping

| CLI Flag | API Parameter |
|----------|---------------|
| `--title-only` | Uses `qInTitle` instead of `q` |
| `--sources` | `sources` |
| `--domains` | `domains` |
| `--exclude` | `excludeDomains` |
| `--from` | `from` |
| `--to` | `to` |
| `--lang` | `language` |
| `--sort` | `sortBy` (`date` maps to `publishedAt`) |
| `--limit` | `pageSize` |
| `--page` | `page` |
| `--in` | `searchIn` |

---

## Top Headlines Endpoint

`GET https://newsapi.org/v2/top-headlines`

Live breaking headlines for a country, category, or source.

### Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `country` | string | No | 2-letter ISO 3166-1 code. **Cannot mix with `sources`**. |
| `category` | string | No | Category. **Cannot mix with `sources`**. |
| `sources` | string | No | Comma-separated source IDs. **Cannot mix with `country` or `category`**. |
| `q` | string | No | Keywords to search in headlines. |
| `pageSize` | int | No | Results per page. Default: 20. Max: 100. |
| `page` | int | No | Page number. Default: 1. |
| `apiKey` | string | **Yes** | Your API key. |

### CLI Mapping

| CLI Flag | API Parameter |
|----------|---------------|
| `--headlines` | Switches to top-headlines endpoint |
| `--country` | `country` |
| `--category` | `category` |
| `--sources` | `sources` |
| `--limit` | `pageSize` |
| `--page` | `page` |

---

## Sources Endpoint

`GET https://newsapi.org/v2/top-headlines/sources`

List available news sources.

### Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `country` | string | No | Filter by 2-letter country code. |
| `category` | string | No | Filter by category. |
| `language` | string | No | Filter by 2-letter language code. |
| `apiKey` | string | **Yes** | Your API key. |

### Response Fields

| Field | Description |
|-------|-------------|
| `id` | Source identifier (use with `--sources`). |
| `name` | Display name. |
| `description` | Source description. |
| `url` | Homepage URL. |
| `category` | News category. |
| `language` | 2-letter language code. |
| `country` | 2-letter country code. |

---

## Query Syntax

NewsAPI supports Google's search syntax for the `q` parameter.

### Operators

| Syntax | Meaning | Example |
|--------|---------|---------|
| `"phrase"` | Exact match | `"police manhunt"` |
| `+word` | Must include | `+manhunt +fugitive` |
| `-word` | Must exclude | `manhunt -gaming` |
| `AND` | Both required | `police AND manhunt` |
| `OR` | Either acceptable | `manhunt OR hunt` |
| `NOT` | Exclude | `manhunt NOT game` |
| `()` | Grouping | `(police OR federal) AND manhunt` |

### Examples

```bash
# Exact phrase in title
node scripts/search.js '"police manhunt"' --title-only

# Must include police, exclude gaming
node scripts/search.js '+police +manhunt -gaming'

# Terrorist OR fugitive, plus manhunt
node scripts/search.js '(terrorist OR fugitive) AND manhunt'
```

---

## Language Codes

| Code | Language |
|------|----------|
| `ar` | Arabic |
| `de` | German |
| `en` | English |
| `es` | Spanish |
| `fr` | French |
| `he` | Hebrew |
| `it` | Italian |
| `nl` | Dutch |
| `no` | Norwegian |
| `pt` | Portuguese |
| `ru` | Russian |
| `sv` | Swedish |
| `ud` | Urdu |
| `zh` | Chinese |

Use `--lang any` to disable language filtering.

---

## Country Codes

Common country codes for headlines/sources:

| Code | Country |
|------|---------|
| `au` | Australia |
| `br` | Brazil |
| `ca` | Canada |
| `cn` | China |
| `de` | Germany |
| `fr` | France |
| `gb` | United Kingdom |
| `in` | India |
| `it` | Italy |
| `jp` | Japan |
| `ru` | Russia |
| `us` | United States |

Full list: https://newsapi.org/sources

---

## Category Codes

Available for headlines and source filtering:

| Code | Description |
|------|-------------|
| `business` | Business news |
| `entertainment` | Entertainment/gossip |
| `general` | General news |
| `health` | Health & medicine |
| `science` | Science & technology |
| `sports` | Sports |
| `technology` | Technology |

---

## Sort Options

| Option | Description |
|--------|-------------|
| `relevancy` | Articles most relevant to query first. |
| `publishedAt` | Newest articles first. |
| `popularity` | Articles from popular sources first. |

CLI shortcuts: `--sort relevancy`, `--sort date` (maps to `publishedAt`), `--sort popularity`.

---

## Error Codes

| Code | Meaning | Resolution |
|------|---------|------------|
| `apiKeyMissing` | No API key provided | Add `NEWSAPI_KEY` to `.env` |
| `apiKeyInvalid` | Invalid API key | Verify key at newsapi.org |
| `rateLimited` | Too many requests | Wait or upgrade plan |
| `parametersMissing` | Required params missing | Check query/topic provided |
| `parametersInvalid` | Invalid parameter value | Check date formats, codes |
| `sourceDoesNotExist` | Source ID not found | Run `sources.js` to list valid IDs |


path: references/examples.md
content: # NewsAPI Search Examples

Common workflows and search patterns for research and monitoring.

## Table of Contents

- [Research Workflows](#research-workflows)
- [Source Filtering](#source-filtering)
- [Time-Based Searches](#time-based-searches)
- [Headlines Monitoring](#headlines-monitoring)
- [Advanced Query Patterns](#advanced-query-patterns)

---

## Research Workflows

### Track a Topic Over Time

Paginate through results to build a comprehensive dataset:

```bash
# Get first page
node scripts/search.js "climate summit" --weeks 2 --limit 100 --page 1 > page1.json

# Get second page
node scripts/search.js "climate summit" --weeks 2 --limit 100 --page 2 > page2.json

# Sort by date for chronological analysis
node scripts/search.js "climate summit" --weeks 2 --sort date --limit 100
```

### Monitor Breaking Developments

Check last 24 hours for new stories:

```bash
node scripts/search.js "product launch" --hours 24 --sort date
```

### Historical Analysis

Search specific date ranges:

```bash
# Specific event period
node scripts/search.js "olympics" --from 2024-07-26 --to 2024-08-11

# Year-long study
node scripts/search.js "electric vehicles" --from 2025-01-01 --to 2025-12-31 --limit 100
```

---

## Source Filtering

### Quality News Only

Filter to major reputable outlets:

```bash
node scripts/search.js "technology" --days 7 \
  --sources bbc-news,reuters,associated-press,al-jazeera-english,the-guardian-uk
```

### Regional Focus

UK sources only:

```bash
# Method 1: Source IDs
node scripts/search.js "business" --sources bbc-news,the-guardian-uk,daily-mail

# Method 2: Domain filtering
node scripts/search.js "business" --domains bbc.co.uk,theguardian.com

# Method 3: List UK sources first
node scripts/sources.js --country gb
```

### Exclude Low-Quality Sources

Remove tabloids, gossip, and aggregators:

```bash
node scripts/search.js "finance" --days 7 \
  --exclude tmz.com,radaronline.com,dailystar.co.uk
```

### Academic/Policy Sources

Search think tanks and policy publications:

```bash
node scripts/search.js "economics" --domains brookings.edu,cfr.org,foreignpolicy.com
```

---

## Time-Based Searches

### Recency Levels

```bash
# Breaking (last hour)
node scripts/search.js "superbowl" --hours 1

# Recent (today)
node scripts/search.js "superbowl" --hours 24

# This week
node scripts/search.js "superbowl" --days 7

# This month
node scripts/search.js "superbowl" --weeks 4

# Quarter
node scripts/search.js "superbowl" --months 3
```

### Date Ranges for Studies

```bash
# Specific event period
node scripts/search.js "world cup" --from 2022-11-20 --to 2022-12-18

# Annual comparison
node scripts/search.js "electric vehicles" --from 2024-01-01 --to 2024-12-31

# Pre/post event analysis
node scripts/search.js "stock market" --from 2025-06-01 --to 2025-07-01
```

---

## Headlines Monitoring

### Current Breaking News

```bash
# US breaking news
node scripts/search.js --headlines --country us --limit 20

# With keyword filter
node scripts/search.js "elon musk" --headlines --country us

# By category
node scripts/search.js --headlines --country us --category business
node scripts/search.js --headlines --country gb --category technology
```

### International Headlines

```bash
node scripts/search.js --headlines --country de  # Germany
node scripts/search.js --headlines --country fr  # France
node scripts/search.js --headlines --country au  # Australia
```

### Source-Specific Headlines

```bash
# BBC headlines only
node scripts/search.js --headlines --sources bbc-news

# Multiple sources
node scripts/search.js --headlines --sources bbc-news,cnn,reuters
```

---

## Advanced Query Patterns

### Precise Term Matching

```bash
# Exact phrase in title (high precision)
node scripts/search.js '"climate change"' --title-only

# Multiple required terms
node scripts/search.js '+apple +iphone +review' --days 7

# Phrase with exclusions
node scripts/search.js '"superbowl halftime" -commercial -ads' --days 7
```

### Concept Expansion

```bash
# Synonym expansion with OR
node scripts/search.js '(electric OR ev OR battery) AND (car OR vehicle)' --days 7

# Different actor types
node scripts/search.js '(government OR federal OR state) AND policy' --days 7

# Geographic variations
node scripts/search.js 'summit AND (London OR Paris OR Berlin OR Madrid)' --weeks 2
```

### Boolean Logic

```bash
# Complex grouping
node scripts/search.js '(sports OR athletics OR competition) AND (olympics OR world cup) NOT (video OR game)'

# Multiple conditions
node scripts/search.js '(+tesla +elon) OR (+spacex +rocket)' --days 7
```

---

## Language & Region Patterns

### Non-English Sources

```bash
# Spanish language
node scripts/search.js "fútbol" --lang es --days 7

# German language
node scripts/search.js "bundesliga" --lang de --days 7

# French language
node scripts/search.js "coupe du monde" --lang fr --days 7

# All languages (no filter)
node scripts/search.js "climate" --lang any --days 7
```

### Regional by Domain

```bash
# UK press
node scripts/search.js "business" --domains bbc.co.uk,theguardian.com,dailymail.co.uk

# US press
node scripts/search.js "business" --domains nytimes.com,cnn.com,foxnews.com

# International mix
node scripts/search.js "business" --domains bbc.co.uk,aljazeera.com,reuters.com
```

---

## Sorting Strategies

### By Relevance (Default)

Best for focused research on specific topics:

```bash
node scripts/search.js '"artificial intelligence"' --sort relevancy --limit 20
```

### By Date

Best for tracking chronology:

```bash
# Newest first (breaking news)
node scripts/search.js "product launch" --sort date --hours 24

# For historical timeline analysis
node scripts/search.js "product launch" --sort date --weeks 4
```

### By Popularity

Best for finding "most discussed" stories:

```bash
node scripts/search.js "celebrity news" --sort popularity --days 7
```

---

## Discovery Workflows

### Find Related Sources

```bash
# 1. Search broadly
node scripts/search.js "technology" --days 1 --limit 100 > results.json

# 2. Extract unique sources from results
# (Use jq: cat results.json | jq -r '.results[].source' | sort | uniq)

# 3. Get source IDs
node scripts/sources.js --country us --category general

# 4. Re-search with specific sources
node scripts/search.js "technology" --sources source-id-1,source-id-2
```

### Source Quality Assessment

```bash
# List sources in a country
node scripts/sources.js --country us

# Test each major source
node scripts/search.js "test" --sources bbc-news --limit 1
node scripts/search.js "test" --sources cnn --limit 1
```

---

## Research Output Workflows

### Save Results for Analysis

```bash
# JSON for programmatic analysis
node scripts/search.js "electric vehicles" --weeks 2 --limit 100 > ev_articles.json

# View with jq
# cat ev_articles.json | jq '.results[] | {title, source, date: .publishedAt}'

# Extract URLs for archiving
# cat ev_articles.json | jq -r '.results[].url'
```

### Comparative Searches

```bash
# Compare coverage of different terms
node scripts/search.js "iphone" --weeks 1 > term1.json
node scripts/search.js "android" --weeks 1 > term2.json

# Compare counts
# jq '.totalResults' term1.json term2.json
```

---

## Common Research Scenarios

### Scenario: Market Research

```bash
# Comprehensive search across quality sources
node scripts/search.js "product launch" --months 6 \
  --sources bbc-news,reuters,associated-press,techcrunch,the-verge \
  --sort date \
  --limit 100
```

### Scenario: Media Monitoring

```bash
# Daily check for new stories
node scripts/search.js "company OR brand" --hours 24 --sort date

# Exclude entertainment/gaming false positives
node scripts/search.js "apple" --hours 24 --exclude polygon.com,kotaku.com,ign.com
```

### Scenario: Event Tracking

```bash
# During active event (every few hours)
node scripts/search.js "superbowl 2026" --hours 6 --sort date

# After event (daily digest)
node scripts/search.js "superbowl recap" --hours 24 --sort relevancy
```

### Scenario: Comparative Regional Study

```bash
# Same topic, different countries
node scripts/search.js "electric vehicles" --headlines --country us > us_headlines.json
node scripts/search.js "electric vehicles" --headlines --country gb > uk_headlines.json
node scripts/search.js "electric vehicles" --headlines --country de > de_headlines.json

# Compare volume
# jq '.totalResults' *_headlines.json
```


path: scripts/sources.js
content: #!/usr/bin/env node

const https = require('https');
const fs = require('fs');
const path = require('path');

// Load environment variables
function loadEnv() {
  const envPath = path.join(process.env.HOME, '.openclaw', '.env');
  if (fs.existsSync(envPath)) {
    const envContent = fs.readFileSync(envPath, 'utf-8');
    envContent.split('\n').forEach(line => {
      const match = line.match(/^([^#=]+)=(.*)$/);
      if (match) {
        process.env[match[1].trim()] = match[2].trim();
      }
    });
  }
}

loadEnv();

const [REDACTED_SECRET];

function showUsage() {
  console.log(`
Usage: node sources.js [options]

List available news sources from NewsAPI.

Options:
  --country CODE     Filter by 2-letter country code (e.g., us, gb, de)
  --category CAT     Filter by category: business, entertainment, general, health, science, sports, technology
  --lang CODE        Filter by 2-letter language code (e.g., en, es, de)
  --json             Output raw JSON instead of formatted list

Examples:
  node sources.js
  node sources.js --country us
  node sources.js --category business --lang en
  node sources.js --json > sources.json
`);
}

function parseArgs(args) {
  const options = {
    country: null,
    category: null,
    language: null,
    json: false
  };

  for (let i = 0; i < args.length; i++) {
    const arg = args[i];

    if (arg === '--help' || arg === '-h') {
      showUsage();
      process.exit(0);
    } else if (arg === '--country' && args[i + 1]) {
      options.country = args[++i];
    } else if (arg === '--category' && args[i + 1]) {
      options.category = args[++i];
    } else if (arg === '--lang' && args[i + 1]) {
      options.language = args[++i];
    } else if (arg === '--json') {
      options.json = true;
    }
  }

  return options;
}

function makeRequest(url) {
  return new Promise((resolve, reject) => {
    const options = new URL(url);
    const requestOptions = {
      hostname: options.hostname,
      path: options.pathname + options.search,
      method: 'GET',
      headers: {
        'User-Agent': 'NewsAPISearch/1.0 (Research Tool)'
      }
    };

    const req = https.request(requestOptions, (res) => {
      let data = '';
      res.on('data', chunk => data += chunk);
      res.on('end', () => {
        try {
          resolve(JSON.parse(data));
        } catch (e) {
          reject(new Error('Invalid JSON response'));
        }
      });
    });
    req.on('error', reject);
    req.setTimeout(30000, () => {
      req.destroy();
      reject(new Error('Request timeout'));
    });
    req.end();
  });
}

async function getSources(filters = {}) {
  if (!API_KEY) {
    throw new Error('NEWSAPI_KEY not found in environment. Add it to ~/.openclaw/.env');
  }

  const params = new URLSearchParams({
    [REDACTED_SECRET]
  });

  if (filters.country) {
    params.append('country', filters.country);
  }
  if (filters.category) {
    params.append('category', filters.category);
  }
  if (filters.language) {
    params.append('language', filters.language);
  }

  const url = `https://newsapi.org/v2/top-headlines/sources?${params.toString()}`;

  try {
    const data = await makeRequest(url);

    if (data.status === 'error') {
      throw new Error(`NewsAPI error: ${data.message || 'Unknown error'}`);
    }

    return data.sources || [];
  } catch (error) {
    throw new Error(`Failed to fetch sources: ${error.message}`);
  }
}

function formatSources(sources) {
  // Group by country
  const byCountry = {};
  sources.forEach(source => {
    const country = source.country?.toUpperCase() || 'Unknown';
    if (!byCountry[country]) {
      byCountry[country] = [];
    }
    byCountry[country].push(source);
  });

  // Sort countries
  const sortedCountries = Object.keys(byCountry).sort();

  let output = `Found ${sources.length} sources\n`;
  output += '='.repeat(50) + '\n\n';

  for (const country of sortedCountries) {
    output += `\n${country} (${byCountry[country].length} sources)\n`;
    output += '-'.repeat(40) + '\n';
    
    // Sort sources alphabetically
    const sortedSources = byCountry[country].sort((a, b) => a.name.localeCompare(b.name));
    
    for (const source of sortedSources) {
      output += `  ${source.id}\n`;
      output += `    Name: ${source.name}\n`;
      output += `    Category: ${source.category || 'N/A'}\n`;
      output += `    Language: ${source.language || 'N/A'}\n`;
      output += `    URL: ${source.url}\n\n`;
    }
  }

  return output;
}

async function main() {
  const args = process.argv.slice(2);

  if (args.includes('--help') || args.includes('-h')) {
    showUsage();
    process.exit(0);
  }

  const options = parseArgs(args);

  if (!API_KEY) {
    console.error('Error: NEWSAPI_KEY not found');
    console.error('Add NEWSAPI_KEY=your_key to ~/.openclaw/.env');
    process.exit(1);
  }

  try {
    const sources = await getSources({
      country: options.country,
      category: options.category,
      language: options.language
    });

    if (options.json) {
      console.log(JSON.stringify(sources, null, 2));
    } else {
      console.log(formatSources(sources));
    }
  } catch (error) {
    console.error(`Error: ${error.message}`);
    process.exit(1);
  }
}

if (require.main === module) {
  main();
}

module.exports = { getSources, makeRequest };


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)