Skill: grenzlinie/semantic-scholar (version 1.0.0)

--- name: semantic-scholar description: Search, retrieve, and organize scholarly metadata with the Semantic Scholar APIs. Use when Codex needs to find papers or authors, build paper sets from complex queries, fetch records in batch by IDs, get related-paper recommendations from seed papers, or decide between Graph API, Recommendations API, and Datasets API workflows for Semantic Scholar. --- # Semantic Scholar ## Overview Choose the right Semantic Scholar API workflow before writing code or issuing requests. Prefer small, field-scoped online calls for interactive search, `paper/search/bulk` for large retrieval jobs, recommendations when the user already has seed papers, and datasets only for offline or release-based data pulls. ## Workflow Decision Tree Start by classifying the task: - Use the Graph API for live paper or author lookup, metadata retrieval, query refinement, and batch fetches by known IDs. - Use the Recommendations API when the user already has one or more relevant papers and wants similar or related work. - Use the Datasets API when the user needs offline snapshots, release-to-release diffs, or corpus-scale ingestion rather than interactive search. Then choose the endpoint pattern: - Use `paper/search` for normal interactive search, smaller result sets, ranking, and iterative query tuning. - Use `paper/search/bulk` for large result collection; it uses continuation-token pagination and is the default for broad literature harvesting. - Use `paper/batch` or `author/batch` when the user already has IDs and wants metadata efficiently. - Use `author/search` for author discovery by name or affiliation-like clues. - Use recommendations for "papers like this one" workflows. ## Operating Rules - Request only the fields needed for the task. Semantic Scholar explicitly supports field projection; smaller field lists are faster and less error-prone. - Prefer API key authentication via `SEMANTIC_SCHOLAR_API_KEY` when available, especially for repeated or larger jobs. - Handle pagination explicitly. `paper/search` and `author/search` are interactive search flows; `paper/search/bulk` uses continuation tokens. - Add retry and backoff for `429` and transient `5xx` responses. - Preserve raw results before flattening or post-processing them. - For broad discovery, write Boolean-rich queries instead of a single brittle phrase. Use exact phrases only when the user asks for them. - Do not route normal search tasks to Datasets API. Use datasets only when the user truly needs offline release files or diffs. ## Typical Workflows ### Search papers interactively Use this for "find papers about X", "search by title keywords", or "filter by year/citations/open access". - Start with `paper/search` if the user expects inspection and refinement. - Keep `fields` minimal. - If the search must collect many records, switch to `paper/search/bulk`. - Read [references/query-recipes.md](references/query-recipes.md) for query patterns. - Read [references/graph-api.md](references/graph-api.md) for endpoint details. ### Harvest a broad paper set Use this for literature review corpora, screening spreadsheets, or downstream ranking. - Prefer `scripts/semantic_scholar_bulk_search.py`. - Save raw output to JSONL and only then export CSV if the user needs tabular review. - Expose query, year filter, sort, and field selection as parameters instead of hardcoding them. ### Fetch by known IDs Use `paper/batch` or `author/batch` when IDs are already known from previous steps or user input. - Batch fetch is usually better than repeated single-record lookups. - Ask for only the fields required for the analysis or export. ### Expand from seed papers Use recommendations when the user says things like "find papers similar to this", "expand from these seed papers", or "build a related-work set". - Use the Recommendations API instead of trying to approximate similarity with a new keyword query. - Keep the seed-paper IDs and result set separate from keyword-search results so provenance stays clear. - Read [references/recommendations-api.md](references/recommendations-api.md). ### Pull datasets or release diffs Use the Datasets API only for offline ingestion or change tracking between releases. - Read [references/datasets-api.md](references/datasets-api.md). - Confirm storage expectations before downloading large files. - Document the exact release identifiers used in the workflow. ## References - Read [references/api-overview.md](references/api-overview.md) first when deciding which API family to use. - Read [references/graph-api.md](references/graph-api.md) for paper and author search, batch, pagination, and field projection. - Read [references/recommendations-api.md](references/recommendations-api.md) for related-paper workflows. - Read [references/datasets-api.md](references/datasets-api.md) for release, download, and diff workflows. - Read [references/query-recipes.md](references/query-recipes.md) for Boolean query templates and search heuristics. ## Script - Use [scripts/semantic_scholar_bulk_search.py](scripts/semantic_scholar_bulk_search.py) for broad paper harvesting with retries, continuation-token pagination, JSONL output, and optional CSV export. - Use [scripts/paper_batch_fetch.py](scripts/paper_batch_fetch.py) to fetch paper metadata in batches from known paper IDs. - Use [scripts/author_batch_fetch.py](scripts/author_batch_fetch.py) to fetch author metadata in batches from known author IDs. - Use [scripts/recommend_papers.py](scripts/recommend_papers.py) to retrieve recommendation results from seed paper IDs. - Use [scripts/smoke_test_live.sh](scripts/smoke_test_live.sh) to run one live end-to-end smoke test across all four API scripts.

path: scripts/semantic_scholar_bulk_search.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-

"""
Semantic Scholar bulk search script

Purpose
-------
Batch search papers from the Semantic Scholar Academic Graph API using:
    GET /graph/v1/paper/search/bulk

This script is suitable for literature retrieval tasks such as:
- Materials Science Multi Agent System
- agentic AI for materials discovery
- multi-agent systems in computational materials science

Features
--------
1. Uses the bulk search endpoint recommended for large-scale retrieval
2. Automatically handles token-based pagination
3. Saves raw results to JSONL (one JSON object per line)
4. Optionally exports a flattened CSV
5. Supports API key via environment variable
6. Includes retry logic for rate limits and transient server errors
7. Includes example command-line usages

Install
-------
pip install requests pandas

API key (recommended)
---------------------
export SEMANTIC_SCHOLAR_API_KEY="your_api_key"

Basic usage
-----------
python semantic_scholar_bulk_search.py

Example usages
--------------
1) Use default query:
python semantic_scholar_bulk_search.py

2) Search exact phrase:
python semantic_scholar_bulk_search.py \
    --query '"Materials Science Multi Agent System"'

3) A more realistic literature query:
python semantic_scholar_bulk_search.py \
    --query '("materials science" | "materials discovery" | "computational materials") + ("multi-agent" | "multi agent" | agentic | "autonomous agent" | "llm agent")'

4) Only search papers from 2020 onward:
python semantic_scholar_bulk_search.py \
    --year 2020-

5) Sort by citation count descending:
python semantic_scholar_bulk_search.py \
    --sort citationCount:desc

6) Sort by publication date descending:
python semantic_scholar_bulk_search.py \
    --sort publicationDate:desc

7) Only keep papers with at least 10 citations:
python semantic_scholar_bulk_search.py \
    --min-citation-count 10

8) Only retrieve papers with open-access PDF:
python semantic_scholar_bulk_search.py \
    --open-access-pdf

9) Limit to first 3 pages for quick testing:
python semantic_scholar_bulk_search.py \
    --max-pages 3

10) Save to custom JSONL file:
python semantic_scholar_bulk_search.py \
    --output my_results.jsonl

11) Export both JSONL and CSV:
python semantic_scholar_bulk_search.py \
    --output my_results.jsonl \
    --csv-output my_results.csv

12) Use a custom field list:
python semantic_scholar_bulk_search.py \
    --fields title,abstract,year,url,authors,citationCount,publicationDate,venue

13) A recommended query for your topic:
python semantic_scholar_bulk_search.py \
    --query '("materials science" | "materials informatics" | "materials discovery") + ("multi-agent system" | "multi-agent" | "agentic" | "llm agent" | "autonomous research agent")' \
    --year 2020- \
    --sort citationCount:desc \
    --csv-output materials_multi_agent.csv

Output
------
- JSONL: raw API results, one paper per line
- CSV: flattened results, convenient for Excel / pandas / screening

Notes
-----
1. Using an API key is optional but strongly recommended.
2. Bulk search uses token-based pagination, not offset/limit pagination.
3. For broad literature surveys, use a query with Boolean operators instead of a single rigid phrase.
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import time
from typing import Any, Dict, List, Optional

import requests

try:
    import pandas as pd
except ImportError:
    pd = None


BASE_URL = "https://api.semanticscholar.org/graph/v1/paper/search/bulk"


def build_headers() -> Dict[str, str]:
    """
    Build request headers. API key is optional but recommended.
    """
    [REDACTED_SECRET]"SEMANTIC_SCHOLAR_API_KEY", "").strip()
    headers = {
        "Accept": "application/json",
        "User-Agent": "semantic-scholar-bulk-search/1.0",
    }
    if [REDACTED_SECRET]"x-api-key"] = api_key
    return headers


def request_with_retry(
    session: requests.Session,
    url: str,
    params: Dict[str, Any],
    headers: Dict[str, str],
    max_retries: int = 6,
    timeout: int = 60,
) -> Dict[str, Any]:
    """
    Send GET request with retry logic for:
    - rate limits (429)
    - transient server errors (5xx)
    - network errors

    Returns parsed JSON on success.
    """
    for attempt in range(max_retries):
        try:
            response = session.get(url, params=params, headers=headers, timeout=timeout)

            if response.status_code == 200:
                return response.json()

            if response.status_code == 429:
                wait_time = min(2 ** attempt, 30)
                print(
                    f"[WARN] HTTP 429 Too Many Requests. Sleeping {wait_time}s before retry...",
                    file=sys.stderr,
                )
                time.sleep(wait_time)
                continue

            if 500 <= response.status_code < 600:
                wait_time = min(2 ** attempt, 30)
                print(
                    f"[WARN] HTTP {response.status_code} server error. Sleeping {wait_time}s before retry...",
                    file=sys.stderr,
                )
                time.sleep(wait_time)
                continue

            raise RuntimeError(
                f"Request failed with HTTP {response.status_code}\nResponse body:\n{response.text}"
            )

        except requests.RequestException as exc:
            wait_time = min(2 ** attempt, 30)
            print(
                f"[WARN] Network/request error: {exc}. Sleeping {wait_time}s before retry...",
                file=sys.stderr,
            )
            time.sleep(wait_time)

    raise RuntimeError("Request failed after maximum retries.")


def flatten_authors(authors: Any) -> str:
    """
    Convert author list to a readable string.
    """
    if not authors:
        return ""
    if not isinstance(authors, list):
        return ""
    names = []
    for author in authors:
        if isinstance(author, dict):
            name = author.get("name")
            if name:
                names.append(str(name))
    return "; ".join(names)


def flatten_open_access_pdf(open_access_pdf: Any) -> str:
    """
    Extract open access PDF URL if available.
    """
    if isinstance(open_access_pdf, dict):
        return str(open_access_pdf.get("url", "") or "")
    return ""


def paper_to_csv_row(paper: Dict[str, Any]) -> Dict[str, Any]:
    """
    Flatten one paper record for CSV export.
    """
    return {
        "paperId": paper.get("paperId"),
        "title": paper.get("title"),
        "year": paper.get("year"),
        "publicationDate": paper.get("publicationDate"),
        "citationCount": paper.get("citationCount"),
        "url": paper.get("url"),
        "venue": paper.get("venue"),
        "publicationTypes": json.dumps(paper.get("publicationTypes", []), ensure_ascii=False),
        "authors": flatten_authors(paper.get("authors")),
        "openAccessPdfUrl": flatten_open_access_pdf(paper.get("openAccessPdf")),
        "abstract": paper.get("abstract"),
    }


def export_csv(jsonl_path: str, csv_output: str) -> None:
    """
    Export JSONL results to CSV.
    """
    if pd is None:
        raise RuntimeError(
            "CSV export requires pandas. Please install it with: pip install pandas"
        )

    rows: List[Dict[str, Any]] = []
    with open(jsonl_path, "r", encoding="utf-8") as fin:
        for line in fin:
            line = line.strip()
            if not line:
                continue
            paper = json.loads(line)
            rows.append(paper_to_csv_row(paper))

    df = pd.DataFrame(rows)
    df.to_csv(csv_output, index=False, encoding="utf-8-sig")
    print(f"[DONE] CSV exported to: {csv_output}")


def bulk_search_papers(
    query: str,
    output_path: str,
    fields: str,
    year: Optional[str] = None,
    sort: Optional[str] = None,
    min_citation_count: Optional[int] = None,
    open_access_pdf: Optional[bool] = None,
    publication_types: Optional[str] = None,
    fields_of_study: Optional[str] = None,
    venue: Optional[str] = None,
    max_pages: Optional[int] = None,
    sleep_seconds: float = 1.0,
) -> int:
    """
    Perform bulk search with token-based pagination.
    Save each paper record as one JSON line.

    Returns
    -------
    int
        Number of retrieved papers written to disk.
    """
    headers = build_headers()
    session = requests.Session()

    params: Dict[str, Any] = {
        "query": query,
        "fields": fields,
    }

    if year:
        params["year"] = year
    if sort:
        params["sort"] = sort
    if min_citation_count is not None:
        params["minCitationCount"] = min_citation_count
    if open_access_pdf is not None:
        params["openAccessPdf"] = str(open_access_pdf).lower()
    if publication_types:
        params["publicationTypes"] = publication_types
    if fields_of_study:
        params["fieldsOfStudy"] = fields_of_study
    if venue:
        params["venue"] = venue

    total_written = 0
    page_count = 0

    with open(output_path, "w", encoding="utf-8") as fout:
        while True:
            response_data = request_with_retry(
                session=session,
                url=BASE_URL,
                params=params,
                headers=headers,
            )

            if page_count == 0:
                estimated_total = response_data.get("total")
                print(f"[INFO] Estimated total matched papers: {estimated_total}")

            data = response_data.get("data", [])
            if not data:
                print("[INFO] No more data returned.")
                break

            for paper in data:
                fout.write(json.dumps(paper, ensure_ascii=False) + "\n")

            total_written += len(data)
            page_count += 1
            print(
                f"[INFO] Page {page_count}: wrote {len(data)} papers, total written = {total_written}"
            )

            next_token = response_data.get("token")
            if not next_token:
                print("[INFO] No continuation token found. Finished.")
                break

            if max_pages is not None and page_count >= max_pages:
                print(f"[INFO] Reached max_pages={max_pages}. Stopping early.")
                break

            params["token"] = next_token

            if sleep_seconds > 0:
                time.sleep(sleep_seconds)

    print(f"[DONE] JSONL saved to: {output_path}")
    print(f"[DONE] Total papers written: {total_written}")
    return total_written


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Batch search papers from Semantic Scholar Academic Graph API."
    )

    parser.add_argument(
        "--query",
        type=str,
        default='"Materials Science Multi Agent System"',
        help='Search query. Default: \'"Materials Science Multi Agent System"\'',
    )
    parser.add_argument(
        "--output",
        type=str,
        default="materials_science_multi_agent_system.jsonl",
        help="Output JSONL file path.",
    )
    parser.add_argument(
        "--csv-output",
        type=str,
        default=None,
        help="Optional CSV output path.",
    )
    parser.add_argument(
        "--fields",
        type=str,
        default="title,abstract,year,url,authors,citationCount,publicationDate,publicationTypes,openAccessPdf,venue",
        help="Comma-separated fields to request.",
    )
    parser.add_argument(
        "--year",
        type=str,
        default=None,
        help='Year filter, e.g. "2020-" or "2020-2025".',
    )
    parser.add_argument(
        "--sort",
        type=str,
        default=None,
        help='Optional sort, e.g. "citationCount:desc" or "publicationDate:desc".',
    )
    parser.add_argument(
        "--min-citation-count",
        type=int,
        default=None,
        help="Minimum citation count filter.",
    )
    parser.add_argument(
        "--open-access-pdf",
        action="store_true",
        help="Only retrieve papers with open access PDF.",
    )
    parser.add_argument(
        "--publication-types",
        type=str,
        default=None,
        help='Filter by publication types, e.g. "JournalArticle" or "Review".',
    )
    parser.add_argument(
        "--fields-of-study",
        type=str,
        default=None,
        help='Filter by field of study, e.g. "Materials Science".',
    )
    parser.add_argument(
        "--venue",
        type=str,
        default=None,
        help='Filter by venue name.',
    )
    parser.add_argument(
        "--max-pages",
        type=int,
        default=None,
        help="Maximum number of pages to retrieve, useful for quick testing.",
    )
    parser.add_argument(
        "--sleep-seconds",
        type=float,
        default=1.0,
        help="Sleep interval between page requests. Default: 1.0",
    )

    return parser.parse_args()


def main() -> None:
    args = parse_args()

    total_written = bulk_search_papers(
        query=args.query,
        output_path=args.output,
        fields=args.fields,
        year=args.year,
        sort=args.sort,
        min_citation_count=args.min_citation_count,
        open_access_pdf=True if args.open_access_pdf else None,
        publication_types=args.publication_types,
        fields_of_study=args.fields_of_study,
        venue=args.venue,
        max_pages=args.max_pages,
        sleep_seconds=args.sleep_seconds,
    )

    if args.csv_output:
        if total_written == 0:
            print("[INFO] No results found, skipping CSV export.")
        else:
            export_csv(args.output, args.csv_output)


if __name__ == "__main__":
    main()


path: scripts/recommend_papers.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-

"""
Semantic Scholar paper recommendations script.

Purpose
-------
Retrieve recommended papers from seed paper IDs using:
    POST /recommendations/v1/papers
"""

from __future__ import annotations

import argparse
import csv
import json
import os
import sys
import time
from typing import Any, Dict, List, Optional

import requests


BASE_URL = "https://api.semanticscholar.org/recommendations/v1/papers"


def build_headers() -> Dict[str, str]:
    [REDACTED_SECRET]"SEMANTIC_SCHOLAR_API_KEY", "").strip()
    headers = {
        "Accept": "application/json",
        "Content-Type": "application/json",
        "User-Agent": "semantic-scholar-recommendations/1.0",
    }
    if [REDACTED_SECRET]"x-api-key"] = api_key
    return headers


def request_with_retry(
    session: requests.Session,
    url: str,
    payload: Dict[str, Any],
    params: Dict[str, Any],
    headers: Dict[str, str],
    max_retries: int = 6,
    timeout: int = 60,
) -> Dict[str, Any]:
    for attempt in range(max_retries):
        try:
            response = session.post(
                url,
                params=params,
                headers=headers,
                json=payload,
                timeout=timeout,
            )
            if response.status_code == 200:
                data = response.json()
                if not isinstance(data, dict):
                    raise RuntimeError(f"Unexpected response type: {type(data)!r}")
                return data

            if response.status_code == 429:
                wait_time = min(2 ** attempt, 30)
                print(
                    f"[WARN] HTTP 429 Too Many Requests. Sleeping {wait_time}s before retry...",
                    file=sys.stderr,
                )
                time.sleep(wait_time)
                continue

            if 500 <= response.status_code < 600:
                wait_time = min(2 ** attempt, 30)
                print(
                    f"[WARN] HTTP {response.status_code} server error. Sleeping {wait_time}s before retry...",
                    file=sys.stderr,
                )
                time.sleep(wait_time)
                continue

            raise RuntimeError(
                f"Request failed with HTTP {response.status_code}\nResponse body:\n{response.text}"
            )
        except requests.RequestException as exc:
            wait_time = min(2 ** attempt, 30)
            print(
                f"[WARN] Network/request error: {exc}. Sleeping {wait_time}s before retry...",
                file=sys.stderr,
            )
            time.sleep(wait_time)

    raise RuntimeError("Request failed after maximum retries.")


def parse_id_list(raw: Optional[str]) -> List[str]:
    if not raw:
        return []
    return [value.strip() for value in raw.split(",") if value.strip()]


def flatten_authors(authors: Any) -> str:
    if not isinstance(authors, list):
        return ""
    return "; ".join(
        str(author.get("name"))
        for author in authors
        if isinstance(author, dict) and author.get("name")
    )


def flatten_open_access_pdf(open_access_pdf: Any) -> str:
    if isinstance(open_access_pdf, dict):
        return str(open_access_pdf.get("url", "") or "")
    return ""


def paper_to_csv_row(paper: Dict[str, Any]) -> Dict[str, Any]:
    return {
        "paperId": paper.get("paperId"),
        "title": paper.get("title"),
        "year": paper.get("year"),
        "publicationDate": paper.get("publicationDate"),
        "citationCount": paper.get("citationCount"),
        "url": paper.get("url"),
        "venue": paper.get("venue"),
        "authors": flatten_authors(paper.get("authors")),
        "openAccessPdfUrl": flatten_open_access_pdf(paper.get("openAccessPdf")),
        "abstract": paper.get("abstract"),
    }


def export_csv(jsonl_path: str, csv_output: str) -> None:
    rows: List[Dict[str, Any]] = []
    with open(jsonl_path, "r", encoding="utf-8") as fin:
        for line in fin:
            line = line.strip()
            if not line:
                continue
            payload = json.loads(line)
            rows.append(paper_to_csv_row(payload["paper"]))

    if not rows:
        print("[INFO] No rows available for CSV export.")
        return

    fieldnames = list(rows[0].keys())
    with open(csv_output, "w", encoding="utf-8-sig", newline="") as fout:
        writer = csv.DictWriter(fout, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(rows)
    print(f"[DONE] CSV exported to: {csv_output}")


def recommend_papers(
    positive_ids: List[str],
    output_path: str,
    fields: str,
    limit: int = 20,
    negative_ids: Optional[List[str]] = None,
    pool_from: Optional[str] = None,
) -> int:
    if not positive_ids:
        raise ValueError("At least one positive paper ID is required.")

    headers = build_headers()
    session = requests.Session()
    params = {"fields": fields}
    payload: Dict[str, Any] = {
        "positivePaperIds": positive_ids,
    }
    params: Dict[str, Any] = {"fields": fields, "limit": limit}
    if negative_ids:
        payload["negativePaperIds"] = negative_ids
    if pool_from:
        payload["poolFrom"] = pool_from

    response_data = request_with_retry(
        session=session,
        url=BASE_URL,
        payload=payload,
        params=params,
        headers=headers,
    )
    recommendations = response_data.get("recommendedPapers", [])
    if not isinstance(recommendations, list):
        raise RuntimeError("Unexpected recommendations payload shape.")

    with open(output_path, "w", encoding="utf-8") as fout:
        for rank, paper in enumerate(recommendations, start=1):
            fout.write(
                json.dumps(
                    {
                        "rank": rank,
                        "seedPaperIds": positive_ids,
                        "negativePaperIds": negative_ids or [],
                        "paper": paper,
                    },
                    ensure_ascii=False,
                )
                + "\n"
            )

    print(f"[DONE] JSONL saved to: {output_path}")
    print(f"[DONE] Total recommendations written: {len(recommendations)}")
    return len(recommendations)


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Get paper recommendations from Semantic Scholar seed papers."
    )
    parser.add_argument(
        "--positive-ids",
        type=str,
        required=True,
        help="Comma-separated positive seed paper IDs.",
    )
    parser.add_argument(
        "--negative-ids",
        type=str,
        default=None,
        help="Optional comma-separated negative seed paper IDs.",
    )
    parser.add_argument(
        "--limit",
        type=int,
        default=20,
        help="Maximum number of recommendations to request. Default: 20",
    )
    parser.add_argument(
        "--pool-from",
        type=str,
        default=None,
        help="Optional pool selector supported by the recommendations API.",
    )
    parser.add_argument(
        "--output",
        type=str,
        default="paper_recommendations.jsonl",
        help="Output JSONL file path.",
    )
    parser.add_argument(
        "--csv-output",
        type=str,
        default=None,
        help="Optional CSV output path.",
    )
    parser.add_argument(
        "--fields",
        type=str,
        default="title,abstract,year,url,authors,citationCount,publicationDate,openAccessPdf,venue",
        help="Comma-separated fields to request for recommended papers.",
    )
    args = parser.parse_args()
    if args.limit < 1:
        parser.error("--limit must be at least 1.")
    return args


def main() -> None:
    args = parse_args()
    total_written = recommend_papers(
        positive_ids=parse_id_list(args.positive_ids),
        negative_ids=parse_id_list(args.negative_ids),
        output_path=args.output,
        fields=args.fields,
        limit=args.limit,
        pool_from=args.pool_from,
    )
    if args.csv_output and total_written > 0:
        export_csv(args.output, args.csv_output)


if __name__ == "__main__":
    main()


path: scripts/smoke_test_live.sh
content: #!/usr/bin/env bash
set -euo pipefail

# Live smoke test for Semantic Scholar scripts.
# It runs:
# 1) bulk search
# 2) paper batch fetch (using IDs from step 1)
# 3) author batch fetch (using IDs from step 1)
# 4) recommendations (using one paper ID from step 1)

ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
OUT_DIR="${1:-/tmp/semantic_scholar_smoke_$(date +%Y%m%d_%H%M%S)}"

if ! command -v uv >/dev/null 2>&1; then
  echo "[ERROR] 'uv' is required but was not found in PATH." >&2
  exit 1
fi

mkdir -p "${OUT_DIR}"

BULK_JSONL="${OUT_DIR}/bulk.jsonl"
PAPER_IDS_FILE="${OUT_DIR}/paper_ids.txt"
AUTHOR_IDS_FILE="${OUT_DIR}/author_ids.txt"
PAPER_BATCH_JSONL="${OUT_DIR}/paper_batch.jsonl"
PAPER_BATCH_CSV="${OUT_DIR}/paper_batch.csv"
AUTHOR_BATCH_JSONL="${OUT_DIR}/author_batch.jsonl"
AUTHOR_BATCH_CSV="${OUT_DIR}/author_batch.csv"
RECO_JSONL="${OUT_DIR}/recommendations.jsonl"
RECO_CSV="${OUT_DIR}/recommendations.csv"

echo "[STEP] 1/4 bulk search"
uv run python "${ROOT_DIR}/semantic_scholar_bulk_search.py" \
  --query "graph neural network" \
  --fields "paperId,title,year,authors,url,citationCount" \
  --max-pages 1 \
  --sleep-seconds 0 \
  --output "${BULK_JSONL}"

echo "[STEP] extract paper/author IDs from bulk results"
uv run python - "${BULK_JSONL}" "${PAPER_IDS_FILE}" "${AUTHOR_IDS_FILE}" <<'PY'
import json
import sys
from pathlib import Path

bulk_path = Path(sys.argv[1])
paper_ids_path = Path(sys.argv[2])
author_ids_path = Path(sys.argv[3])

paper_ids = []
author_ids = []
seen_paper = set()
seen_author = set()

with bulk_path.open("r", encoding="utf-8") as fin:
    for line in fin:
        row = json.loads(line)
        paper_id = row.get("paperId")
        if paper_id and paper_id not in seen_paper:
            seen_paper.add(paper_id)
            paper_ids.append(paper_id)
        for author in row.get("authors") or []:
            if not isinstance(author, dict):
                continue
            author_id = author.get("authorId")
            if author_id and author_id not in seen_author:
                seen_author.add(author_id)
                author_ids.append(author_id)
        if len(paper_ids) >= 3 and len(author_ids) >= 3:
            break

if len(paper_ids) < 1 or len(author_ids) < 1:
    raise SystemExit("Not enough IDs extracted from bulk results.")

paper_ids_path.write_text("\n".join(paper_ids[:3]) + "\n", encoding="utf-8")
author_ids_path.write_text("\n".join(author_ids[:3]) + "\n", encoding="utf-8")
print("paper_ids", paper_ids[:3])
print("author_ids", author_ids[:3])
PY

echo "[STEP] 2/4 paper batch fetch"
uv run python "${ROOT_DIR}/paper_batch_fetch.py" \
  --ids-file "${PAPER_IDS_FILE}" \
  --fields "paperId,title,year,authors,url,citationCount,publicationDate,venue" \
  --output "${PAPER_BATCH_JSONL}" \
  --csv-output "${PAPER_BATCH_CSV}" \
  --chunk-size 2 \
  --sleep-seconds 0

echo "[STEP] 3/4 author batch fetch"
uv run python "${ROOT_DIR}/author_batch_fetch.py" \
  --ids-file "${AUTHOR_IDS_FILE}" \
  --output "${AUTHOR_BATCH_JSONL}" \
  --csv-output "${AUTHOR_BATCH_CSV}" \
  --chunk-size 2 \
  --sleep-seconds 0

echo "[STEP] 4/4 recommendations"
RECO_OK=0
while IFS= read -r SEED_ID; do
  SEED_ID="$(echo "${SEED_ID}" | tr -d '\r\n')"
  if [[ -z "${SEED_ID}" ]]; then
    continue
  fi
  echo "[INFO] Trying recommendation seed: ${SEED_ID}"
  uv run python "${ROOT_DIR}/recommend_papers.py" \
    --positive-ids "${SEED_ID}" \
    --limit 5 \
    --fields "paperId,title,year,authors,url,citationCount,publicationDate,venue" \
    --output "${RECO_JSONL}" \
    --csv-output "${RECO_CSV}" || continue

  RECO_COUNT="$(wc -l < "${RECO_JSONL}" | tr -d ' ')"
  if [[ "${RECO_COUNT}" -gt 0 ]]; then
    RECO_OK=1
    break
  fi
done < "${PAPER_IDS_FILE}"

if [[ "${RECO_OK}" -eq 0 ]]; then
  echo "[WARN] Recommendations endpoint returned 0 rows for tested seeds. Request succeeded, but no recommendations were returned."
fi

echo "[DONE] Live smoke test completed."
echo "[DONE] Output directory: ${OUT_DIR}"


path: scripts/paper_batch_fetch.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-

"""
Semantic Scholar paper batch fetch script.

Purpose
-------
Fetch paper metadata in batches using:
    POST /graph/v1/paper/batch

Features
--------
1. Reads paper IDs from CLI or file
2. Sends batched POST requests with retry/backoff
3. Saves raw results to JSONL
4. Optionally exports a flattened CSV
5. Supports API key via environment variable
"""

from __future__ import annotations

import argparse
import csv
import json
import os
import sys
import time
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional

import requests


BASE_URL = "https://api.semanticscholar.org/graph/v1/paper/batch"


def build_headers() -> Dict[str, str]:
    [REDACTED_SECRET]"SEMANTIC_SCHOLAR_API_KEY", "").strip()
    headers = {
        "Accept": "application/json",
        "Content-Type": "application/json",
        "User-Agent": "semantic-scholar-paper-batch/1.0",
    }
    if [REDACTED_SECRET]"x-api-key"] = api_key
    return headers


def request_with_retry(
    session: requests.Session,
    url: str,
    payload: Dict[str, Any],
    params: Dict[str, Any],
    headers: Dict[str, str],
    max_retries: int = 6,
    timeout: int = 60,
) -> List[Dict[str, Any]]:
    for attempt in range(max_retries):
        try:
            response = session.post(
                url,
                params=params,
                headers=headers,
                json=payload,
                timeout=timeout,
            )

            if response.status_code == 200:
                data = response.json()
                if not isinstance(data, list):
                    raise RuntimeError(f"Unexpected response type: {type(data)!r}")
                return data

            if response.status_code == 429:
                wait_time = min(2 ** attempt, 30)
                print(
                    f"[WARN] HTTP 429 Too Many Requests. Sleeping {wait_time}s before retry...",
                    file=sys.stderr,
                )
                time.sleep(wait_time)
                continue

            if 500 <= response.status_code < 600:
                wait_time = min(2 ** attempt, 30)
                print(
                    f"[WARN] HTTP {response.status_code} server error. Sleeping {wait_time}s before retry...",
                    file=sys.stderr,
                )
                time.sleep(wait_time)
                continue

            raise RuntimeError(
                f"Request failed with HTTP {response.status_code}\nResponse body:\n{response.text}"
            )
        except requests.RequestException as exc:
            wait_time = min(2 ** attempt, 30)
            print(
                f"[WARN] Network/request error: {exc}. Sleeping {wait_time}s before retry...",
                file=sys.stderr,
            )
            time.sleep(wait_time)

    raise RuntimeError("Request failed after maximum retries.")


def load_ids_from_file(path: str) -> List[str]:
    ids: List[str] = []
    with open(path, "r", encoding="utf-8") as fin:
        for line in fin:
            value = line.strip()
            if value:
                ids.append(value)
    return ids


def parse_id_args(ids_arg: Optional[str], ids_file: Optional[str]) -> List[str]:
    ids: List[str] = []
    if ids_arg:
        ids.extend([value.strip() for value in ids_arg.split(",") if value.strip()])
    if ids_file:
        ids.extend(load_ids_from_file(ids_file))

    # Deduplicate while preserving order.
    seen = set()
    deduped: List[str] = []
    for item in ids:
        if item in seen:
            continue
        seen.add(item)
        deduped.append(item)
    return deduped


def chunked(items: List[str], chunk_size: int) -> Iterable[List[str]]:
    for index in range(0, len(items), chunk_size):
        yield items[index : index + chunk_size]


def flatten_authors(authors: Any) -> str:
    if not isinstance(authors, list):
        return ""
    return "; ".join(
        str(author.get("name"))
        for author in authors
        if isinstance(author, dict) and author.get("name")
    )


def flatten_open_access_pdf(open_access_pdf: Any) -> str:
    if isinstance(open_access_pdf, dict):
        return str(open_access_pdf.get("url", "") or "")
    return ""


def paper_to_csv_row(paper: Dict[str, Any]) -> Dict[str, Any]:
    return {
        "paperId": paper.get("paperId"),
        "title": paper.get("title"),
        "year": paper.get("year"),
        "publicationDate": paper.get("publicationDate"),
        "citationCount": paper.get("citationCount"),
        "influentialCitationCount": paper.get("influentialCitationCount"),
        "url": paper.get("url"),
        "venue": paper.get("venue"),
        "publicationTypes": json.dumps(paper.get("publicationTypes", []), ensure_ascii=False),
        "authors": flatten_authors(paper.get("authors")),
        "openAccessPdfUrl": flatten_open_access_pdf(paper.get("openAccessPdf")),
        "abstract": paper.get("abstract"),
    }


def export_csv(jsonl_path: str, csv_output: str) -> None:
    rows: List[Dict[str, Any]] = []
    with open(jsonl_path, "r", encoding="utf-8") as fin:
        for line in fin:
            line = line.strip()
            if not line:
                continue
            rows.append(paper_to_csv_row(json.loads(line)))

    if not rows:
        print("[INFO] No rows available for CSV export.")
        return

    fieldnames = list(rows[0].keys())
    with open(csv_output, "w", encoding="utf-8-sig", newline="") as fout:
        writer = csv.DictWriter(fout, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(rows)
    print(f"[DONE] CSV exported to: {csv_output}")


def fetch_papers(
    ids: List[str],
    output_path: str,
    fields: str,
    chunk_size: int = 500,
    sleep_seconds: float = 0.5,
) -> int:
    if not ids:
        raise ValueError("No paper IDs provided.")

    headers = build_headers()
    session = requests.Session()
    params = {"fields": fields}

    total_written = 0
    with open(output_path, "w", encoding="utf-8") as fout:
        for index, batch_ids in enumerate(chunked(ids, chunk_size), start=1):
            payload = {"ids": batch_ids}
            records = request_with_retry(
                session=session,
                url=BASE_URL,
                payload=payload,
                params=params,
                headers=headers,
            )

            written_this_batch = 0
            for record in records:
                if not record:
                    continue
                fout.write(json.dumps(record, ensure_ascii=False) + "\n")
                written_this_batch += 1

            total_written += written_this_batch
            print(
                f"[INFO] Batch {index}: requested {len(batch_ids)} IDs, wrote {written_this_batch} records, total written = {total_written}"
            )
            if sleep_seconds > 0:
                time.sleep(sleep_seconds)

    print(f"[DONE] JSONL saved to: {output_path}")
    print(f"[DONE] Total records written: {total_written}")
    return total_written


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Batch fetch paper metadata from Semantic Scholar."
    )
    parser.add_argument(
        "--ids",
        type=str,
        default=None,
        help="Comma-separated paper IDs.",
    )
    parser.add_argument(
        "--ids-file",
        type=str,
        default=None,
        help="Text file with one paper ID per line.",
    )
    parser.add_argument(
        "--output",
        type=str,
        default="paper_batch_results.jsonl",
        help="Output JSONL file path.",
    )
    parser.add_argument(
        "--csv-output",
        type=str,
        default=None,
        help="Optional CSV output path.",
    )
    parser.add_argument(
        "--fields",
        type=str,
        default="title,abstract,year,url,authors,citationCount,influentialCitationCount,publicationDate,publicationTypes,openAccessPdf,venue",
        help="Comma-separated fields to request.",
    )
    parser.add_argument(
        "--chunk-size",
        type=int,
        default=500,
        help="Number of IDs per request. Default: 500",
    )
    parser.add_argument(
        "--sleep-seconds",
        type=float,
        default=0.5,
        help="Sleep interval between requests. Default: 0.5",
    )
    args = parser.parse_args()

    if not args.ids and not args.ids_file:
        parser.error("Provide --ids or --ids-file.")
    if args.chunk_size < 1:
        parser.error("--chunk-size must be at least 1.")
    return args


def main() -> None:
    args = parse_args()
    ids = parse_id_args(args.ids, args.ids_file)
    total_written = fetch_papers(
        ids=ids,
        output_path=args.output,
        fields=args.fields,
        chunk_size=args.chunk_size,
        sleep_seconds=args.sleep_seconds,
    )
    if args.csv_output and total_written > 0:
        export_csv(args.output, args.csv_output)


if __name__ == "__main__":
    main()


path: scripts/author_batch_fetch.py
content: #!/usr/bin/env python3
# -*- coding: utf-8 -*-

"""
Semantic Scholar author batch fetch script.

Purpose
-------
Fetch author metadata in batches using:
    POST /graph/v1/author/batch
"""

from __future__ import annotations

import argparse
import csv
import json
import os
import sys
import time
from typing import Any, Dict, Iterable, List, Optional

import requests


BASE_URL = "https://api.semanticscholar.org/graph/v1/author/batch"


def build_headers() -> Dict[str, str]:
    [REDACTED_SECRET]"SEMANTIC_SCHOLAR_API_KEY", "").strip()
    headers = {
        "Accept": "application/json",
        "Content-Type": "application/json",
        "User-Agent": "semantic-scholar-author-batch/1.0",
    }
    if [REDACTED_SECRET]"x-api-key"] = api_key
    return headers


def request_with_retry(
    session: requests.Session,
    url: str,
    payload: Dict[str, Any],
    params: Dict[str, Any],
    headers: Dict[str, str],
    max_retries: int = 6,
    timeout: int = 60,
) -> List[Dict[str, Any]]:
    for attempt in range(max_retries):
        try:
            response = session.post(
                url,
                params=params,
                headers=headers,
                json=payload,
                timeout=timeout,
            )
            if response.status_code == 200:
                data = response.json()
                if not isinstance(data, list):
                    raise RuntimeError(f"Unexpected response type: {type(data)!r}")
                return data

            if response.status_code == 429:
                wait_time = min(2 ** attempt, 30)
                print(
                    f"[WARN] HTTP 429 Too Many Requests. Sleeping {wait_time}s before retry...",
                    file=sys.stderr,
                )
                time.sleep(wait_time)
                continue

            if 500 <= response.status_code < 600:
                wait_time = min(2 ** attempt, 30)
                print(
                    f"[WARN] HTTP {response.status_code} server error. Sleeping {wait_time}s before retry...",
                    file=sys.stderr,
                )
                time.sleep(wait_time)
                continue

            raise RuntimeError(
                f"Request failed with HTTP {response.status_code}\nResponse body:\n{response.text}"
            )
        except requests.RequestException as exc:
            wait_time = min(2 ** attempt, 30)
            print(
                f"[WARN] Network/request error: {exc}. Sleeping {wait_time}s before retry...",
                file=sys.stderr,
            )
            time.sleep(wait_time)

    raise RuntimeError("Request failed after maximum retries.")


def load_ids_from_file(path: str) -> List[str]:
    ids: List[str] = []
    with open(path, "r", encoding="utf-8") as fin:
        for line in fin:
            value = line.strip()
            if value:
                ids.append(value)
    return ids


def parse_id_args(ids_arg: Optional[str], ids_file: Optional[str]) -> List[str]:
    ids: List[str] = []
    if ids_arg:
        ids.extend([value.strip() for value in ids_arg.split(",") if value.strip()])
    if ids_file:
        ids.extend(load_ids_from_file(ids_file))

    seen = set()
    deduped: List[str] = []
    for item in ids:
        if item in seen:
            continue
        seen.add(item)
        deduped.append(item)
    return deduped


def chunked(items: List[str], chunk_size: int) -> Iterable[List[str]]:
    for index in range(0, len(items), chunk_size):
        yield items[index : index + chunk_size]


def flatten_aliases(aliases: Any) -> str:
    if not isinstance(aliases, list):
        return ""
    return "; ".join(str(value) for value in aliases if value)


def flatten_papers(papers: Any) -> str:
    if not isinstance(papers, list):
        return ""
    titles: List[str] = []
    for paper in papers:
        if isinstance(paper, dict) and paper.get("title"):
            titles.append(str(paper["title"]))
    return "; ".join(titles)


def author_to_csv_row(author: Dict[str, Any]) -> Dict[str, Any]:
    return {
        "authorId": author.get("authorId"),
        "name": author.get("name"),
        "aliases": flatten_aliases(author.get("aliases")),
        "paperCount": author.get("paperCount"),
        "citationCount": author.get("citationCount"),
        "hIndex": author.get("hIndex"),
        "homepage": author.get("homepage"),
        "url": author.get("url"),
        "affiliations": json.dumps(author.get("affiliations", []), ensure_ascii=False),
        "papers": flatten_papers(author.get("papers")),
    }


def export_csv(jsonl_path: str, csv_output: str) -> None:
    rows: List[Dict[str, Any]] = []
    with open(jsonl_path, "r", encoding="utf-8") as fin:
        for line in fin:
            line = line.strip()
            if not line:
                continue
            rows.append(author_to_csv_row(json.loads(line)))

    if not rows:
        print("[INFO] No rows available for CSV export.")
        return

    fieldnames = list(rows[0].keys())
    with open(csv_output, "w", encoding="utf-8-sig", newline="") as fout:
        writer = csv.DictWriter(fout, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(rows)
    print(f"[DONE] CSV exported to: {csv_output}")


def fetch_authors(
    ids: List[str],
    output_path: str,
    fields: str,
    chunk_size: int = 500,
    sleep_seconds: float = 0.5,
) -> int:
    if not ids:
        raise ValueError("No author IDs provided.")

    headers = build_headers()
    session = requests.Session()
    params = {"fields": fields}

    total_written = 0
    with open(output_path, "w", encoding="utf-8") as fout:
        for index, batch_ids in enumerate(chunked(ids, chunk_size), start=1):
            payload = {"ids": batch_ids}
            records = request_with_retry(
                session=session,
                url=BASE_URL,
                payload=payload,
                params=params,
                headers=headers,
            )

            written_this_batch = 0
            for record in records:
                if not record:
                    continue
                fout.write(json.dumps(record, ensure_ascii=False) + "\n")
                written_this_batch += 1

            total_written += written_this_batch
            print(
                f"[INFO] Batch {index}: requested {len(batch_ids)} IDs, wrote {written_this_batch} records, total written = {total_written}"
            )
            if sleep_seconds > 0:
                time.sleep(sleep_seconds)

    print(f"[DONE] JSONL saved to: {output_path}")
    print(f"[DONE] Total records written: {total_written}")
    return total_written


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(
        description="Batch fetch author metadata from Semantic Scholar."
    )
    parser.add_argument("--ids", type=str, default=None, help="Comma-separated author IDs.")
    parser.add_argument(
        "--ids-file",
        type=str,
        default=None,
        help="Text file with one author ID per line.",
    )
    parser.add_argument(
        "--output",
        type=str,
        default="author_batch_results.jsonl",
        help="Output JSONL file path.",
    )
    parser.add_argument(
        "--csv-output",
        type=str,
        default=None,
        help="Optional CSV output path.",
    )
    parser.add_argument(
        "--fields",
        type=str,
        default="authorId,name,paperCount,citationCount,hIndex,homepage,url,affiliations,papers.title",
        help="Comma-separated fields to request.",
    )
    parser.add_argument(
        "--chunk-size",
        type=int,
        default=500,
        help="Number of IDs per request. Default: 500",
    )
    parser.add_argument(
        "--sleep-seconds",
        type=float,
        default=0.5,
        help="Sleep interval between requests. Default: 0.5",
    )
    args = parser.parse_args()

    if not args.ids and not args.ids_file:
        parser.error("Provide --ids or --ids-file.")
    if args.chunk_size < 1:
        parser.error("--chunk-size must be at least 1.")
    return args


def main() -> None:
    args = parse_args()
    ids = parse_id_args(args.ids, args.ids_file)
    total_written = fetch_authors(
        ids=ids,
        output_path=args.output,
        fields=args.fields,
        chunk_size=args.chunk_size,
        sleep_seconds=args.sleep_seconds,
    )
    if args.csv_output and total_written > 0:
        export_csv(args.output, args.csv_output)


if __name__ == "__main__":
    main()


path: agents/openai.yaml
content: interface:
  display_name: "Semantic Scholar"
  short_description: "Search papers, authors, and dataset releases"
  default_prompt: "Use $semantic-scholar to search Semantic Scholar papers, authors, recommendations, or datasets with the right API workflow."


path: references/api-overview.md
content: # Semantic Scholar API Overview

## API Families

- Graph API: interactive paper and author search, metadata lookup, and batch fetch by IDs.
- Recommendations API: related-paper retrieval from seed paper IDs.
- Datasets API: offline data downloads by release, including incremental diffs between releases.

## Selection Guide

- Choose Graph API when the user wants live search, filtering, ranking, author lookup, or metadata retrieval for a manageable set of records.
- Choose Recommendations API when the user already has relevant papers and wants adjacent literature.
- Choose Datasets API when the user needs corpus-scale offline files or release-to-release changes.

## Cross-Cutting Practices

- Authenticate with `x-api-key` via `SEMANTIC_SCHOLAR_API_KEY` when available.
- Use field projection aggressively. Ask only for fields needed by the task.
- Add retry and backoff around `429` and temporary `5xx` failures.
- Record exact endpoint, parameters, and release identifiers so results are reproducible.
- Preserve raw API output before flattening or filtering it.

## Typical User Requests

- "Find papers about agentic materials discovery from 2020 onward" -> Graph API search or bulk search.
- "Fetch metadata for these paper IDs" -> Graph API batch.
- "Find similar papers to these seed articles" -> Recommendations API.
- "Download the latest papers dataset and compare it with the prior release" -> Datasets API.


path: references/recommendations-api.md
content: # Recommendations API

## Purpose

Use the Recommendations API when the user already has one or more seed papers and wants related literature.

## Primary Workflow

- Start from trusted seed paper IDs.
- Call the recommendations endpoint for related papers.
- If the user wants richer metadata than the recommendations response provides, follow up with `paper/batch`.
- Keep recommendation-derived results separate from query-derived results so provenance stays explicit.

## When It Beats Search

- The user says "papers similar to this one" or "expand this seed set".
- Keyword search is too brittle or misses nearby work.
- The goal is a related-work neighborhood rather than exhaustive keyword coverage.

## Practical Notes

- Recommendations complement search; they do not replace broad Boolean search for exhaustive reviews.
- Save the input seed IDs with the output records.
- Use recommendations first, then merge with Graph API search results only if the user wants both views.


path: references/graph-api.md
content: # Graph API

## Contents

- Paper search
- Bulk paper search
- Paper batch
- Author search
- Author batch
- Working rules

## Paper Search

Use `GET /graph/v1/paper/search` for interactive search and query refinement.

- Best for smaller result sets and iterative exploration.
- Supports query text plus filters such as year, fields of study, publication types, open-access constraints, and sort options when supported by the endpoint.
- Keep `fields` minimal.

## Bulk Paper Search

Use `GET /graph/v1/paper/search/bulk` for broad retrieval jobs.

- Best for literature harvesting, screening sets, and downstream export.
- Uses continuation-token pagination rather than normal offset paging.
- Save raw records to JSONL before flattening them to CSV or spreadsheets.
- The bundled script `scripts/semantic_scholar_bulk_search.py` implements retries, continuation handling, and CSV export.

## Paper Batch

Use `POST /graph/v1/paper/batch` when paper IDs are already known.

- Better than issuing many single-record calls.
- Useful after search, recommendations, or external ID collection.
- Ask only for required fields.

## Author Search

Use `GET /graph/v1/author/search` to discover candidate authors by name.

- Use this before author batch fetches when only names or loose clues are known.
- Expect ambiguity for common names; use affiliation, topic, or publication clues where available.

## Author Batch

Use `POST /graph/v1/author/batch` when author IDs are known.

- Efficient for pulling metadata for many authors at once.
- Keep returned fields tight.

## Working Rules

- Prefer Graph API for live, user-facing retrieval tasks.
- Use bulk search instead of repeated interactive search calls when the goal is corpus construction.
- Batch endpoints are the default once IDs are known.
- For endpoint-specific parameter details, verify against the live docs before relying on a rarely used filter.


path: references/query-recipes.md
content: # Query Recipes

## Building Queries

- Use Boolean structure for broad topic coverage.
- Use quoted phrases only for exact concepts that must stay intact.
- Combine synonym groups with `|` when supported by the search syntax shown in the tutorial.
- Narrow with year filters, citation thresholds, sort order, and open-access filters when relevant.

## Example Patterns

Broad topical search:

```text
("materials science" | "materials informatics" | "materials discovery")
+ ("multi-agent" | "multi agent" | agentic | "autonomous agent" | "llm agent")
```

Exact phrase:

```text
"materials science multi agent system"
```

Recent papers:

```text
query=... with year filter 2020-
```

High-impact screening:

```text
query=... with sort=citationCount:desc and a minimum citation threshold in downstream filtering
```

Open-access preference:

```text
query=... with open-access filtering when the endpoint supports it
```

## Heuristics

- Start broad, inspect results, then tighten the query.
- Prefer `paper/search/bulk` for large exports even if the first draft query was tested interactively.
- Do not overfit a query to one exact title unless the user wants that specific paper.


path: references/datasets-api.md
content: # Datasets API

## Purpose

Use the Datasets API for offline Semantic Scholar data ingestion, release-based downloads, and incremental diffs between releases.

## Core Workflow

- Discover the relevant release identifier.
- List or identify the dataset files available for that release.
- Download the needed files.
- If comparing releases, use the incremental diff workflow instead of re-downloading everything blindly.

## When To Use

- The user wants corpus-scale offline data.
- The user needs reproducible release snapshots.
- The user wants changes between releases rather than ad hoc online search results.

## When Not To Use

- The user only wants to search a topic or fetch a modest set of papers.
- The user needs quick interactive iteration.
- Storage or download size constraints are unclear.

## Working Rules

- Confirm local storage expectations before downloading.
- Record the exact release identifiers used.
- Treat dataset release workflows as data-engineering tasks, not standard search tasks.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)