Skill: vectorsss/openreview-review-analyzer (version 1.0.1)

--- name: openreview-review-analyzer description: Fetch and analyze peer reviews from OpenReview for any academic paper. Use this skill when the user mentions OpenReview, asks about reviews for a paper, wants a review summary or synthesis, provides an openreview.net URL, mentions a paper forum ID, asks about reviewer opinions or scores for a conference submission (ICLR, NeurIPS, ICML, AAAI, etc.), or wants to understand what reviewers think about a specific paper. Also trigger when the user says things like 'what did reviewers say about this paper', 'summarize the reviews', 'get reviews for this submission', or 'analyze reviewer feedback'. Even if the user just pastes an OpenReview link, this skill should trigger. metadata: {"openclaw":{"emoji":"📝","requires":{"bins":["python3"]}}} --- # OpenReview Review Analyzer Fetch all public peer reviews for any paper on OpenReview and generate a structured synthesis report. ## When to Use - User provides an OpenReview URL (e.g., `https://openreview.net/forum?id=XXXXX`) - User asks to analyze or summarize reviews for a conference paper - User mentions a paper's OpenReview forum ID - User wants to understand reviewer consensus, disagreements, or key concerns - User asks about scores, ratings, or review content for any venue on OpenReview ## Workflow ### Step 1: Extract Forum ID Parse the OpenReview URL or forum ID from user input. The forum ID is the `id` parameter in the URL: - `https://openreview.net/forum?id=xxxxxxx` → forum ID = `xxxxxxx` ### Step 2: Fetch Reviews via Script Run the Python script to fetch all reviews and metadata: ```bash python3 {baseDir}/scripts/fetch_reviews.py <forum_id> ``` The script has **zero external dependencies** — it uses Python's built-in `urllib`. If `requests` is installed it will use that instead, but it's not required. The script outputs a JSON file at `/tmp/openreview_<forum_id>.json` containing: - Paper metadata (title, authors, abstract, venue, keywords) - All official reviews with ratings, confidence, strengths, weaknesses, questions, and full review text - All official comments (author responses, reviewer discussions) - Meta-review if available If the script fails (e.g., network restrictions, reviews not public, paper withdrawn), use these fallback methods in order: **Fallback 1 — web_fetch the API directly:** ``` web_fetch https://api2.openreview.net/notes?forum=<forum_id> ``` Parse the JSON response to get the submission, then: ``` web_fetch https://api2.openreview.net/notes?forum=<forum_id>&trash=true ``` to get all replies including reviews. Filter replies where `invitations` contains `Official_Review`. **Fallback 2 — web_search for review content:** Search for `"<forum_id>" review site:openreview.net` or `"<paper_title>" review <venue>` to find discussions, blog posts, or cached review content. **Fallback 3 — inform the user:** If no review data is accessible, explain that reviews may not be public yet, or suggest the user check the OpenReview page directly. ### Step 3: Generate Synthesis Report Read the JSON output and produce a structured report following `{baseDir}/references/report-template.md`. Key analysis points: 1. **Score Distribution** — list each reviewer's rating and confidence, compute average 2. **Consensus Points** — identify strengths/weaknesses mentioned by multiple reviewers 3. **Key Disagreements** — where reviewers diverge in opinion 4. **Critical Issues** — weaknesses flagged as major by any reviewer 5. **Questions Raised** — important unresolved questions 6. **Author Responses** — summarize rebuttal if available, and whether reviewers updated scores 7. **Meta-Review** — include AC recommendation if available 8. **Overall Assessment** — synthesize into a clear verdict ## Important Notes - OpenReview content is public for completed review cycles. Some venues keep reviews private until decisions are made. - For withdrawn papers, reviews may or may not be visible depending on venue policy. - Always attribute opinions to specific reviewers (e.g., "Reviewer 1 (rating: 5)") when citing specific claims. - The script uses the OpenReview API v2 by default (for venues from 2024+) and falls back to API v1 for older venues. - No authentication is needed for reading public reviews. ## Output Language Match the user's language. If the user writes in Chinese, output the report in Chinese. If in English, output in English.

path: scripts/fetch_reviews.py
content: #!/usr/bin/env python3
"""
Fetch all public reviews and comments for a given OpenReview forum ID.
Outputs structured JSON to /tmp/openreview_<forum_id>.json

Uses the OpenReview REST API directly — NO openreview-py dependency.
Works with just Python standard library (urllib). Uses requests if available.

Usage:
    python3 fetch_reviews.py <forum_id>
    python3 fetch_reviews.py uZWbPNVBUU
    python3 fetch_reviews.py "https://openreview.net/forum?id=uZWbPNVBUU"
"""

import json
import sys
from datetime import datetime
from urllib.parse import urlparse, parse_qs

# ---------------------------------------------------------------------------
# HTTP helper — use requests if available, otherwise fall back to urllib
# ---------------------------------------------------------------------------

try:
    import requests

    def http_get(url, params=None):
        r = requests.get(url, params=params, timeout=30)
        r.raise_for_status()
        return r.json()

except ImportError:
    import urllib.request
    import urllib.error

    def http_get(url, params=None):
        if params:
            from urllib.parse import urlencode
            url = url + "?" + urlencode(params)
        req = urllib.request.Request(url, headers={"User-Agent": "OpenReview-Fetcher/1.0"})
        with urllib.request.urlopen(req, timeout=30) as resp:
            return json.loads(resp.read().decode("utf-8"))

# ---------------------------------------------------------------------------
# API endpoints
# ---------------------------------------------------------------------------

API_V2 = "https://api2.openreview.net"
API_V1 = "https://api.openreview.net"

# ---------------------------------------------------------------------------
# Content extraction helpers
# ---------------------------------------------------------------------------

def extract_value(obj):
    """Extract value from OpenReview content field (handles both v1 and v2 formats)."""
    if isinstance(obj, dict) and "value" in obj:
        return obj["value"]
    return obj


def extract_content(content_dict):
    """Flatten a note's content dict, extracting all values."""
    if not content_dict:
        return {}
    return {k: extract_value(v) for k, v in content_dict.items()}


def classify_note(note, api_version=2):
    """Classify a note by its invitation type."""
    if api_version == 2:
        invitations = note.get("invitations", [])
        inv_str = " ".join(invitations).lower()
    else:
        inv_str = (note.get("invitation", "") or "").lower()

    if "official_review" in inv_str or "/review" in inv_str:
        return "review"
    elif "meta_review" in inv_str or "metareview" in inv_str:
        return "meta_review"
    elif "decision" in inv_str:
        return "decision"
    elif "rebuttal" in inv_str:
        return "rebuttal"
    else:
        return "comment"


def format_timestamp(ts_ms):
    """Convert millisecond timestamp to readable date string."""
    if not ts_ms:
        return "N/A"
    try:
        return datetime.fromtimestamp(ts_ms / 1000).strftime("%Y-%m-%d %H:%M")
    except (ValueError, OSError):
        return "N/A"


def parse_rating(value):
    """Extract numeric rating from string like '5: Borderline accept' or plain int."""
    if isinstance(value, (int, float)):
        return value
    if isinstance(value, str):
        try:
            return int(value.split(":")[0].strip())
        except ValueError:
            return value
    return value


def build_note_entry(note, api_version=2):
    """Build a standardized entry dict from a raw note."""
    content = extract_content(note.get("content", {}))
    return {
        "id": note.get("id"),
        "signatures": note.get("signatures", []),
        "invitations": note.get("invitations", []) if api_version == 2 else [note.get("invitation", "")],
        "created": format_timestamp(note.get("cdate")),
        "content": content,
    }


# ---------------------------------------------------------------------------
# API v2 fetcher
# ---------------------------------------------------------------------------

def fetch_v2(forum_id):
    """Fetch paper + all replies via OpenReview API v2."""
    print("  Fetching submission...")
    data = http_get(f"{API_V2}/notes", params={"id": forum_id})
    notes = data.get("notes", [])
    if not notes:
        raise Exception(f"No submission found for forum ID: {forum_id}")
    submission = notes[0]

    paper_content = extract_content(submission.get("content", {}))
    paper_info = {
        "id": submission["id"],
        "title": paper_content.get("title", "N/A"),
        "authors": paper_content.get("authors", []),
        "abstract": paper_content.get("abstract", "N/A"),
        "keywords": paper_content.get("keywords", []),
        "venue": paper_content.get("venue", "N/A"),
        "venueid": paper_content.get("venueid", "N/A"),
        "domain": submission.get("domain", "N/A"),
        "url": f"https://openreview.net/forum?id={forum_id}",
    }

    # Fetch all replies in the forum
    print("  Fetching all replies...")
    all_replies = []
    offset = 0
    limit = 1000
    while True:
        data = http_get(f"{API_V2}/notes", params={
            "forum": forum_id,
            "offset": offset,
            "limit": limit,
        })
        batch = data.get("notes", [])
        all_replies.extend(batch)
        if len(batch) < limit:
            break
        offset += limit

    # Classify replies
    result = {"review": [], "comment": [], "meta_review": [], "decision": [], "rebuttal": []}
    for note in all_replies:
        if note["id"] == forum_id:
            continue
        category = classify_note(note, api_version=2)
        result[category].append(build_note_entry(note, api_version=2))

    return paper_info, result


# ---------------------------------------------------------------------------
# API v1 fetcher (fallback for older venues)
# ---------------------------------------------------------------------------

def fetch_v1(forum_id):
    """Fetch paper + all replies via OpenReview API v1."""
    print("  Fetching submission (API v1)...")
    data = http_get(f"{API_V1}/notes", params={"id": forum_id})
    notes = data.get("notes", [])
    if not notes:
        raise Exception(f"No submission found for forum ID: {forum_id} (API v1)")
    submission = notes[0]

    content = submission.get("content", {})
    paper_info = {
        "id": submission["id"],
        "title": content.get("title", "N/A"),
        "authors": content.get("authors", []),
        "abstract": content.get("abstract", "N/A"),
        "keywords": content.get("keywords", []),
        "venue": content.get("venue", "N/A"),
        "venueid": content.get("venueid", "N/A"),
        "url": f"https://openreview.net/forum?id={forum_id}",
    }

    print("  Fetching all replies (API v1)...")
    all_replies = []
    offset = 0
    limit = 1000
    while True:
        data = http_get(f"{API_V1}/notes", params={
            "forum": forum_id,
            "offset": offset,
            "limit": limit,
        })
        batch = data.get("notes", [])
        all_replies.extend(batch)
        if len(batch) < limit:
            break
        offset += limit

    result = {"review": [], "comment": [], "meta_review": [], "decision": [], "rebuttal": []}
    for note in all_replies:
        if note["id"] == forum_id:
            continue
        category = classify_note(note, api_version=1)
        result[category].append(build_note_entry(note, api_version=1))

    return paper_info, result


# ---------------------------------------------------------------------------
# Rating / confidence extraction
# ---------------------------------------------------------------------------

def extract_scores(reviews):
    """Extract ratings and confidences from review entries."""
    ratings = []
    confidences = []

    for r in reviews:
        c = r.get("content", {})

        # Rating — try common field names
        for key in ["rating", "recommendation", "score", "overall_assessment", "overall"]:
            if key in c:
                ratings.append(parse_rating(c[key]))
                break

        # Confidence
        for key in ["confidence", "reviewer_confidence"]:
            if key in c:
                confidences.append(parse_rating(c[key]))
                break

    return ratings, confidences


# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------

def main():
    if len(sys.argv) < 2:
        print("Usage: python3 fetch_reviews.py <forum_id_or_url>")
        print("Example: python3 fetch_reviews.py uZWbPNVBUU")
        print('Example: python3 fetch_reviews.py "https://openreview.net/forum?id=uZWbPNVBUU"')
        sys.exit(1)

    raw_input = sys.argv[1].strip()

    # Parse URL if provided
    if "openreview.net" in raw_input:
        parsed = urlparse(raw_input)
        params = parse_qs(parsed.query)
        forum_id = params.get("id", [raw_input])[0]
    else:
        forum_id = raw_input

    print(f"Fetching reviews for forum ID: {forum_id}")
    print(f"URL: https://openreview.net/forum?id={forum_id}\n")

    paper_info = None
    classified = None

    # Try API v2 first, then v1
    for api_name, fetch_fn in [("API v2", fetch_v2), ("API v1", fetch_v1)]:
        try:
            print(f"Trying {api_name}...")
            paper_info, classified = fetch_fn(forum_id)
            print(f"  ✓ Success with {api_name}\n")
            break
        except Exception as e:
            print(f"  ✗ {api_name} failed: {e}\n")

    if paper_info is None:
        print("ERROR: Could not fetch paper from either API.")
        print("Possible reasons:")
        print("  - Invalid forum ID")
        print("  - Reviews are not yet public")
        print("  - Paper has been deleted")
        print("  - Network connectivity issue")
        sys.exit(1)

    reviews = classified["review"]
    ratings, confidences = extract_scores(reviews)

    summary = {
        "total_reviews": len(reviews),
        "total_comments": len(classified["comment"]),
        "total_meta_reviews": len(classified["meta_review"]),
        "total_decisions": len(classified["decision"]),
        "total_rebuttals": len(classified["rebuttal"]),
        "ratings": ratings,
        "confidences": confidences,
    }

    if ratings and all(isinstance(r, (int, float)) for r in ratings):
        summary["avg_rating"] = round(sum(ratings) / len(ratings), 2)
    if confidences and all(isinstance(c, (int, float)) for c in confidences):
        summary["avg_confidence"] = round(sum(confidences) / len(confidences), 2)

    output = {
        "paper": paper_info,
        "reviews": reviews,
        "comments": classified["comment"],
        "meta_reviews": classified["meta_review"],
        "decisions": classified["decision"],
        "rebuttals": classified["rebuttal"],
        "summary": summary,
    }

    output_path = f"/tmp/openreview_{forum_id}.json"
    with open(output_path, "w", encoding="utf-8") as f:
        json.dump(output, f, ensure_ascii=False, indent=2, default=str)

    # Print summary
    print("=" * 55)
    print(f"  Paper:   {paper_info.get('title', 'N/A')}")
    print(f"  Venue:   {paper_info.get('venue', 'N/A')}")
    print(f"  Domain:  {paper_info.get('domain', 'N/A')}")
    print("-" * 55)
    print(f"  Reviews:      {len(reviews)}")
    print(f"  Comments:     {len(classified['comment'])}")
    print(f"  Meta-reviews: {len(classified['meta_review'])}")
    print(f"  Decisions:    {len(classified['decision'])}")
    print(f"  Rebuttals:    {len(classified['rebuttal'])}")
    if ratings:
        print(f"  Ratings:      {ratings}")
        if "avg_rating" in summary:
            print(f"  Avg rating:   {summary['avg_rating']}")
    if confidences:
        print(f"  Confidences:  {confidences}")
    print("=" * 55)
    print(f"\n✓ Output saved to: {output_path}")


if __name__ == "__main__":
    main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)