Skill: geoly-geo/geo-hallucination-checker (version 0.1.0)

--- name: geo-hallucination-checker description: > Detect and annotate hallucinations, unsupported claims, fabricated studies, and incorrect conclusions in text so that AI only cites verifiable, trustworthy content. Use this skill whenever the user asks you to fact-check, validate sources, check for hallucinations, or ensure that generated content is grounded in real evidence, even if they do not explicitly use the word "hallucination". compatibility: [] --- ## Overview The `geo-hallucination-checker` skill is a hallucination and false-information detection tool. It helps you review any piece of content (articles, landing pages, product descriptions, FAQs, GEO-optimized drafts, etc.) and: - Identify **unsupported factual claims** - Flag **fabricated or suspicious studies, reports, and statistics** - Highlight **incorrect or overconfident conclusions** - Suggest **safer, evidence-friendly rephrasings** The primary goal is to ensure that AI systems only cite **truthful, well-grounded content** and clearly mark anything that looks like hallucination risk. Use this skill aggressively whenever there is any risk that the model might invent data, sources, or conclusions. ## When to use this skill Use `geo-hallucination-checker` whenever: - The user asks you to **fact-check, verify, or validate** content. - The task involves **medical, financial, legal, scientific, or technical** claims. - A draft includes **numbers, percentages, dates, or strong superlatives** (e.g., “the best”, “number one”, “guaranteed”, “clinically proven”). - A text mentions **studies, universities, journals, or institutions** without clear, verifiable details. - You are preparing **GEO-optimized content** that might be quoted by AI models and needs to be extra reliable. - You are asked to **rewrite content to avoid hallucinations or false claims**. If you are unsure whether hallucinations are a concern, **assume they are** and apply this skill. ## Inputs this skill supports This skill can be used on: - A single paragraph or answer - A long-form article, blog post, or whitepaper - A product page or landing page draft - FAQ content or knowledge base articles - Generated GEO content that will be cited by AI models The user may also provide: - **Explicit sources or references** (links, documents, citations) - **Constraints** (e.g., “do not use external web search”, “only use these PDFs as ground truth”) Always respect any constraints the user provides. ## Core workflow When using this skill, follow this workflow: 1. **Clarify the task mode** - If the user only asks to “check for hallucinations” or “verify content”, focus on **analysis**. - If the user asks you to “rewrite safely”, “make this citation-safe”, or “fix hallucinations”, perform **analysis first**, then produce a **hallucination-safe rewrite**. 2. **Parse the content and extract claims** - Read the entire text carefully before judging specific parts. - Break the content into **atomic factual claims**. A claim is a statement that could, in principle, be checked as true or false. - Ignore purely stylistic or obviously subjective language unless it is presented as an objective fact. 3. **Check available evidence** - Prefer **explicit sources provided by the user** (links, documents, citations). - If tools are available and allowed, you may use them to consult: - Official documentation or first-party sources - Well-known reference material - If you **cannot confidently verify a claim**, treat it as **unsupported** rather than assuming it is true. 4. **Classify each claim** For each atomic factual claim, assign: - `status`: - `Supported` – clearly backed by the provided sources or well-established knowledge. - `Unsupported` – no clear support; could be true, but you do not see evidence. - `Problematic` – exaggerated, misleading, overconfident, or very unlikely without strong evidence. - `Contradicted` – clearly conflicts with known facts or given sources. - `Speculative` – forward-looking, predictive, or hypothetical, presented without clear caveats. - `risk_level`: - `Low` – unlikely to cause harm or serious misinformation. - `Medium` – could mislead, but impact is moderate or limited. - `High` – serious risk of harm, legal issues, medical/financial danger, or major reputational damage. - `reason`: - A short explanation of **why** you assigned that status and risk (e.g., “no source for extreme 500% performance claim”). - `suggested_fix`: - A concrete recommendation such as: - “Remove this claim unless you can provide a real citation.” - “Rephrase as a possibility, not a guarantee.” - “Add a specific, verifiable source (e.g., link, DOI, report).” 5. **Look for common hallucination patterns** Pay special attention to: - **Fabricated studies and journals** - Vague references like “a 2026 MIT study” or “Journal of Advanced AI Research” with no details. - Journals or conferences that do not exist or sound suspiciously generic. - **Overconfident medical or scientific claims** - “Clinically proven to cure…” - “Guaranteed to reduce X by 80%.” - **Overly precise unsourced statistics** - Very specific percentages, sample sizes, or timeframes with no citation. - **Superlatives and absolutes** - “The only solution that…” - “Best in the world”, “100% safe”, “zero risk”. - **Misuse of authority** - Name-dropping famous institutions or companies without any concrete evidence. Treat these as **high-risk** unless there is strong, clear evidence. 6. **Produce a structured hallucination analysis** Always output a clear, structured analysis with two parts: 1. **High-level summary** - Briefly describe: - Overall hallucination risk (low/medium/high) - The most critical issues to fix before publication or citation 2. **Claim-level table** - Use a markdown table with the following columns: - `#` – sequential index - `claim_text` – the exact or paraphrased claim - `status` – Supported / Unsupported / Problematic / Contradicted / Speculative - `risk_level` – Low / Medium / High - `reason` – a short explanation - `suggested_fix` – what to do about it Example structure (illustrative, not prescriptive content): | # | claim_text | status | risk_level | reason | suggested_fix | | - | ---------- | ------ | ---------- | ------ | ------------- | | 1 | “Clinically proven to reduce depression by 80% in 2 weeks” | Problematic | High | No specific clinical trial or citation provided; extreme effect size is unlikely without strong evidence. | Add concrete trial details with citation or downgrade to cautious, non-clinical language. | 7. **(Optional) Hallucination-safe rewrite** If the user explicitly requests a rewrite or safer version, after the table: - Provide a section titled **“Hallucination-safe version”**. - Rewrite the original content: - Remove or soften high-risk claims. - Replace overconfident language with cautious, transparent wording. - Explicitly signal uncertainty where facts are not known (e.g., “Some users report…”, “Early results suggest…”). - Do **not** invent: - Study names, DOIs, journal titles, or URLs. - Exact statistics or dates you cannot justify. - If a strong claim is important but currently unsupported, suggest a placeholder note such as: - “[Insert verified statistic with citation here]” ## Constraints and safety rules - **Never invent sources.** - Do not fabricate papers, DOIs, journal names, or institutional reports. - If you are not sure a source exists, treat the claim as unsupported or problematic. - **Err on the side of caution.** - It is better to mark a real claim as “Unsupported” than to let a hallucinated claim pass as fact. - **Separate facts from marketing.** - Marketing language is acceptable **only if** it is not masquerading as hard evidence. - When in doubt, suggest softer, more honest language and disclose uncertainty. - **Respect user constraints about tools and data.** - If the user forbids external web search or asks you to rely only on given documents, follow that rule strictly. - Under such constraints, label claims based on what you can see, and explain that some might be true but remain “Unsupported” due to limited data. ## How this skill interacts with other GEO skills When used together with other GEO-oriented skills (e.g., content optimization, schema generation, or conversion optimization): - Run `geo-hallucination-checker` **after** content is drafted but **before** finalizing output that might be cited. - Use the hallucination analysis to: - Remove or soften risky claims. - Add explicit “needs citation” notes where appropriate. - Ensure all structured data (e.g., Schema.org fields) does not encode hallucinated facts. If there is a conflict between persuasive copywriting and factual accuracy, **prioritize factual accuracy and safety**. ## Output format summary Unless the user specifies a different format, always: 1. Start with a **short summary**: - Overall hallucination risk level. - 2–5 bullets with the most important issues. 2. Provide a **markdown table** as described in the workflow section. 3. If requested, append a **“Hallucination-safe version”** that rewrites the content according to your analysis. Aim for clarity and directness so that humans and AI systems can easily see which parts of the text are safe to cite and which require caution or correction.

path: evals/evals.json
content: {
  "skill_name": "geo-hallucination-checker",
  "evals": [
    {
      "id": 1,
      "prompt": "You are given a blog post draft about a new 'QuantumBoost-9' processor that claims a 500% performance increase over all existing CPUs, references a '2026 MIT study in the Journal of Impossible Physics', and quotes several exact statistics with no sources. Use the geo-hallucination-checker skill to analyze the text, flag hallucinated or unsupported claims, and propose safer phrasing that avoids inventing sources.",
      "expected_output": "The assistant identifies fabricated or suspicious elements (e.g., non-existent or dubious journal, extreme performance claims, unsourced statistics), marks them as high-risk hallucinations or unsupported claims, explains why they are risky, and suggests conservative rephrasing or removal instead of inventing sources.",
      "files": []
    },
    {
      "id": 2,
      "prompt": "A landing page draft claims that a wellness app is 'clinically proven to reduce depression by 80% in 2 weeks' and cites an unspecified 'Harvard 2025 study' without a link, authors, or details. Use the geo-hallucination-checker skill to review these claims and output a structured hallucination analysis table plus a safer alternative copy block.",
      "expected_output": "The assistant highlights that the clinical proof and Harvard study are unspecified and likely hallucinated or at least unsupported, labels them as high-risk problematic claims, recommends either adding real study details or softening language to avoid false medical claims, and outputs both a hallucination analysis table and a rewritten, safer version.",
      "files": []
    },
    {
      "id": 3,
      "prompt": "You receive a product page draft for an AI camera that includes realistic but unsourced specs (sensor size, resolution, frame rate) and modest claims such as 'used by hundreds of creators worldwide'. Use the geo-hallucination-checker skill to distinguish between low-risk, plausible claims and high-risk, potentially hallucinated details, and clearly mark which ones require external verification.",
      "expected_output": "The assistant classifies each claim with status and risk level, treating specific technical specs and numbers as needing verification while allowing modest qualitative claims as lower risk, marks clearly which claims require external verification, and avoids inventing citations or new statistics.",
      "files": []
    }
  ]
}



path: references/hallucination_guide.md
content: # Hallucination Guide for geo-hallucination-checker

This reference file defines key concepts, patterns, and examples for the
`geo-hallucination-checker` skill. Use it when you need more nuance about
what counts as a hallucination and how to classify different kinds of claims.

## 1. Core definitions

- **Factual claim**: A statement that can be true or false in the real world.
- **Hallucination**: A confident factual claim that is not supported by
  available evidence or that contradicts known facts.
- **Unsupported claim**: A factual statement that might be true but for which
  you cannot see or access evidence in the current context.
- **Problematic claim**: A statement that is misleading, exaggerated, or
  overconfident relative to the evidence.
- **Speculative claim**: A forward-looking or hypothetical statement that is
  acceptable only when clearly labeled as such (e.g., “may”, “could”, “might”).

The practical rule: when in doubt, prefer labeling a claim as
**Unsupported** or **Speculative** rather than silently treating it as true.

## 2. Taxonomy of hallucination patterns

Use this taxonomy to guide your analysis:

1. **Fabricated sources**
   - Non-existent journals, conferences, or organizations.
   - Vague references like “a 2024 study from a leading university” without
     authors, title, or venue.
   - Citations that cannot be traced back to a real publication, given the
     information provided.

2. **Invented statistics**
   - Precise numbers (e.g., “83.7% of users”) with no clear data source.
   - Sample sizes, p-values, or confidence intervals that appear simply as
     decoration.

3. **Overconfident medical, financial, or legal claims**
   - “Clinically proven to cure…”
   - “Guaranteed returns of 30% per year.”
   - “Zero risk of side effects.”

4. **Misuse of authority**
   - Name-dropping famous institutions (e.g., “Harvard”, “MIT”, “World Health
     Organization”) without specific, verifiable details.
   - Claims that rest entirely on “big names” instead of evidence.

5. **Out-of-date or oversimplified facts**
   - Presenting old information as current without context.
   - Ignoring known exceptions or important caveats.

6. **Over-generalization**
   - Turning limited or early findings into universal laws.
   - “Works for everyone”, “in all cases”, “across every industry”.

## 3. Classification guidance

When assigning `status` and `risk_level`, consider:

- **Domain sensitivity**:
  - Medical, financial, legal, and safety-related claims usually warrant
    a higher risk level.
  - Marketing fluff may be lower risk unless it crosses into deception.

- **Specificity and extremeness**:
  - Extreme performance or effect size claims (e.g., “500% faster”, “80% cure
    rate in two weeks”) are high risk without strong evidence.
  - Vague, modest claims (e.g., “many users find this helpful”) are lower risk.

- **Availability of evidence**:
  - If the user provides detailed sources (links, PDFs, citations), prefer
    checking those before labeling something problematic.
  - If tools are restricted and you cannot verify, it is fine to say:
    “This might be true, but appears unsupported given the available data.”

## 4. Example classifications

Example 1:
- Claim: “Clinically proven to reduce depression by 80% in just 2 weeks.”
- Classification:
  - `status`: Problematic
  - `risk_level`: High
  - Reason: Extreme medical effect size, no trial details or citation.
  - Suggested fix: Add real clinical trial details and citation, or downgrade
    to cautious language (e.g., “Some users report feeling better after two
    weeks, but results vary.”).

Example 2:
- Claim: “Used by hundreds of creators worldwide.”
- Classification:
  - `status`: Unsupported (if no data is shown)
  - `risk_level`: Low to Medium
  - Reason: Vague, modest marketing claim; could be true but not proven.
  - Suggested fix: Either supply real usage data or soften language.

Example 3:
- Claim: “A 2026 MIT study in the Journal of Impossible Physics proved a 500%
  increase in performance over all existing CPUs.”
- Classification:
  - `status`: Problematic or Contradicted (depending on context)
  - `risk_level`: High
  - Reason: Suspicious journal name, extreme claim, no accessible details.
  - Suggested fix: Treat as hallucinated until a real, verifiable citation is
    provided; otherwise, remove or replace with a modest, clearly hypothetical
    statement.

## 5. Integration with content rewriting

When rewriting content to be hallucination-safe:

- Preserve the **legitimate core value proposition** where possible.
- Remove or soften claims that:
  - Rely on fabricated or unverifiable sources.
  - Overstate certainty or effect sizes.
  - Could mislead users about safety or guarantees.
- Use cautious language:
  - “may help”, “can support”, “is designed to”, “early results suggest”.
- Make uncertainty explicit instead of hiding it.

The goal is not to make content bland, but to keep it **honest, grounded,
and safe to cite**.



path: scripts/hallucination_checker.py
content: """
Simple heuristic-based helper for the geo-hallucination-checker skill.

This script is not a full fact-checking engine. Instead, it:
- Splits text into rough sentences.
- Scores each sentence for hallucination risk based on lexical patterns.
- Returns a list of sentences with basic risk scores and matched patterns.

It is intended as an optional helper for deterministic workflows or
batch pre-screening, not as a replacement for careful reasoning in the skill.
"""

from __future__ import annotations

import re
from dataclasses import dataclass, asdict
from typing import List, Dict, Any


@dataclass
class SentenceRisk:
    sentence: str
    risk_score: int
    matched_patterns: List[str]


SUSPICIOUS_PATTERNS = [
    r"\b\d{2,3}%\b",  # very specific percentages
    r"\bclinical(ly)? proven\b",
    r"\bguaranteed\b",
    r"\bno risk\b",
    r"\bzero risk\b",
    r"\bthe only\b",
    r"\bworld(-|\s)?class\b",
    r"\bnumber one\b",
    r"\b#1\b",
    r"\baccording to (a|an|the)?\s*(study|report|research)\b",
    r"\b(in|from)\s+\d{4}\s+(study|trial|paper)\b",
    r"\b(randomized|double-blind|placebo-controlled)\b",
    r"\b(guarantee|warranty) of\b",
]


SUSPICIOUS_INSTITUTION_PATTERNS = [
    r"\bHarvard\b",
    r"\bMIT\b",
    r"\bStanford\b",
    r"\bOxford\b",
    r"\bCambridge\b",
    r"\bYale\b",
    r"\bWorld Health Organization\b",
    r"\bWHO\b",
    r"\bFDA\b",
    r"\bEMA\b",
    r"\bNobel\b",
]


def split_into_sentences(text: str) -> List[str]:
    """Very naive sentence splitter. Good enough for heuristic scanning."""
    # Replace newlines with spaces and collapse multiple spaces.
    cleaned = re.sub(r"\s+", " ", text.strip())
    if not cleaned:
        return []

    # Split on sentence-ending punctuation while keeping things simple.
    parts = re.split(r"(?<=[.!?])\s+", cleaned)
    sentences = [p.strip() for p in parts if p.strip()]
    return sentences


def score_sentence(sentence: str) -> SentenceRisk:
    """
    Score a single sentence for hallucination risk.

    Higher scores indicate more potential hallucination risk.
    """
    matched: List[str] = []
    score = 0

    lower = sentence.lower()

    for pattern in SUSPICIOUS_PATTERNS:
        if re.search(pattern, sentence, flags=re.IGNORECASE):
            matched.append(pattern)
            score += 2

    for pattern in SUSPICIOUS_INSTITUTION_PATTERNS:
        if re.search(pattern, sentence, flags=re.IGNORECASE):
            matched.append(pattern)
            score += 1

    # Extra penalty for mixture of strong claims and institutions.
    if score >= 3 and ("study" in lower or "trial" in lower or "proven" in lower):
        score += 1

    return SentenceRisk(sentence=sentence, risk_score=score, matched_patterns=matched)


def analyze_text(text: str, min_score: int = 2) -> List[Dict[str, Any]]:
    """
    Analyze a block of text and return potentially risky sentences.

    :param text: Input content to scan.
    :param min_score: Minimum risk_score to include in the output.
    :return: List of dicts with sentence, risk_score, and matched_patterns.
    """
    sentences = split_into_sentences(text)
    results: List[Dict[str, Any]] = []

    for sent in sentences:
        risk = score_sentence(sent)
        if risk.risk_score >= min_score:
            results.append(asdict(risk))

    return results


def main() -> None:
    import sys
    import json

    if sys.stdin.isatty():
        print("Paste text on stdin to analyze hallucination risk.", file=sys.stderr)
        print("Example: python hallucination_checker.py < input.txt", file=sys.stderr)
        sys.exit(1)

    text = sys.stdin.read()
    data = analyze_text(text)
    json.dump(data, sys.stdout, indent=2, ensure_ascii=False)


if __name__ == "__main__":
    main()




Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)