Skill: morvanzhou/improve-skill-with-best-practices (version 1.0.3)

--- name: google-analytics-and-search-improve description: Understand website goals and user journeys first, then analyze GSC/GA4 data and audit the live site to validate whether users behave as intended. Identify gaps between goals and reality, and produce actionable, goal-aligned improvement plans. Use when user wants to diagnose website problems, improve search rankings, optimize traffic, analyze Google Analytics or Search Console data, audit website performance, or create a data-backed improvement roadmap. --- # Goal-Driven Website Analytics & Improvement First understand what the site does and what it wants users to do, then use GSC/GA4 data combined with live-site auditing to verify whether users actually follow the intended journey. Identify the gap between goals and reality, and generate prioritized, actionable improvement plans. ## Data Storage All runtime data is stored in `$DATA_DIR`, separated from skill code. ``` <project_root>/.skills-data/google-analytics-and-search-improve/ .env # Configuration (auth, URLs, etc.), auto-loaded by scripts data/ # Raw data only: API responses, user-uploaded CSVs (GSC/GA4/PSI JSON/CSV) analysis/ # Analysis outputs: reports, audit results, website profile, funnel analysis charts/ # Generated chart images (PNG) for embedding in analysis reports scripts/ # Analysis scripts: code that reads from data/ and writes to analysis/ tmp/ # Screenshots and temporary files cache/ # API response cache configs/ # Config files (including Service Account JSON keys) logs/ # Execution logs venv/ # Python virtual environment (managed by uv) ``` **Directory separation principle**: - `data/` = raw, unprocessed data from APIs or user exports (input) - `analysis/` = processed results, reports, insights, audit findings (output) - `charts/` = generated chart images (PNG) that are embedded in analysis reports via relative paths - `scripts/` = analysis code that transforms `data/` → `analysis/` ## Core Principles ### Goal → Data → Gap → Action Every analysis must start from the website's purpose and intended user journey. Data is only valuable when measured against a goal. The skill's job is to: 1. Understand what the site wants users to do (goal) 2. Use data to check if users actually do it (reality) 3. Find where the gap is (diagnosis) 4. Tell the developer what to do next (action) ### Code-Driven Data Analysis & Visualization **All data analysis MUST be done through code, not by directly reading JSON and manually summarizing.** When analyzing raw data from `$DATA_DIR/data/` (GSC JSON, GA4 JSON, PSI JSON, Bing JSON, etc.): 1. **Write a Python script** in `$DATA_DIR/scripts/` that reads the raw JSON, processes and aggregates the data, and outputs the structured results 2. **Execute the script** to produce accurate, reproducible results 3. **Generate charts** in the same script (or a companion script) — every analysis phase that produces quantitative data MUST generate accompanying charts saved to `$DATA_DIR/charts/` 4. **Use the script output and charts** to write the analysis report — embed chart images in Markdown reports **Why**: Raw JSON files from APIs can be large and complex. Manually reading JSON and converting to text is error-prone — it's easy to miscount, misinterpret nested structures, or miss data. Code execution guarantees data accuracy and reproducibility. Charts generated from the same code ensure visual evidence is consistent with the data. This principle applies to **all phases**: GSC analysis, GA4 analysis, funnel analysis, site audit data extraction, and final report data summarization. Never skip the code step. Standard code patterns (data analysis + chart generation in one script), chart type selection guide, per-phase chart requirements, and CJK font support: see [references/data-visualization-guide.md](references/data-visualization-guide.md). ## Workflow ``` Analysis Progress: - [ ] Phase 0: Website reconnaissance & goal definition - [ ] Phase 1: Select data source & collect data - [ ] Phase 2: GSC data analysis (aligned to goals) - [ ] Phase 3: GA4 data analysis (aligned to goals) - [ ] Phase 3b: GA4 funnel exploration (optional, if user has custom events) - [ ] Phase 4: Live site audit - [ ] Phase 5: Source code review - [ ] Phase 5b: SEO & GEO optimization checklist audit - [ ] Phase 6: Generate goal-aligned improvement report ``` --- ### Phase 0: Website Reconnaissance & Goal Definition > **Purpose**: Without understanding what the site does and what it wants users to do, all subsequent data analysis is directionless. **Steps**: 1. **Visit the site** using `agent-browser` — explore homepage + 3-5 key pages from navigation. Take full-page screenshots, extract page metadata (title, headings, nav links, CTAs, forms). Save screenshots to `$DATA_DIR/tmp/`. 2. **Classify the website type** (Content/Blog, SaaS/Tool, E-commerce, Lead Gen, etc.) based on observed elements. 3. **Infer goals and present to user** — determine Primary Goal, Secondary Goals, Intended User Journey, and Key Conversion Points. Ask user to confirm or correct. 4. **Rank analysis dimensions** — select and rank which dimensions matter most for this site type (Acquisition/SEO, Conversion Funnel, Content Engagement, UX, Performance, Retention, Technical SEO/GEO). 5. **Ask user for additional context** — traffic sources, GA4 custom events, known pain points, recent changes, target audience. Save the Website Goal Profile to `$DATA_DIR/analysis/website-profile.md`. Detailed browser commands, classification tables, and templates: see [references/website-reconnaissance-reference.md](references/website-reconnaissance-reference.md). --- ### Phase 1: Select Data Source & Collect Data **1a. Initialize directories & Python environment**: ```bash DATA_DIR=".skills-data/google-analytics-and-search-improve" mkdir -p "$DATA_DIR"/{data,analysis,charts,scripts,cache,logs,tmp,configs} ``` Set up a Python 3.12 virtual environment via `uv` and install dependencies (only needed once): ```bash uv venv "$DATA_DIR/venv" --python 3.12 uv pip install -p "$DATA_DIR/venv" -r skills/google-analytics-and-search-improve/scripts/pyproject.toml ``` > **IMPORTANT**: All Python script executions in this skill MUST activate the venv first, then use `python` directly. Never use the system Python. > ```bash > source "$DATA_DIR/venv/bin/activate" > python scripts/xxx.py ... > ``` **1b. Ask user to choose data source**: | Mode | Description | When to Use | |------|-------------|-------------| | **A. API auto-collection** | Create Service Account, configure `.env`, scripts auto-collect data | Most complete data, recommended | | **B. Manual CSV export** | User exports CSV from GA4/GSC web consoles | Zero config, simplest | | **C. Browser audit only** | Visit site directly, no GA4/GSC data | Quick technical checks | **Mode A**: Check `$DATA_DIR/.env` for required config. If missing, guide user to fill in `SITE_URL`, `GSC_SITE_URL`, `GA4_PROPERTY_ID`. Place Service Account JSON key in `$DATA_DIR/configs/` (scripts auto-discover it via `utils.py`). Run collection scripts to save data to `$DATA_DIR/data/`. **Mode B**: Send export instructions to user. After receiving CSV files in `$DATA_DIR/data/`, proceed to Phase 2-3. **Mode C**: Only collect `SITE_URL`, skip to Phase 4. Detailed configuration, collection commands, CSV export instructions, and custom query guidance: see [references/data-collection-reference.md](references/data-collection-reference.md). --- ### Phase 2: GSC Data Analysis (Goal-Aligned) > **Purpose**: Understand how users find the site through search, and whether the right pages rank for the right queries. Read GSC data from `$DATA_DIR/data/`. Review the Website Goal Profile from `$DATA_DIR/analysis/website-profile.md` before analyzing. **Write analysis scripts** in `$DATA_DIR/scripts/` to process raw GSC JSON data. Use code to extract, aggregate, and calculate metrics — do not manually read JSON. **Goal-aligned analysis**: 1. **Map queries to user intent stages** — classify top queries by Awareness / Consideration / Decision / Retention stages of the intended user journey. 2. **Check page-query alignment** — are users arriving at stage-appropriate pages, or landing on irrelevant pages? 3. **Standard SEO diagnostics** — high-impression low-CTR keywords, keywords ranked 4-10, declining pages, index coverage and sitemap health. Analyze against thresholds in [references/metrics-glossary.md](references/metrics-glossary.md). **Bing analysis** (optional): When `BING_WEBMASTER_API_KEY` is configured, include cross-engine comparison (Bing vs Google), keyword research via Bing's unique endpoints, backlink and crawl health analysis. Details in [references/data-collection-reference.md](references/data-collection-reference.md) §8. **Output**: Top 10 SEO optimization opportunities organized by user journey stage. Save to `$DATA_DIR/analysis/gsc_analysis.md`. Analysis scripts should also generate Phase 2 charts (see [references/data-visualization-guide.md](references/data-visualization-guide.md)) and save to `$DATA_DIR/charts/gsc_*.png`. --- ### Phase 3: GA4 Data Analysis (Goal-Aligned) > **Purpose**: Compare the intended user journey with actual behavior — find where reality diverges from intention. Read GA4 data from `$DATA_DIR/data/`. Review the Website Goal Profile before analyzing. **Write analysis scripts** in `$DATA_DIR/scripts/` to process raw GA4 JSON data. Use code to extract, aggregate, and calculate metrics — do not manually read JSON. **Goal-aligned analysis**: 1. **Intended Journey vs Actual Behavior** — landing page alignment, path progression, conversion point performance. 2. **Identify behavior gaps** — dead-end pages (high traffic, no next step), key pages with low traffic, high-bounce pages, device/channel disparities. 3. **Standard GA4 diagnostics** — traffic trends, channel effectiveness, mobile vs desktop gaps, conversion events. 4. **Flag missing data** — explicitly tell the user what additional data would help explain gaps. Analyze against thresholds in [references/metrics-glossary.md](references/metrics-glossary.md). **Output**: Top 10 GA4 insights framed as "expected behavior vs actual behavior". Save to `$DATA_DIR/analysis/ga4_analysis.md`. Analysis scripts should also generate Phase 3 charts (see [references/data-visualization-guide.md](references/data-visualization-guide.md)) and save to `$DATA_DIR/charts/ga4_*.png`. --- ### Phase 3b: GA4 Funnel Exploration (Optional) > **Purpose**: Deep-dive into multi-step conversion funnels when the user has custom events. Ask the user if they have custom events for funnel analysis (e.g., signup flow, purchase flow). Skip if no custom events. Use `ga4_funnel.py` to generate funnel reports. Supports: step-by-step completion/abandonment rates, device/channel breakdown, daily trends, advanced JSON config for complex filters. **Write analysis scripts** in `$DATA_DIR/scripts/` to process raw funnel JSON data. Use code to calculate drop-off rates and segment comparisons. > **API note**: `ga4_funnel.py` uses GA4 v1alpha API — functional but may have breaking changes. No additional auth needed beyond existing Service Account. Script usage, CLI options, JSON config format, output format: see [references/data-collection-reference.md](references/data-collection-reference.md) §7. **Output**: Funnel conversion analysis with per-step metrics. Save to `$DATA_DIR/analysis/funnel_analysis.md`. Analysis scripts should also generate Phase 3b charts (see [references/data-visualization-guide.md](references/data-visualization-guide.md)) and save to `$DATA_DIR/charts/funnel_*.png`. --- ### Phase 4: Live Site Audit > **Purpose**: Performance audit and visual inspection of the live site. **Steps**: 1. **Screenshot the site** (desktop + mobile) using `agent-browser`. Save to `$DATA_DIR/tmp/`. 2. **Run PageSpeed Insights** (mobile + desktop) — save JSON to `$DATA_DIR/data/psi_mobile.json` and `psi_desktop.json`. Extract Core Web Vitals; thresholds in [references/metrics-glossary.md](references/metrics-glossary.md). 3. **Screenshot top landing pages** (if GA4 data available) — desktop + mobile for each of the Top 10 landing pages. 4. **Extract front-end metadata** via browser (when no source code available) — title, meta description, H1, JSON-LD, images without alt, viewport, canonical. **Write analysis scripts** in `$DATA_DIR/scripts/` to process raw PSI JSON data. Use code to extract scores and CWV metrics — do not manually read JSON. PSI data collection commands: see [references/data-collection-reference.md](references/data-collection-reference.md). **Output**: Performance scores + visual issue checklist. Save to `$DATA_DIR/analysis/site_audit.md`. Analysis scripts should also generate Phase 4 charts (see [references/data-visualization-guide.md](references/data-visualization-guide.md)) and save to `$DATA_DIR/charts/audit_*.png`. --- ### Phase 5: Source Code Review > **Purpose**: Find code-level SEO and performance issues. If `SOURCE_CODE_PATH` is configured in `.env`, analyze project source code. Skip if no source code is available. Check items detailed in the "Technical Issues" checklist in [references/metrics-glossary.md](references/metrics-glossary.md). Core focus: - **SEO**: Meta tag completeness, JSON-LD, robots.txt / sitemap.xml, image alt, H1 conventions - **Performance**: JS/CSS splitting and lazy loading, image formats and responsive images, third-party scripts, render-blocking resources - **Technical**: `<html lang>`, viewport, HTTPS, canonical URL, internal dead links **Output**: Code-level improvement checklist. Save to `$DATA_DIR/analysis/code_review.md`. --- ### Phase 5b: SEO & GEO Optimization Checklist Audit > **Purpose**: Evaluate the site's search engine and generative AI readiness. Run audit scripts against the checklist in [references/SEO-GEO-Optimization-Checklist.md](references/SEO-GEO-Optimization-Checklist.md): ```bash source "$DATA_DIR/venv/bin/activate" set -a; source "$DATA_DIR/.env"; set +a python scripts/seo_audit.py --url "$SITE_URL" --sitemap -o "$DATA_DIR/analysis/seo_audit.json" python scripts/geo_audit.py --url "$SITE_URL" --sitemap -o "$DATA_DIR/analysis/geo_audit.json" python scripts/perf_audit.py --url "$SITE_URL" --sitemap -o "$DATA_DIR/analysis/perf_audit.json" ``` Each script supports `--url`, `--sitemap`, `--pages`, `--max-pages`, `-o`. Checks: Structured Data, Meta Tags, Headings, Sitemap, AI Readability, Content Depth, Performance, Security. **Write analysis scripts** in `$DATA_DIR/scripts/` to process audit JSON results. Use code to aggregate pass/fail counts and classify by priority — do not manually read JSON. **Output**: SEO/GEO readiness checklist with pass/fail status, classified by P0-P3 priority. Save to `$DATA_DIR/analysis/seo_geo_checklist.md`. --- ### Phase 6: Generate Goal-Aligned Improvement Report > **Purpose**: Synthesize all findings into a single, actionable, goal-organized report. Organize output around the **website goals defined in Phase 0**, using the "Priority Matrix" (P0-P3) in [references/metrics-glossary.md](references/metrics-glossary.md). **Write analysis scripts** in `$DATA_DIR/scripts/` to aggregate key metrics from all phases into a summary. Use code to calculate goal achievement percentages and priority distributions. Follow the report template in [references/report-template.md](references/report-template.md). Key sections: Website Profile, Executive Summary, Goal Achievement Status, Data Overview, Key Findings, Improvement Plan (P0-P3), Missing Data, Next Steps, Execution Roadmap. Analysis scripts should also generate Phase 6 charts (executive dashboard, goal scorecard, priority distribution — see [references/data-visualization-guide.md](references/data-visualization-guide.md)). Reuse Phase 2-4 charts in Detailed Analysis. Save the report to `$DATA_DIR/analysis/improvement-report.md`. ## User Persona Analysis > **When to use**: When the user wants to understand who their users are, build data-driven user personas, or get recommendations for product iteration based on user profiling. This skill supports a complete data-driven user persona analysis workflow. Instead of following the standard Phase 0-6 improvement pipeline, use the dedicated persona analysis flow: **Steps**: 1. **Collect persona data** — run GA4 demographics, behavioral, and acquisition presets plus GSC query data. The persona-specific presets include: `demographics_age`, `demographics_gender`, `demographics_geo`, `demographics_language`, `demographics_interests`, `new_vs_returning`. Combine with existing presets (`device_breakdown`, `user_behavior`, `landing_pages`, `user_acquisition`, `top_pages`) and custom queries for time patterns and cross-segments. 2. **Segment & cluster** — apply behavioral, channel, demographic, and intent-based segmentation strategies to identify 3-5 distinct user groups. 3. **Build persona cards** — for each group, construct a data-backed persona with demographics, behavioral fingerprint, journey patterns, needs, and product iteration opportunities. 4. **Validate & extract insights** — cross-validate segments, analyze acquisition efficiency, engagement quality, conversion potential, and retention signals. 5. **Generate iteration recommendations** — map each persona to specific product, content, UX, and marketing recommendations. **Write analysis scripts** in `$DATA_DIR/scripts/` to process all persona raw data, produce segmented profiles, and generate persona charts. Save charts to `$DATA_DIR/charts/persona_*.png`. **Output**: User persona report with persona cards, cross-persona comparison, and actionable recommendations. Save to `$DATA_DIR/analysis/user-persona-report.md`. Full methodology, data collection commands, segmentation strategies, persona card template, visualization requirements, and report structure: see [references/user-persona-analysis-reference.md](references/user-persona-analysis-reference.md). --- ## Companion Skills - SEO implementation → `seo-geo` - Browser automation → `agent-browser` - Frontend redesign → `frontend-design` ## Reference Docs | Document | Contents | |----------|----------| | [references/data-collection-reference.md](references/data-collection-reference.md) | Auth setup (GSC/GA4/Bing), .env configuration, collection commands, script usage (gsc_query.py, ga4_query.py, ga4_funnel.py, bing_query.py), dimensions & metrics, CSV export instructions, custom queries, PSI collection | | [references/metrics-glossary.md](references/metrics-glossary.md) | Six analysis dimensions: thresholds, diagnostics, priority matrix | | [references/SEO-GEO-Optimization-Checklist.md](references/SEO-GEO-Optimization-Checklist.md) | SEO & GEO optimization checklist: structured data, AI readability, content depth, performance | | [references/data-visualization-guide.md](references/data-visualization-guide.md) | Chart generation patterns, type selection, per-phase chart requirements, CJK font support | | [references/website-reconnaissance-reference.md](references/website-reconnaissance-reference.md) | Browser commands, website type classification, analysis dimensions, goal profile templates | | [references/report-template.md](references/report-template.md) | Phase 6 final report Markdown template | | [references/user-persona-analysis-reference.md](references/user-persona-analysis-reference.md) | User persona analysis: five-dimension model, segmentation strategies, persona card template, GA4 demographics presets, visualization requirements, iteration recommendations |

path: scripts/perf_audit.py
content: #!/usr/bin/env python3
from __future__ import annotations

"""Page performance & security audit tool.

Checks loading speed, compression, HSTS, HTML size, HTTP version,
and other technical SEO performance signals.

Usage:
    # Audit homepage
    python perf_audit.py --url "https://example.com/"

    # Audit multiple pages
    python perf_audit.py --url "https://example.com/" --pages "/about,/pricing"

    # Audit all pages from sitemap
    python perf_audit.py --url "https://example.com/" --sitemap

    # Output to file
    python perf_audit.py --url "https://example.com/" -o perf_report.json

Reads .env from: .skills-data/google-analytics-and-search-improve/.env
Env var: SITE_URL (fallback if --url not provided)
"""

import argparse
import json
import os
import re
import ssl
import sys
import time
from pathlib import Path
from urllib.parse import urljoin, urlparse

import utils  # noqa: F401  — triggers .env loading & warning suppression

try:
    import requests
except ImportError:
    print("Error: 'requests' package required. Install with: pip install requests", file=sys.stderr)
    sys.exit(1)


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------

USER_AGENT = (
    "Mozilla/5.0 (compatible; PerfAuditBot/1.0; +https://github.com/skills/perf-audit)"
)
REQUEST_TIMEOUT = 20


def extract_sitemap_urls(sitemap_url: str) -> list[str]:
    try:
        resp = requests.get(sitemap_url, headers={"User-Agent": USER_AGENT},
                            timeout=REQUEST_TIMEOUT, allow_redirects=True)
        if resp.status_code != 200:
            return []
        content = resp.text
        urls = []
        if "<sitemapindex" in content.lower():
            sub_sitemaps = re.findall(r'<loc>\s*(.*?)\s*</loc>', content)
            for sub_url in sub_sitemaps:
                try:
                    sub_resp = requests.get(sub_url, headers={"User-Agent": USER_AGENT},
                                            timeout=REQUEST_TIMEOUT, allow_redirects=True)
                    if sub_resp.status_code == 200:
                        urls.extend(re.findall(r'<loc>\s*(.*?)\s*</loc>', sub_resp.text))
                except Exception:
                    pass
        else:
            urls = re.findall(r'<loc>\s*(.*?)\s*</loc>', content)
        return urls
    except Exception:
        return []


# ---------------------------------------------------------------------------
# Performance audit
# ---------------------------------------------------------------------------

def audit_page_performance(url: str) -> dict:
    """Measure performance metrics for a single page."""
    result = {
        "url": url,
        "status": None,
        "timing": {
            "dns_ms": None,
            "connect_ms": None,
            "tls_ms": None,
            "ttfb_ms": None,
            "download_ms": None,
            "total_ms": None,
        },
        "size": {
            "html_bytes": 0,
            "html_kb": 0,
            "transfer_bytes": 0,
            "transfer_kb": 0,
        },
        "compression": {
            "encoding": None,
            "is_compressed": False,
            "compression_ratio": None,
        },
        "headers": {
            "hsts": None,
            "hsts_ok": False,
            "content_type": None,
            "cache_control": None,
            "x_frame_options": None,
            "content_security_policy": False,
            "x_content_type_options": None,
            "server": None,
        },
        "https": {
            "is_https": url.startswith("https://"),
            "tls_version": None,
        },
        "redirect_count": 0,
        "issues": [],
        "checks": {},
    }

    try:
        # Make request with timing
        start_time = time.time()

        session = requests.Session()
        resp = session.get(
            url,
            headers={
                "User-Agent": USER_AGENT,
                "Accept-Encoding": "gzip, br, deflate",
            },
            timeout=REQUEST_TIMEOUT,
            allow_redirects=True,
        )
        total_time = time.time() - start_time

        result["status"] = resp.status_code
        result["redirect_count"] = len(resp.history)

        # Timing (approximate — for precise timing, curl is better)
        # We use elapsed from requests (excludes DNS in some cases)
        elapsed_ms = resp.elapsed.total_seconds() * 1000
        result["timing"]["total_ms"] = round(total_time * 1000, 1)
        result["timing"]["ttfb_ms"] = round(elapsed_ms, 1)

        # Size
        html_content = resp.text
        html_bytes = len(html_content.encode("utf-8"))
        result["size"]["html_bytes"] = html_bytes
        result["size"]["html_kb"] = round(html_bytes / 1024, 1)

        # Transfer size (compressed)
        transfer_bytes = len(resp.content)
        result["size"]["transfer_bytes"] = transfer_bytes
        result["size"]["transfer_kb"] = round(transfer_bytes / 1024, 1)

        # Compression
        encoding = resp.headers.get("Content-Encoding", "").lower()
        result["compression"]["encoding"] = encoding if encoding else "none"
        result["compression"]["is_compressed"] = encoding in ("gzip", "br", "deflate")
        if html_bytes > 0 and transfer_bytes > 0:
            result["compression"]["compression_ratio"] = round(transfer_bytes / html_bytes, 3)

        # Security headers
        headers = resp.headers
        hsts = headers.get("Strict-Transport-Security", "")
        result["headers"]["hsts"] = hsts if hsts else None
        result["headers"]["hsts_ok"] = bool(hsts) and "max-age=" in hsts.lower()
        result["headers"]["content_type"] = headers.get("Content-Type", "")
        result["headers"]["cache_control"] = headers.get("Cache-Control", "")
        result["headers"]["x_frame_options"] = headers.get("X-Frame-Options", "")
        result["headers"]["content_security_policy"] = bool(headers.get("Content-Security-Policy", ""))
        result["headers"]["x_content_type_options"] = headers.get("X-Content-Type-Options", "")
        result["headers"]["server"] = headers.get("Server", "")

        # TLS version
        if url.startswith("https://"):
            try:
                parsed = urlparse(url)
                hostname = parsed.hostname
                port = parsed.port or 443
                ctx = ssl.create_default_context()
                with ctx.wrap_socket(
                    __import__("socket").create_connection((hostname, port), timeout=5),
                    server_hostname=hostname,
                ) as ssock:
                    result["https"]["tls_version"] = ssock.version()
            except Exception:
                result["https"]["tls_version"] = "unknown"

        # --- Checks & Issues ---

        # Speed
        total_ms = result["timing"]["total_ms"]
        result["checks"]["fast_load"] = total_ms < 1000  # < 1s
        result["checks"]["very_fast_load"] = total_ms < 400  # < 0.4s (GEO target)

        if total_ms > 2000:
            result["issues"].append(f"Slow page load: {total_ms:.0f}ms (target < 400ms for GEO)")
        elif total_ms > 1000:
            result["issues"].append(f"Page load could be faster: {total_ms:.0f}ms (target < 400ms)")

        # Size
        result["checks"]["html_under_100kb"] = html_bytes < 102400
        if html_bytes > 102400:
            result["issues"].append(f"HTML too large: {result['size']['html_kb']}KB (target < 100KB)")

        # Compression
        result["checks"]["has_compression"] = result["compression"]["is_compressed"]
        result["checks"]["has_brotli"] = encoding == "br"
        if not result["compression"]["is_compressed"]:
            result["issues"].append("No compression detected — enable Brotli or gzip")
        elif encoding != "br":
            result["issues"].append(f"Using {encoding} compression — consider upgrading to Brotli (br)")

        # HSTS
        result["checks"]["has_hsts"] = result["headers"]["hsts_ok"]
        if not result["headers"]["hsts_ok"]:
            result["issues"].append("Missing or invalid HSTS header")

        # HTTPS
        result["checks"]["is_https"] = result["https"]["is_https"]
        if not result["https"]["is_https"]:
            result["issues"].append("Not using HTTPS — critical for SEO and trust")

        # Security headers
        result["checks"]["has_x_frame_options"] = bool(result["headers"]["x_frame_options"])
        result["checks"]["has_csp"] = result["headers"]["content_security_policy"]
        result["checks"]["has_x_content_type_options"] = bool(result["headers"]["x_content_type_options"])

        # Cache
        cache = result["headers"]["cache_control"]
        result["checks"]["has_cache_control"] = bool(cache)
        if not cache:
            result["issues"].append("No Cache-Control header — consider adding caching directives")

        # Redirects
        if result["redirect_count"] > 2:
            result["issues"].append(f"Too many redirects: {result['redirect_count']} (target ≤ 1)")

    except requests.exceptions.SSLError as exc:
        result["status"] = "ssl_error"
        result["issues"].append(f"SSL/TLS error: {exc}")
    except requests.exceptions.Timeout:
        result["status"] = "timeout"
        result["issues"].append(f"Request timed out after {REQUEST_TIMEOUT}s")
    except Exception as exc:
        result["status"] = "error"
        result["issues"].append(f"Request failed: {exc}")

    return result


# ---------------------------------------------------------------------------
# CDN detection
# ---------------------------------------------------------------------------

def detect_cdn(headers: dict) -> str | None:
    """Try to detect CDN from response headers."""
    server = headers.get("server", "").lower()
    via = headers.get("via", "").lower()
    cf_ray = headers.get("cf-ray", "")
    x_cache = headers.get("x-cache", "").lower()
    x_served_by = headers.get("x-served-by", "").lower()

    if cf_ray or "cloudflare" in server:
        return "Cloudflare"
    if "fastly" in via or "fastly" in x_served_by:
        return "Fastly"
    if "cloudfront" in via or "cloudfront" in x_cache:
        return "CloudFront (AWS)"
    if "akamai" in server or "akamai" in x_cache:
        return "Akamai"
    if "vercel" in server or "vercel" in x_cache:
        return "Vercel"
    if "netlify" in server:
        return "Netlify"
    if "bunny" in server.lower() or "bunnycdn" in server.lower():
        return "BunnyCDN"
    return None


# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(description="Page performance & security audit tool")
    parser.add_argument("--url", default=os.environ.get("SITE_URL"),
                        help="Base URL of the site (or set SITE_URL env)")
    parser.add_argument("--sitemap", action="store_true",
                        help="Audit all pages from sitemap.xml")
    parser.add_argument("--pages", help="Comma-separated page paths to audit")
    parser.add_argument("--max-pages", type=int, default=20,
                        help="Max pages to audit from sitemap (default: 20)")
    parser.add_argument("--output", "-o", help="Output file path (default: stdout)")

    args = parser.parse_args()

    if not args.url:
        print("Error: --url required or set SITE_URL", file=sys.stderr)
        sys.exit(1)

    base_url = args.url.rstrip("/")
    if not base_url.startswith("http"):
        base_url = f"https://{base_url}"

    # Determine pages
    page_urls = []
    if args.pages:
        for path in args.pages.split(","):
            path = path.strip()
            if not path.startswith("/"):
                path = f"/{path}"
            page_urls.append(f"{base_url}{path}")
    elif args.sitemap:
        sitemap_url = f"{base_url}/sitemap.xml"
        print(f"Fetching sitemap: {sitemap_url}", file=sys.stderr)
        page_urls = extract_sitemap_urls(sitemap_url)
        if not page_urls:
            print("Warning: No URLs in sitemap, falling back to homepage", file=sys.stderr)
            page_urls = [f"{base_url}/"]
        else:
            print(f"Found {len(page_urls)} pages in sitemap", file=sys.stderr)
            if len(page_urls) > args.max_pages:
                print(f"Limiting to first {args.max_pages} pages", file=sys.stderr)
                page_urls = page_urls[:args.max_pages]
    else:
        page_urls = [f"{base_url}/"]

    print(f"Auditing performance on {len(page_urls)} page(s)...", file=sys.stderr)

    page_results = []
    for i, url in enumerate(page_urls, 1):
        print(f"  [{i}/{len(page_urls)}] {url}", file=sys.stderr)
        pr = audit_page_performance(url)

        # CDN detection (from first page's headers)
        if isinstance(pr.get("headers"), dict) and i == 1:
            # Refetch headers for CDN detection
            try:
                head_resp = requests.head(url, headers={"User-Agent": USER_AGENT},
                                          timeout=10, allow_redirects=True)
                cdn = detect_cdn(dict(head_resp.headers))
            except Exception:
                cdn = None
        else:
            cdn = None

        if cdn:
            pr["cdn_detected"] = cdn
        page_results.append(pr)
        if i < len(page_urls):
            time.sleep(0.5)

    # Aggregate
    all_issues = []
    for pr in page_results:
        for issue in pr["issues"]:
            all_issues.append({"url": pr["url"], "issue": issue})

    # Stats
    load_times = [pr["timing"]["total_ms"] for pr in page_results
                  if pr["timing"]["total_ms"] is not None]
    html_sizes = [pr["size"]["html_kb"] for pr in page_results
                  if pr["size"]["html_kb"] > 0]

    stats = {}
    if load_times:
        stats["avg_load_ms"] = round(sum(load_times) / len(load_times), 1)
        stats["min_load_ms"] = round(min(load_times), 1)
        stats["max_load_ms"] = round(max(load_times), 1)
        stats["pages_under_400ms"] = sum(1 for t in load_times if t < 400)
        stats["pages_under_1000ms"] = sum(1 for t in load_times if t < 1000)
    if html_sizes:
        stats["avg_html_kb"] = round(sum(html_sizes) / len(html_sizes), 1)
        stats["max_html_kb"] = round(max(html_sizes), 1)

    # Aggregate checks
    check_keys = set()
    for pr in page_results:
        check_keys.update(pr.get("checks", {}).keys())
    agg_checks = {}
    for key in sorted(check_keys):
        values = [pr["checks"].get(key) for pr in page_results if key in pr.get("checks", {})]
        passed = sum(1 for v in values if v)
        agg_checks[key] = {"passed": passed, "total": len(values)}

    # CDN info
    cdns = [pr.get("cdn_detected") for pr in page_results if pr.get("cdn_detected")]
    cdn_detected = cdns[0] if cdns else None

    result = {
        "tool": "perf_audit",
        "base_url": base_url,
        "pages_audited": len(page_results),
        "cdn_detected": cdn_detected,
        "summary": {
            "total_issues": len(all_issues),
            "stats": stats,
            "checks": agg_checks,
        },
        "issues": all_issues,
        "pages": page_results,
    }

    output = json.dumps(result, indent=2, ensure_ascii=False, default=str)
    if args.output:
        with open(args.output, "w", encoding="utf-8") as f:
            f.write(output)
        print(f"\nOutput written to {args.output}", file=sys.stderr)
    else:
        print(output)

    # Quick summary
    print(f"\n{'='*60}", file=sys.stderr)
    print(f"Performance Audit Summary: {base_url}", file=sys.stderr)
    print(f"Pages audited: {len(page_results)}", file=sys.stderr)
    print(f"Total issues: {len(all_issues)}", file=sys.stderr)
    if stats.get("avg_load_ms"):
        print(f"Avg load time: {stats['avg_load_ms']:.0f}ms", file=sys.stderr)
        geo_pass = stats.get("pages_under_400ms", 0)
        print(f"Pages under 400ms (GEO target): {geo_pass}/{len(page_results)}", file=sys.stderr)
    if stats.get("avg_html_kb"):
        print(f"Avg HTML size: {stats['avg_html_kb']:.1f}KB", file=sys.stderr)
    if cdn_detected:
        print(f"CDN: {cdn_detected}", file=sys.stderr)
    else:
        print(f"CDN: ❌ Not detected", file=sys.stderr)
    print(f"{'='*60}", file=sys.stderr)


if __name__ == "__main__":
    main()


path: scripts/ga4_funnel.py
content: #!/usr/bin/env python3
from __future__ import annotations
"""GA4 Funnel Exploration report via the Data API v1alpha.

Uses AlphaAnalyticsDataClient.run_funnel_report() — the only GA4 exploration
type that has a dedicated API method.  Free-form explorations are already
covered by ga4_query.py (runReport); path explorations have no API support.

Usage examples:

    # Quick funnel with event names (comma-separated)
    python ga4_funnel.py --steps "first_visit,page_view,signup,purchase"

    # With custom step names
    python ga4_funnel.py \
        --steps "first_visit,page_view,signup,purchase" \
        --step-names "First Visit,View Page,Sign Up,Purchase"

    # Open funnel (users can enter at any step)
    python ga4_funnel.py --steps "page_view,add_to_cart,purchase" --open

    # Breakdown by device category
    python ga4_funnel.py --steps "page_view,signup" --breakdown deviceCategory

    # Trended funnel (shows daily trend per step)
    python ga4_funnel.py --steps "page_view,purchase" --trended

    # Use a JSON config file for complex filters
    python ga4_funnel.py --config funnel_config.json

Reads .env from: .skills-data/google-analytics-and-search-improve/.env
Env vars: GA4_PROPERTY_ID
"""

import argparse
import json
import os
import sys
from pathlib import Path

from utils import require_google_credentials

# Auto-discover credentials before importing Google client libs
require_google_credentials()


# ---------------------------------------------------------------------------
# Imports from v1alpha (funnel API is alpha-only)
# ---------------------------------------------------------------------------

from google.analytics.data_v1alpha import AlphaAnalyticsDataClient
from google.analytics.data_v1alpha.types import (
    DateRange,
    Dimension,
    Funnel,
    FunnelBreakdown,
    FunnelEventFilter,
    FunnelFieldFilter,
    FunnelFilterExpression,
    FunnelFilterExpressionList,
    FunnelStep,
    RunFunnelReportRequest,
    StringFilter,
)


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------

def _build_event_filter(event_name: str) -> FunnelFilterExpression:
    """Create a FunnelFilterExpression matching a single event name."""
    return FunnelFilterExpression(
        funnel_event_filter=FunnelEventFilter(event_name=event_name),
    )


def _build_or_event_filter(event_names: list[str]) -> FunnelFilterExpression:
    """Create a FunnelFilterExpression matching any of several event names."""
    if len(event_names) == 1:
        return _build_event_filter(event_names[0])
    return FunnelFilterExpression(
        or_group=FunnelFilterExpressionList(
            expressions=[_build_event_filter(e) for e in event_names],
        ),
    )


def _build_field_filter(
    field_name: str,
    value: str,
    match_type: str = "EXACT",
) -> FunnelFilterExpression:
    """Create a FunnelFilterExpression matching a dimension field value."""
    mt = StringFilter.MatchType[match_type.upper()]
    return FunnelFilterExpression(
        funnel_field_filter=FunnelFieldFilter(
            field_name=field_name,
            string_filter=StringFilter(match_type=mt, value=value),
        ),
    )


def build_steps_from_events(
    event_names: list[str],
    step_names: list[str] | None = None,
) -> list[FunnelStep]:
    """Build FunnelStep list from simple event name strings.

    Each element in *event_names* can contain ``|`` to denote OR logic,
    e.g. ``"first_open|first_visit"`` matches either event.
    """
    steps = []
    for i, raw in enumerate(event_names):
        events = [e.strip() for e in raw.split("|") if e.strip()]
        name = step_names[i] if step_names and i < len(step_names) else events[0]
        steps.append(
            FunnelStep(
                name=name,
                filter_expression=_build_or_event_filter(events),
            )
        )
    return steps


def build_steps_from_config(config_steps: list[dict]) -> list[FunnelStep]:
    """Build FunnelStep list from a richer JSON config.

    Each dict may have:
        name            – step display name (required)
        events          – list of event names (OR logic)
        field_filter    – {field_name, value, match_type} for dimension match
        directly_followed_by – bool
        within_duration      – str like "300s"
    """
    steps = []
    for cfg in config_steps:
        # Build filter expression
        exprs: list[FunnelFilterExpression] = []
        for ev in cfg.get("events", []):
            exprs.append(_build_event_filter(ev))
        ff = cfg.get("field_filter")
        if ff:
            exprs.append(
                _build_field_filter(
                    ff["field_name"], ff["value"], ff.get("match_type", "EXACT"),
                )
            )

        if len(exprs) == 0:
            print(f"Error: step '{cfg.get('name')}' has no events or field_filter", file=sys.stderr)
            sys.exit(1)
        elif len(exprs) == 1:
            filter_expr = exprs[0]
        else:
            filter_expr = FunnelFilterExpression(
                or_group=FunnelFilterExpressionList(expressions=exprs),
            )

        step = FunnelStep(
            name=cfg["name"],
            filter_expression=filter_expr,
        )
        if cfg.get("directly_followed_by"):
            step.is_directly_followed_by = True
        if cfg.get("within_duration"):
            step.within_duration_from_prior_step = cfg["within_duration"]

        steps.append(step)

    return steps


# ---------------------------------------------------------------------------
# Report execution
# ---------------------------------------------------------------------------

def run_funnel_report(
    property_id: str,
    steps: list[FunnelStep],
    start_date: str = "28daysAgo",
    end_date: str = "yesterday",
    is_open_funnel: bool = False,
    breakdown_dimension: str | None = None,
    trended: bool = False,
) -> dict:
    """Execute a funnel report and return structured results."""
    client = AlphaAnalyticsDataClient()

    funnel = Funnel(steps=steps, is_open_funnel=is_open_funnel)

    request = RunFunnelReportRequest(
        property=f"properties/{property_id}",
        date_ranges=[DateRange(start_date=start_date, end_date=end_date)],
        funnel=funnel,
    )

    if breakdown_dimension:
        request.funnel_breakdown = FunnelBreakdown(
            breakdown_dimension=Dimension(name=breakdown_dimension),
        )

    if trended:
        request.funnel_visualization_type = (
            RunFunnelReportRequest.FunnelVisualizationType.TRENDED_FUNNEL
        )

    response = client.run_funnel_report(request)
    return _parse_funnel_response(response)


def _parse_sub_report(sub_report) -> dict:
    """Parse a FunnelSubReport (funnel_table or funnel_visualization) into a dict."""
    dim_headers = [h.name for h in sub_report.dimension_headers]
    metric_headers = [h.name for h in sub_report.metric_headers]
    rows = []
    for row in sub_report.rows:
        r: dict = {}
        for i, dv in enumerate(row.dimension_values):
            r[dim_headers[i]] = dv.value
        for i, mv in enumerate(row.metric_values):
            val = mv.value
            try:
                val = float(val)
                if val == int(val):
                    val = int(val)
            except (ValueError, TypeError):
                pass
            r[metric_headers[i]] = val
        rows.append(r)
    return {"dimensions": dim_headers, "metrics": metric_headers, "rows": rows}


def _parse_funnel_response(response) -> dict:
    """Convert protobuf funnel response to a plain dict.

    NOTE: ``response.funnel_table`` and ``response.funnel_visualization`` are
    ``FunnelSubReport`` objects directly (they do **not** have an inner
    ``.sub_report`` attribute).
    """
    result = {}

    # --- funnel table (main data) ---
    if response.funnel_table:
        result["funnel_table"] = _parse_sub_report(response.funnel_table)

    # --- funnel visualization ---
    if response.funnel_visualization:
        result["funnel_visualization"] = _parse_sub_report(response.funnel_visualization)

    # --- sampling metadata ---
    if response.funnel_table and getattr(response.funnel_table, "metadata", None):
        meta = response.funnel_table.metadata
        if getattr(meta, "sampling_metadatas", None):
            samples = []
            for sm in meta.sampling_metadatas:
                samples.append({
                    "samples_read_count": sm.samples_read_count,
                    "sampling_space_size": sm.sampling_space_size,
                })
            result["sampling"] = samples

    return result


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(
        description="GA4 Funnel Exploration report (v1alpha API)",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""Examples:
  python ga4_funnel.py --steps "first_visit,page_view,signup,purchase"
  python ga4_funnel.py --steps "page_view,add_to_cart,purchase" --open --breakdown deviceCategory
  python ga4_funnel.py --config funnel_config.json --trended
""",
    )
    parser.add_argument(
        "--property-id",
        default=os.environ.get("GA4_PROPERTY_ID"),
        help="GA4 property ID (or set GA4_PROPERTY_ID env)",
    )
    parser.add_argument(
        "--steps",
        help='Comma-separated event names for funnel steps. '
             'Use "|" within a step for OR logic, e.g. "first_open|first_visit,page_view,purchase"',
    )
    parser.add_argument(
        "--step-names",
        help="Comma-separated display names for each step (must match --steps count)",
    )
    parser.add_argument(
        "--config",
        help="Path to a JSON config file defining funnel steps (advanced mode)",
    )
    parser.add_argument("--start-date", default="28daysAgo")
    parser.add_argument("--end-date", default="yesterday")
    parser.add_argument(
        "--open", action="store_true",
        help="Open funnel — users can enter at any step (default: closed)",
    )
    parser.add_argument(
        "--breakdown",
        help="Dimension to break down by (e.g. deviceCategory, country, sessionSource)",
    )
    parser.add_argument(
        "--trended", action="store_true",
        help="Show trended funnel (daily trend per step)",
    )
    parser.add_argument("--output", "-o", help="Output file path (default: stdout)")

    args = parser.parse_args()

    if not args.property_id:
        print("Error: --property-id required or set GA4_PROPERTY_ID", file=sys.stderr)
        sys.exit(1)

    # --- Build funnel steps ---
    if args.config:
        with open(args.config, "r", encoding="utf-8") as f:
            config = json.load(f)
        steps = build_steps_from_config(config.get("steps", config))
        # Config may also specify other options
        if isinstance(config, dict):
            if "open" in config:
                args.open = config["open"]
            if "breakdown" in config and not args.breakdown:
                args.breakdown = config["breakdown"]
            if "trended" in config:
                args.trended = config["trended"]
    elif args.steps:
        event_list = [s.strip() for s in args.steps.split(",")]
        step_names = None
        if args.step_names:
            step_names = [n.strip() for n in args.step_names.split(",")]
        steps = build_steps_from_events(event_list, step_names)
    else:
        print("Error: provide --steps or --config", file=sys.stderr)
        sys.exit(1)

    if len(steps) < 2:
        print("Error: funnel requires at least 2 steps", file=sys.stderr)
        sys.exit(1)

    # --- Run report ---
    result = run_funnel_report(
        property_id=args.property_id,
        steps=steps,
        start_date=args.start_date,
        end_date=args.end_date,
        is_open_funnel=args.open,
        breakdown_dimension=args.breakdown,
        trended=args.trended,
    )

    result["query"] = {
        "property_id": args.property_id,
        "date_range": {"start": args.start_date, "end": args.end_date},
        "is_open_funnel": args.open,
        "breakdown": args.breakdown,
        "trended": args.trended,
        "step_count": len(steps),
        "steps": [s.name for s in steps],
    }

    output = json.dumps(result, indent=2, ensure_ascii=False)
    if args.output:
        Path(args.output).parent.mkdir(parents=True, exist_ok=True)
        with open(args.output, "w", encoding="utf-8") as f:
            f.write(output)
        print(f"Output written to {args.output}", file=sys.stderr)
    else:
        print(output)


if __name__ == "__main__":
    main()


path: scripts/gsc_query.py
content: #!/usr/bin/env python3
"""Google Search Console API data extraction tool.

Usage:
    python gsc_query.py --site-url "https://example.com" --mode search_analytics \
        --start-date 2025-01-01 --end-date 2025-03-01 --dimensions query,page --limit 100

    python gsc_query.py --site-url "https://example.com" --mode sitemaps

    python gsc_query.py --site-url "https://example.com" --mode inspect \
        --inspect-url "https://example.com/page"

Reads .env from: .skills-data/google-analytics-and-search-improve/.env
Env vars: GSC_SITE_URL
"""

import argparse
import json
import os
import sys
from datetime import datetime, timedelta

from utils import require_google_credentials

from google.oauth2 import service_account
from googleapiclient.discovery import build


SCOPES = ["https://www.googleapis.com/auth/webmasters.readonly"]


def get_credentials():
    creds_path = require_google_credentials()
    return service_account.Credentials.from_service_account_file(
        creds_path, scopes=SCOPES
    )


def build_service():
    credentials = get_credentials()
    return build("searchconsole", "v1", credentials=credentials)


def query_search_analytics(service, site_url, start_date, end_date, dimensions, limit, start_row=0):
    body = {
        "startDate": start_date,
        "endDate": end_date,
        "dimensions": dimensions,
        "rowLimit": limit,
        "startRow": start_row,
    }
    response = service.searchanalytics().query(siteUrl=site_url, body=body).execute()
    return response


def fetch_all_search_analytics(service, site_url, start_date, end_date, dimensions, limit):
    """Paginate through all results up to `limit` rows."""
    all_rows = []
    start_row = 0
    page_size = min(limit, 25000)

    while len(all_rows) < limit:
        response = query_search_analytics(
            service, site_url, start_date, end_date, dimensions, page_size, start_row
        )
        rows = response.get("rows", [])
        if not rows:
            break
        all_rows.extend(rows)
        start_row += len(rows)
        if len(rows) < page_size:
            break

    return all_rows[:limit]


def list_sitemaps(service, site_url):
    response = service.sitemaps().list(siteUrl=site_url).execute()
    return response.get("sitemap", [])


def inspect_url(service, site_url, inspect_url_str):
    body = {
        "inspectionUrl": inspect_url_str,
        "siteUrl": site_url,
    }
    response = service.urlInspection().index().inspect(body=body).execute()
    return response


def main():
    parser = argparse.ArgumentParser(description="Google Search Console API query tool")
    parser.add_argument("--site-url", default=os.environ.get("GSC_SITE_URL"),
                        help="Site URL (or set GSC_SITE_URL env)")
    parser.add_argument("--mode", choices=["search_analytics", "sitemaps", "inspect"],
                        default="search_analytics")
    parser.add_argument("--start-date", default=(datetime.now() - timedelta(days=28)).strftime("%Y-%m-%d"))
    parser.add_argument("--end-date", default=(datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d"))
    parser.add_argument("--dimensions", default="query,page",
                        help="Comma-separated: query,page,country,device,date,searchAppearance")
    parser.add_argument("--limit", type=int, default=1000)
    parser.add_argument("--inspect-url", help="URL to inspect (for inspect mode)")
    parser.add_argument("--output", "-o", help="Output file path (default: stdout)")

    args = parser.parse_args()

    if not args.site_url:
        print("Error: --site-url required or set GSC_SITE_URL", file=sys.stderr)
        sys.exit(1)

    service = build_service()
    result = None

    if args.mode == "search_analytics":
        dims = [d.strip() for d in args.dimensions.split(",")]
        rows = fetch_all_search_analytics(
            service, args.site_url, args.start_date, args.end_date, dims, args.limit
        )
        result = {
            "mode": "search_analytics",
            "site_url": args.site_url,
            "date_range": {"start": args.start_date, "end": args.end_date},
            "dimensions": dims,
            "total_rows": len(rows),
            "rows": rows,
        }
    elif args.mode == "sitemaps":
        sitemaps = list_sitemaps(service, args.site_url)
        result = {
            "mode": "sitemaps",
            "site_url": args.site_url,
            "sitemaps": sitemaps,
        }
    elif args.mode == "inspect":
        if not args.inspect_url:
            print("Error: --inspect-url required for inspect mode", file=sys.stderr)
            sys.exit(1)
        inspection = inspect_url(service, args.site_url, args.inspect_url)
        result = {
            "mode": "inspect",
            "site_url": args.site_url,
            "inspect_url": args.inspect_url,
            "result": inspection,
        }

    output = json.dumps(result, indent=2, ensure_ascii=False)
    if args.output:
        with open(args.output, "w", encoding="utf-8") as f:
            f.write(output)
        print(f"Output written to {args.output}", file=sys.stderr)
    else:
        print(output)


if __name__ == "__main__":
    main()


path: scripts/pyproject.toml
content: [project]
name = "google-analytics-and-search-improve"
version = "0.1.0"
description = "Scripts for Google Analytics, Search Console, and Bing Webmaster data collection & site auditing"
requires-python = ">=3.12"
dependencies = [
    "google-api-python-client",
    "google-auth",
    "google-analytics-data",
    "python-dotenv",
    "requests",
    "matplotlib",
]


path: scripts/seo_audit.py
content: #!/usr/bin/env python3
from __future__ import annotations

"""SEO audit tool — checks JSON-LD, meta tags, headings, sitemap, Open Graph, etc.

Usage:
    # Audit a single page
    python seo_audit.py --url "https://example.com/"

    # Audit all pages from sitemap
    python seo_audit.py --url "https://example.com/" --sitemap

    # Audit specific pages (comma-separated)
    python seo_audit.py --url "https://example.com/" --pages "/about,/pricing,/docs"

    # Output to file
    python seo_audit.py --url "https://example.com/" --sitemap -o report.json

Reads .env from: .skills-data/google-analytics-and-search-improve/.env
Env var: SITE_URL (fallback if --url not provided)
"""

import argparse
import json
import os
import re
import sys
import time
from html.parser import HTMLParser
from pathlib import Path
from urllib.parse import urljoin, urlparse

import utils  # noqa: F401  — triggers .env loading & warning suppression

try:
    import requests
except ImportError:
    print("Error: 'requests' package required. Install with: pip install requests", file=sys.stderr)
    sys.exit(1)


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------

USER_AGENT = (
    "Mozilla/5.0 (compatible; SEOAuditBot/1.0; +https://github.com/skills/seo-audit)"
)
REQUEST_TIMEOUT = 15


def fetch(url: str, *, follow_redirects: bool = True) -> requests.Response:
    """Fetch a URL with standard headers."""
    return requests.get(
        url,
        headers={"User-Agent": USER_AGENT, "Accept-Encoding": "gzip, br"},
        timeout=REQUEST_TIMEOUT,
        allow_redirects=follow_redirects,
    )


def safe_fetch(url: str) -> requests.Response | None:
    """Fetch with error handling — returns None on failure."""
    try:
        return fetch(url)
    except Exception as exc:
        print(f"  ⚠️  Failed to fetch {url}: {exc}", file=sys.stderr)
        return None


# ---------------------------------------------------------------------------
# Extraction helpers
# ---------------------------------------------------------------------------

def extract_json_ld(html: str) -> list[dict]:
    """Extract all JSON-LD blocks from HTML."""
    pattern = re.compile(
        r'<script\s+type=["\']application/ld\+json["\'][^>]*>(.*?)</script>',
        re.DOTALL | re.IGNORECASE,
    )
    results = []
    for match in pattern.finditer(html):
        raw = match.group(1).strip()
        if not raw:
            continue
        try:
            data = json.loads(raw)
            # Handle @graph arrays
            if isinstance(data, dict) and "@graph" in data:
                for item in data["@graph"]:
                    results.append(item)
            elif isinstance(data, list):
                results.extend(data)
            else:
                results.append(data)
        except json.JSONDecodeError:
            results.append({"_parse_error": True, "_raw": raw[:200]})
    return results


def extract_meta_tags(html: str) -> dict:
    """Extract meta tags, title, canonical, hreflang, and OG tags."""
    result = {
        "title": None,
        "title_length": 0,
        "description": None,
        "description_length": 0,
        "canonical": None,
        "hreflang": [],
        "open_graph": {},
        "twitter": {},
        "robots": None,
        "viewport": None,
    }

    # Title
    m = re.search(r'<title[^>]*>([^<]*)</title>', html, re.IGNORECASE)
    if m:
        result["title"] = m.group(1).strip()
        result["title_length"] = len(result["title"])

    # Meta description
    m = re.search(
        r'<meta\s+[^>]*name=["\']description["\'][^>]*content=["\']([^"\']*)["\']',
        html, re.IGNORECASE,
    )
    if not m:
        m = re.search(
            r'<meta\s+[^>]*content=["\']([^"\']*)["\'][^>]*name=["\']description["\']',
            html, re.IGNORECASE,
        )
    if m:
        result["description"] = m.group(1).strip()
        result["description_length"] = len(result["description"])

    # Canonical
    m = re.search(r'<link\s+[^>]*rel=["\']canonical["\'][^>]*href=["\']([^"\']*)["\']', html, re.IGNORECASE)
    if not m:
        m = re.search(r'<link\s+[^>]*href=["\']([^"\']*)["\'][^>]*rel=["\']canonical["\']', html, re.IGNORECASE)
    if m:
        result["canonical"] = m.group(1).strip()

    # Hreflang
    for hm in re.finditer(
        r'<link\s+[^>]*rel=["\']alternate["\'][^>]*hreflang=["\']([^"\']*)["\'][^>]*href=["\']([^"\']*)["\']',
        html, re.IGNORECASE,
    ):
        result["hreflang"].append({"lang": hm.group(1), "href": hm.group(2)})

    # Open Graph
    for og in re.finditer(
        r'<meta\s+[^>]*property=["\']og:([^"\']*)["\'][^>]*content=["\']([^"\']*)["\']',
        html, re.IGNORECASE,
    ):
        result["open_graph"][f"og:{og.group(1)}"] = og.group(2)
    # Also match reversed attribute order
    for og in re.finditer(
        r'<meta\s+[^>]*content=["\']([^"\']*)["\'][^>]*property=["\']og:([^"\']*)["\']',
        html, re.IGNORECASE,
    ):
        result["open_graph"][f"og:{og.group(2)}"] = og.group(1)

    # Twitter Card
    for tw in re.finditer(
        r'<meta\s+[^>]*(?:name|property)=["\']twitter:([^"\']*)["\'][^>]*content=["\']([^"\']*)["\']',
        html, re.IGNORECASE,
    ):
        result["twitter"][f"twitter:{tw.group(1)}"] = tw.group(2)
    for tw in re.finditer(
        r'<meta\s+[^>]*content=["\']([^"\']*)["\'][^>]*(?:name|property)=["\']twitter:([^"\']*)["\']',
        html, re.IGNORECASE,
    ):
        result["twitter"][f"twitter:{tw.group(2)}"] = tw.group(1)

    # Robots
    m = re.search(
        r'<meta\s+[^>]*name=["\']robots["\'][^>]*content=["\']([^"\']*)["\']',
        html, re.IGNORECASE,
    )
    if m:
        result["robots"] = m.group(1).strip()

    # Viewport
    m = re.search(
        r'<meta\s+[^>]*name=["\']viewport["\'][^>]*content=["\']([^"\']*)["\']',
        html, re.IGNORECASE,
    )
    if m:
        result["viewport"] = m.group(1).strip()

    return result


def extract_headings(html: str) -> list[dict]:
    """Extract all headings (h1-h6) from HTML."""
    headings = []
    for m in re.finditer(r'<(h[1-6])[^>]*>(.*?)</\1>', html, re.IGNORECASE | re.DOTALL):
        level = m.group(1).lower()
        text = re.sub(r'<[^>]+>', '', m.group(2)).strip()
        if text:
            headings.append({
                "level": level,
                "text": text,
                "is_question": text.rstrip().endswith("?"),
            })
    return headings


def extract_sitemap_urls(sitemap_url: str) -> list[str]:
    """Parse sitemap.xml and return all <loc> URLs. Handles sitemap index files."""
    resp = safe_fetch(sitemap_url)
    if not resp or resp.status_code != 200:
        return []

    content = resp.text
    urls = []

    # Check if it's a sitemap index
    if "<sitemapindex" in content.lower():
        sub_sitemaps = re.findall(r'<loc>\s*(.*?)\s*</loc>', content)
        for sub_url in sub_sitemaps:
            sub_resp = safe_fetch(sub_url)
            if sub_resp and sub_resp.status_code == 200:
                urls.extend(re.findall(r'<loc>\s*(.*?)\s*</loc>', sub_resp.text))
    else:
        urls = re.findall(r'<loc>\s*(.*?)\s*</loc>', content)

    return urls


# ---------------------------------------------------------------------------
# Audit functions
# ---------------------------------------------------------------------------

def audit_page(url: str) -> dict:
    """Run full SEO audit on a single page."""
    result = {
        "url": url,
        "status": None,
        "redirect_chain": [],
        "json_ld": {"blocks": [], "types": [], "count": 0, "issues": []},
        "meta": {},
        "headings": [],
        "heading_stats": {},
        "issues": [],
        "checks": {},
    }

    # Fetch with redirect tracking
    try:
        resp = requests.get(
            url,
            headers={"User-Agent": USER_AGENT},
            timeout=REQUEST_TIMEOUT,
            allow_redirects=True,
        )
        result["status"] = resp.status_code
        if resp.history:
            result["redirect_chain"] = [
                {"url": r.url, "status": r.status_code} for r in resp.history
            ]
    except Exception as exc:
        result["status"] = "error"
        result["issues"].append(f"Fetch error: {exc}")
        return result

    html = resp.text

    # --- JSON-LD ---
    ld_blocks = extract_json_ld(html)
    result["json_ld"]["blocks"] = ld_blocks
    result["json_ld"]["count"] = len(ld_blocks)
    result["json_ld"]["types"] = [
        b.get("@type", "unknown") for b in ld_blocks if isinstance(b, dict) and not b.get("_parse_error")
    ]

    # JSON-LD checks
    has_website = any(b.get("@type") == "WebSite" for b in ld_blocks if isinstance(b, dict))
    has_webpage = any(b.get("@type") == "WebPage" for b in ld_blocks if isinstance(b, dict))
    has_breadcrumb = any(b.get("@type") == "BreadcrumbList" for b in ld_blocks if isinstance(b, dict))
    has_faq = any(b.get("@type") == "FAQPage" for b in ld_blocks if isinstance(b, dict))
    has_howto = any(b.get("@type") == "HowTo" for b in ld_blocks if isinstance(b, dict))

    result["checks"]["has_json_ld"] = len(ld_blocks) > 0
    result["checks"]["has_website_schema"] = has_website
    result["checks"]["has_webpage_schema"] = has_webpage
    result["checks"]["has_breadcrumb_schema"] = has_breadcrumb
    result["checks"]["has_faq_schema"] = has_faq
    result["checks"]["has_howto_schema"] = has_howto

    if len(ld_blocks) == 0:
        result["json_ld"]["issues"].append("No JSON-LD blocks found")
    if not has_website:
        result["json_ld"]["issues"].append("Missing WebSite schema")
    if not has_webpage:
        result["json_ld"]["issues"].append("Missing WebPage schema")
    if not has_breadcrumb:
        result["json_ld"]["issues"].append("Missing BreadcrumbList schema")

    # Check for parse errors
    parse_errors = [b for b in ld_blocks if isinstance(b, dict) and b.get("_parse_error")]
    if parse_errors:
        result["json_ld"]["issues"].append(f"{len(parse_errors)} JSON-LD block(s) failed to parse")

    # --- Meta tags ---
    meta = extract_meta_tags(html)
    result["meta"] = meta

    # Meta checks
    result["checks"]["has_title"] = bool(meta["title"])
    result["checks"]["title_length_ok"] = 30 <= meta["title_length"] <= 60 if meta["title"] else False
    result["checks"]["has_description"] = bool(meta["description"])
    result["checks"]["description_length_ok"] = 70 <= meta["description_length"] <= 160 if meta["description"] else False
    result["checks"]["has_canonical"] = bool(meta["canonical"])
    result["checks"]["has_hreflang"] = len(meta["hreflang"]) > 0
    result["checks"]["has_og_title"] = "og:title" in meta["open_graph"]
    result["checks"]["has_og_description"] = "og:description" in meta["open_graph"]
    result["checks"]["has_og_image"] = "og:image" in meta["open_graph"]
    result["checks"]["has_twitter_card"] = bool(meta["twitter"])
    result["checks"]["has_viewport"] = bool(meta["viewport"])

    if not meta["title"]:
        result["issues"].append("Missing <title> tag")
    elif meta["title_length"] > 60:
        result["issues"].append(f"Title too long ({meta['title_length']} chars, max 60)")
    elif meta["title_length"] < 30:
        result["issues"].append(f"Title too short ({meta['title_length']} chars, min 30)")

    if not meta["description"]:
        result["issues"].append("Missing meta description")
    elif meta["description_length"] > 160:
        result["issues"].append(f"Description too long ({meta['description_length']} chars, max 160)")

    if not meta["canonical"]:
        result["issues"].append("Missing canonical URL")

    if "og:image" not in meta["open_graph"]:
        result["issues"].append("Missing og:image")

    # --- Headings ---
    headings = extract_headings(html)
    result["headings"] = headings

    h1_count = sum(1 for h in headings if h["level"] == "h1")
    question_headings = sum(1 for h in headings if h["is_question"])
    total_headings = len(headings)

    result["heading_stats"] = {
        "total": total_headings,
        "h1_count": h1_count,
        "question_style_count": question_headings,
        "question_ratio": round(question_headings / total_headings, 2) if total_headings > 0 else 0,
    }

    result["checks"]["has_h1"] = h1_count > 0
    result["checks"]["single_h1"] = h1_count == 1
    result["checks"]["has_question_headings"] = question_headings > 0

    if h1_count == 0:
        result["issues"].append("No H1 heading found")
    elif h1_count > 1:
        result["issues"].append(f"Multiple H1 headings found ({h1_count})")

    if total_headings > 3 and question_headings == 0:
        result["issues"].append("No question-style headings (H2/H3) — hurts GEO visibility")

    # --- SSR check ---
    html_size = len(html.encode("utf-8"))
    result["checks"]["html_size_bytes"] = html_size
    result["checks"]["ssr_likely_ok"] = html_size > 10000  # >10KB likely has real content

    if html_size < 500:
        result["issues"].append(f"HTML extremely small ({html_size} bytes) — possible empty shell / CSR-only")

    # --- Redirect check ---
    if result["redirect_chain"]:
        result["issues"].append(
            f"Redirect detected: {' → '.join(r['url'] for r in result['redirect_chain'])} → {url}"
        )

    return result


def audit_sitemap(base_url: str) -> dict:
    """Check sitemap.xml presence and structure."""
    sitemap_url = urljoin(base_url, "/sitemap.xml")
    result = {
        "url": sitemap_url,
        "exists": False,
        "page_count": 0,
        "has_lastmod": False,
        "declared_in_robots": False,
        "pages": [],
    }

    resp = safe_fetch(sitemap_url)
    if resp and resp.status_code == 200:
        result["exists"] = True
        urls = extract_sitemap_urls(sitemap_url)
        result["page_count"] = len(urls)
        result["pages"] = urls
        result["has_lastmod"] = "<lastmod>" in resp.text.lower()

    # Check robots.txt for sitemap declaration
    robots_resp = safe_fetch(urljoin(base_url, "/robots.txt"))
    if robots_resp and robots_resp.status_code == 200:
        result["declared_in_robots"] = "sitemap:" in robots_resp.text.lower()

    return result


# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(description="SEO audit tool for website pages")
    parser.add_argument("--url", default=os.environ.get("SITE_URL"),
                        help="Base URL of the site (or set SITE_URL env)")
    parser.add_argument("--sitemap", action="store_true",
                        help="Audit all pages from sitemap.xml")
    parser.add_argument("--pages", help="Comma-separated page paths to audit (e.g. /about,/pricing)")
    parser.add_argument("--max-pages", type=int, default=50,
                        help="Max pages to audit from sitemap (default: 50)")
    parser.add_argument("--output", "-o", help="Output file path (default: stdout)")

    args = parser.parse_args()

    if not args.url:
        print("Error: --url required or set SITE_URL", file=sys.stderr)
        sys.exit(1)

    base_url = args.url.rstrip("/")
    if not base_url.startswith("http"):
        base_url = f"https://{base_url}"

    # Determine pages to audit
    page_urls = []

    if args.pages:
        for path in args.pages.split(","):
            path = path.strip()
            if not path.startswith("/"):
                path = f"/{path}"
            page_urls.append(f"{base_url}{path}")
    elif args.sitemap:
        sitemap_url = f"{base_url}/sitemap.xml"
        print(f"Fetching sitemap: {sitemap_url}", file=sys.stderr)
        page_urls = extract_sitemap_urls(sitemap_url)
        if not page_urls:
            print("Warning: No URLs found in sitemap, falling back to homepage", file=sys.stderr)
            page_urls = [f"{base_url}/"]
        else:
            print(f"Found {len(page_urls)} pages in sitemap", file=sys.stderr)
            if len(page_urls) > args.max_pages:
                print(f"Limiting to first {args.max_pages} pages", file=sys.stderr)
                page_urls = page_urls[:args.max_pages]
    else:
        page_urls = [f"{base_url}/"]

    # Run audits
    print(f"Auditing {len(page_urls)} page(s)...", file=sys.stderr)

    page_results = []
    for i, url in enumerate(page_urls, 1):
        print(f"  [{i}/{len(page_urls)}] {url}", file=sys.stderr)
        page_results.append(audit_page(url))
        if i < len(page_urls):
            time.sleep(0.5)  # polite delay

    # Sitemap audit
    sitemap_result = audit_sitemap(base_url)

    # Aggregate summary
    all_issues = []
    all_checks = {}
    for pr in page_results:
        for issue in pr["issues"]:
            all_issues.append({"url": pr["url"], "issue": issue})
        for issue in pr["json_ld"].get("issues", []):
            all_issues.append({"url": pr["url"], "issue": f"[JSON-LD] {issue}"})

    # Aggregate pass/fail counts
    check_keys = set()
    for pr in page_results:
        check_keys.update(pr["checks"].keys())
    for key in sorted(check_keys):
        if key == "html_size_bytes":
            continue
        values = [pr["checks"].get(key) for pr in page_results if key in pr["checks"]]
        passed = sum(1 for v in values if v)
        all_checks[key] = {"passed": passed, "total": len(values)}

    result = {
        "tool": "seo_audit",
        "base_url": base_url,
        "pages_audited": len(page_results),
        "summary": {
            "total_issues": len(all_issues),
            "checks": all_checks,
        },
        "sitemap": sitemap_result,
        "issues": all_issues,
        "pages": page_results,
    }

    output = json.dumps(result, indent=2, ensure_ascii=False, default=str)
    if args.output:
        with open(args.output, "w", encoding="utf-8") as f:
            f.write(output)
        print(f"\nOutput written to {args.output}", file=sys.stderr)
    else:
        print(output)

    # Print quick summary to stderr
    print(f"\n{'='*60}", file=sys.stderr)
    print(f"SEO Audit Summary: {base_url}", file=sys.stderr)
    print(f"Pages audited: {len(page_results)}", file=sys.stderr)
    print(f"Total issues: {len(all_issues)}", file=sys.stderr)
    if sitemap_result["exists"]:
        print(f"Sitemap: ✅ {sitemap_result['page_count']} pages", file=sys.stderr)
    else:
        print(f"Sitemap: ❌ Not found", file=sys.stderr)
    print(f"{'='*60}", file=sys.stderr)


if __name__ == "__main__":
    main()


path: scripts/ga4_query.py
content: #!/usr/bin/env python3
"""Google Analytics 4 Data API query tool.

Usage:
    python ga4_query.py --property-id 123456789 --preset traffic_overview \
        --start-date 2025-01-01 --end-date 2025-03-01

    python ga4_query.py --property-id 123456789 --preset top_pages --limit 50

    python ga4_query.py --property-id 123456789 \
        --dimensions pagePath,deviceCategory \
        --metrics sessions,bounceRate \
        --start-date 30daysAgo --end-date yesterday \
        --order-by="-sessions"

Reads .env from: .skills-data/google-analytics-and-search-improve/.env
Env vars: GA4_PROPERTY_ID
"""

import argparse
import json
import os
import sys

from utils import require_google_credentials

# Auto-discover credentials before importing Google client libs
# (BetaAnalyticsDataClient reads GOOGLE_APPLICATION_CREDENTIALS on init)
require_google_credentials()

from google.analytics.data_v1beta import BetaAnalyticsDataClient
from google.analytics.data_v1beta.types import (
    DateRange,
    Dimension,
    Filter,
    FilterExpression,
    Metric,
    MetricType,
    OrderBy,
    RunReportRequest,
)


PRESETS = {
    "traffic_overview": {
        "dimensions": ["date"],
        "metrics": ["sessions", "totalUsers", "newUsers", "screenPageViews",
                     "bounceRate", "averageSessionDuration", "engagementRate"],
        "order_by": "date",
    },
    "top_pages": {
        "dimensions": ["pagePath", "pageTitle"],
        "metrics": ["screenPageViews", "sessions", "totalUsers",
                     "bounceRate", "averageSessionDuration", "engagementRate"],
        "order_by": "-screenPageViews",
    },
    "user_acquisition": {
        "dimensions": ["sessionDefaultChannelGroup", "sessionSource", "sessionMedium"],
        "metrics": ["sessions", "totalUsers", "newUsers",
                     "bounceRate", "engagementRate", "conversions"],
        "order_by": "-sessions",
    },
    "device_breakdown": {
        "dimensions": ["deviceCategory", "operatingSystem", "browser"],
        "metrics": ["sessions", "totalUsers", "bounceRate",
                     "averageSessionDuration", "engagementRate"],
        "order_by": "-sessions",
    },
    "geo_distribution": {
        "dimensions": ["country", "city"],
        "metrics": ["sessions", "totalUsers", "bounceRate", "engagementRate"],
        "order_by": "-sessions",
    },
    "landing_pages": {
        "dimensions": ["landingPage"],
        "metrics": ["sessions", "totalUsers", "bounceRate",
                     "averageSessionDuration", "engagementRate", "conversions"],
        "order_by": "-sessions",
    },
    "user_behavior": {
        "dimensions": ["pagePath"],
        "metrics": ["screenPageViews", "engagedSessions", "engagementRate",
                     "averageSessionDuration", "eventCount"],
        "order_by": "-screenPageViews",
    },
    "conversion_events": {
        "dimensions": ["eventName", "date"],
        "metrics": ["eventCount", "totalUsers", "conversions"],
        "order_by": "-eventCount",
    },
    # ── User Persona Presets ──────────────────────────────────
    "demographics_age": {
        "dimensions": ["userAgeBracket"],
        "metrics": ["totalUsers", "sessions", "bounceRate",
                     "engagementRate", "averageSessionDuration", "conversions"],
        "order_by": "-totalUsers",
    },
    "demographics_gender": {
        "dimensions": ["userGender"],
        "metrics": ["totalUsers", "sessions", "bounceRate",
                     "engagementRate", "averageSessionDuration", "conversions"],
        "order_by": "-totalUsers",
    },
    "demographics_geo": {
        "dimensions": ["country", "city"],
        "metrics": ["totalUsers", "sessions", "bounceRate",
                     "engagementRate", "conversions"],
        "order_by": "-totalUsers",
    },
    "demographics_language": {
        "dimensions": ["language"],
        "metrics": ["totalUsers", "sessions", "bounceRate",
                     "engagementRate"],
        "order_by": "-totalUsers",
    },
    "demographics_interests": {
        "dimensions": ["brandingInterest"],
        "metrics": ["totalUsers", "sessions", "bounceRate",
                     "engagementRate", "conversions"],
        "order_by": "-totalUsers",
    },
    "new_vs_returning": {
        "dimensions": ["newVsReturning"],
        "metrics": ["totalUsers", "sessions", "sessionsPerUser",
                     "bounceRate", "engagementRate", "averageSessionDuration",
                     "screenPageViewsPerSession", "conversions"],
        "order_by": "-totalUsers",
    },
}


def build_client():
    return BetaAnalyticsDataClient()


def build_order_by(field_name, dimension_names):
    descending = field_name.startswith("-")
    name = field_name.lstrip("-")
    if name in dimension_names:
        return OrderBy(
            dimension=OrderBy.DimensionOrderBy(dimension_name=name),
            desc=descending,
        )
    return OrderBy(
        metric=OrderBy.MetricOrderBy(metric_name=name),
        desc=descending,
    )


def run_report(client, property_id, dimensions, metrics, start_date, end_date,
               limit=1000, order_by_field=None):
    request = RunReportRequest(
        property=f"properties/{property_id}",
        dimensions=[Dimension(name=d) for d in dimensions],
        metrics=[Metric(name=m) for m in metrics],
        date_ranges=[DateRange(start_date=start_date, end_date=end_date)],
        limit=limit,
    )
    if order_by_field:
        request.order_bys = [build_order_by(order_by_field, dimensions)]

    return client.run_report(request)


def response_to_dict(response):
    dim_headers = [h.name for h in response.dimension_headers]
    metric_headers = [h.name for h in response.metric_headers]
    metric_types = [MetricType(h.type_).name for h in response.metric_headers]

    rows = []
    for row in response.rows:
        row_data = {}
        for i, dv in enumerate(row.dimension_values):
            row_data[dim_headers[i]] = dv.value
        for i, mv in enumerate(row.metric_values):
            val = mv.value
            if metric_types[i] in ("TYPE_INTEGER", "TYPE_FLOAT", "TYPE_SECONDS", "TYPE_CURRENCY"):
                try:
                    val = float(val)
                    if val == int(val):
                        val = int(val)
                except ValueError:
                    pass
            row_data[metric_headers[i]] = val
        rows.append(row_data)

    return {
        "row_count": response.row_count,
        "dimensions": dim_headers,
        "metrics": metric_headers,
        "rows": rows,
    }


def main():
    parser = argparse.ArgumentParser(description="GA4 Data API query tool")
    parser.add_argument("--property-id", default=os.environ.get("GA4_PROPERTY_ID"),
                        help="GA4 property ID (or set GA4_PROPERTY_ID env)")
    parser.add_argument("--preset", choices=list(PRESETS.keys()),
                        help="Use a predefined query template")
    parser.add_argument("--dimensions", help="Comma-separated dimension names")
    parser.add_argument("--metrics", help="Comma-separated metric names")
    parser.add_argument("--start-date", default="28daysAgo")
    parser.add_argument("--end-date", default="yesterday")
    parser.add_argument("--limit", type=int, default=1000)
    parser.add_argument("--order-by",
                        help='Metric to order by (prefix with - for desc; quote in shell: --order-by="-sessions")')
    parser.add_argument("--output", "-o", help="Output file path (default: stdout)")

    args = parser.parse_args()

    if not args.property_id:
        print("Error: --property-id required or set GA4_PROPERTY_ID", file=sys.stderr)
        sys.exit(1)

    if args.preset:
        preset = PRESETS[args.preset]
        dimensions = preset["dimensions"]
        metrics = preset["metrics"]
        order_by = args.order_by or preset.get("order_by")
    elif args.dimensions and args.metrics:
        dimensions = [d.strip() for d in args.dimensions.split(",")]
        metrics = [m.strip() for m in args.metrics.split(",")]
        order_by = args.order_by
    else:
        print("Error: provide --preset or both --dimensions and --metrics", file=sys.stderr)
        sys.exit(1)

    client = build_client()
    response = run_report(
        client, args.property_id, dimensions, metrics,
        args.start_date, args.end_date, args.limit, order_by
    )

    result = response_to_dict(response)
    result["query"] = {
        "property_id": args.property_id,
        "preset": args.preset,
        "date_range": {"start": args.start_date, "end": args.end_date},
        "dimensions": dimensions,
        "metrics": metrics,
    }

    output = json.dumps(result, indent=2, ensure_ascii=False)
    if args.output:
        with open(args.output, "w", encoding="utf-8") as f:
            f.write(output)
        print(f"Output written to {args.output}", file=sys.stderr)
    else:
        print(output)


if __name__ == "__main__":
    main()


path: scripts/utils.py
content: """Shared utilities for all skill scripts.

Centralizes:
- Data directory discovery (.skills-data/google-analytics-and-search-improve/)
- .env loading
- Google Service Account credential auto-discovery
- Warning suppression for clean output
"""

import os
import sys
import warnings
from pathlib import Path

# ---------------------------------------------------------------------------
# Suppress noisy warnings before any third-party imports
# ---------------------------------------------------------------------------
# FutureWarning: Python 3.9 EOL notices from google-auth / google-api-core
# NotOpenSSLWarning: urllib3 v2 + LibreSSL on macOS
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", message=".*urllib3.*OpenSSL.*")

from dotenv import load_dotenv


# ---------------------------------------------------------------------------
# Data directory discovery
# ---------------------------------------------------------------------------

_SKILL_DIR_NAME = "google-analytics-and-search-improve"


def find_data_dir() -> Path | None:
    """Locate .skills-data/google-analytics-and-search-improve/ by walking up
    from this script's directory toward the filesystem root.

    Returns the Path if found, or None.
    """
    d = Path(__file__).resolve().parent
    while d != d.parent:
        candidate = d / ".skills-data" / _SKILL_DIR_NAME
        if candidate.is_dir():
            return candidate
        d = d.parent
    return None


DATA_DIR: Path | None = find_data_dir()


# ---------------------------------------------------------------------------
# .env loading (runs once at import time)
# ---------------------------------------------------------------------------

if DATA_DIR:
    _env_path = DATA_DIR / ".env"
    if _env_path.exists():
        load_dotenv(_env_path)


# ---------------------------------------------------------------------------
# Google Service Account credential discovery
# ---------------------------------------------------------------------------

def find_google_credentials() -> str | None:
    """Auto-discover a Google Service Account JSON key.

    Resolution order:
    1. ``GOOGLE_APPLICATION_CREDENTIALS`` env var (if set and file exists)
    2. First ``*.json`` file (alphabetically) inside ``DATA_DIR/configs/``

    When a key is found via (2), ``GOOGLE_APPLICATION_CREDENTIALS`` is set in
    the environment so that Google client libraries that read the env var
    (e.g. ``BetaAnalyticsDataClient``) can pick it up automatically.

    Returns the path string, or None if no key is found.
    """
    # 1. Explicit env var takes priority
    explicit = os.environ.get("GOOGLE_APPLICATION_CREDENTIALS")
    if explicit and Path(explicit).is_file():
        return explicit

    # 2. Auto-discover from configs/
    if DATA_DIR:
        configs_dir = DATA_DIR / "configs"
        if configs_dir.is_dir():
            json_files = sorted(configs_dir.glob("*.json"))
            if json_files:
                path = str(json_files[0])
                os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = path
                return path

    return None


def require_google_credentials() -> str:
    """Like :func:`find_google_credentials` but exits with an error message
    when no credentials are found."""
    path = find_google_credentials()
    if not path:
        print(
            "Error: No Google Service Account JSON key found.\n"
            "Place your *.json key file in: .skills-data/google-analytics-and-search-improve/configs/",
            file=sys.stderr,
        )
        sys.exit(1)
    return path


path: scripts/bing_query.py
content: #!/usr/bin/env python3
"""Bing Webmaster Tools API data extraction tool.

Usage:
    python bing_query.py --mode query_stats -o bing_queries.json
    python bing_query.py --mode page_stats -o bing_pages.json
    python bing_query.py --mode rank_traffic -o bing_traffic.json
    python bing_query.py --mode query_detail --query "search term"
    python bing_query.py --mode page_detail --page "https://example.com/page"
    python bing_query.py --mode keyword --query "keyword" --country us
    python bing_query.py --mode links -o bing_links.json
    python bing_query.py --mode crawl_stats -o bing_crawl.json

Reads .env from: .skills-data/google-analytics-and-search-improve/.env
Env vars: BING_WEBMASTER_API_KEY, SITE_URL
"""

import argparse
import json
import os
import sys
from datetime import datetime, timedelta

import utils  # noqa: F401  — triggers .env loading & warning suppression
import requests


BASE_URL = "https://ssl.bing.com/webmaster/api.svc/json"


def get_api_key():
    [REDACTED_SECRET]"BING_WEBMASTER_API_KEY")
    if not [REDACTED_SECRET]"Error: BING_WEBMASTER_API_KEY not set", file=sys.stderr)
        sys.exit(1)
    return api_key


def api_get(endpoint, params=None):
    """Make a GET request to the Bing Webmaster API."""
    [REDACTED_SECRET])
    url = f"{BASE_URL}/{endpoint}"
    if params is None:
        params = {}
    params["apikey"] = api_key
    response = requests.get(url, params=params, timeout=30)
    if response.status_code != 200:
        print(f"Error: HTTP {response.status_code} — {response.text}", file=sys.stderr)
        sys.exit(1)
    return response.json()


def get_query_stats(site_url):
    """Get traffic statistics for top queries."""
    data = api_get("GetQueryStats", {"siteUrl": site_url})
    return data


def get_page_stats(site_url):
    """Get traffic statistics for top pages."""
    data = api_get("GetPageStats", {"siteUrl": site_url})
    return data


def get_rank_and_traffic_stats(site_url):
    """Get overall rank and traffic stats (daily impressions/clicks)."""
    data = api_get("GetRankAndTrafficStats", {"siteUrl": site_url})
    return data


def get_query_traffic_stats(site_url, query):
    """Get detailed traffic stats for a specific query."""
    data = api_get("GetQueryTrafficStats", {
        "siteUrl": site_url,
        "query": query,
    })
    return data


def get_page_query_stats(site_url, page_url):
    """Get query stats for a specific page."""
    data = api_get("GetPageQueryStats", {
        "siteUrl": site_url,
        "pageUrl": page_url,
    })
    return data


def get_query_page_detail_stats(site_url, query, page_url):
    """Get detailed stats for a specific query + page combination."""
    data = api_get("GetQueryPageDetailStats", {
        "siteUrl": site_url,
        "query": query,
        "pageUrl": page_url,
    })
    return data


def get_keyword(site_url, keyword, country, start_date, end_date):
    """Get keyword impression data for a country and date range."""
    data = api_get("GetKeyword", {
        "siteUrl": site_url,
        "query": keyword,
        "country": country,
        "stringStartDate": start_date,
        "stringEndDate": end_date,
    })
    return data


def get_related_keywords(site_url, keyword, country, start_date, end_date):
    """Get related keywords with impression data."""
    data = api_get("GetRelatedKeywords", {
        "siteUrl": site_url,
        "query": keyword,
        "country": country,
        "stringStartDate": start_date,
        "stringEndDate": end_date,
    })
    return data


def get_link_counts(site_url, page=0):
    """Get pages with inbound link counts."""
    data = api_get("GetLinkCounts", {
        "siteUrl": site_url,
        "page": page,
    })
    return data


def get_crawl_stats(site_url):
    """Get crawl statistics."""
    data = api_get("GetCrawlStats", {"siteUrl": site_url})
    return data


def get_crawl_issues(site_url):
    """Get crawl issues."""
    data = api_get("GetCrawlIssues", {"siteUrl": site_url})
    return data


def main():
    parser = argparse.ArgumentParser(description="Bing Webmaster Tools API query tool")
    parser.add_argument("--site-url", default=os.environ.get("SITE_URL"),
                        help="Site URL (or set SITE_URL env)")
    parser.add_argument("--mode", choices=[
        "query_stats", "page_stats", "rank_traffic",
        "query_detail", "page_detail", "query_page_detail",
        "keyword", "related_keywords",
        "links", "crawl_stats", "crawl_issues",
    ], default="query_stats")
    parser.add_argument("--query", help="Query string (for query_detail, keyword, related_keywords modes)")
    parser.add_argument("--page", help="Page URL (for page_detail, query_page_detail modes)")
    parser.add_argument("--country", default="us", help="Country code (for keyword modes, default: us)")
    parser.add_argument("--start-date",
                        default=(datetime.now() - timedelta(days=28)).strftime("%Y-%m-%d"))
    parser.add_argument("--end-date",
                        default=(datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d"))
    parser.add_argument("--output", "-o", help="Output file path (default: stdout)")

    args = parser.parse_args()

    if not args.site_url:
        print("Error: --site-url required or set SITE_URL", file=sys.stderr)
        sys.exit(1)

    result = None

    if args.mode == "query_stats":
        data = get_query_stats(args.site_url)
        result = {
            "mode": "query_stats",
            "site_url": args.site_url,
            "data": data,
        }

    elif args.mode == "page_stats":
        data = get_page_stats(args.site_url)
        result = {
            "mode": "page_stats",
            "site_url": args.site_url,
            "data": data,
        }

    elif args.mode == "rank_traffic":
        data = get_rank_and_traffic_stats(args.site_url)
        result = {
            "mode": "rank_traffic",
            "site_url": args.site_url,
            "data": data,
        }

    elif args.mode == "query_detail":
        if not args.query:
            print("Error: --query required for query_detail mode", file=sys.stderr)
            sys.exit(1)
        data = get_query_traffic_stats(args.site_url, args.query)
        result = {
            "mode": "query_detail",
            "site_url": args.site_url,
            "query": args.query,
            "data": data,
        }

    elif args.mode == "page_detail":
        if not args.page:
            print("Error: --page required for page_detail mode", file=sys.stderr)
            sys.exit(1)
        data = get_page_query_stats(args.site_url, args.page)
        result = {
            "mode": "page_detail",
            "site_url": args.site_url,
            "page": args.page,
            "data": data,
        }

    elif args.mode == "query_page_detail":
        if not args.query or not args.page:
            print("Error: --query and --page required for query_page_detail mode", file=sys.stderr)
            sys.exit(1)
        data = get_query_page_detail_stats(args.site_url, args.query, args.page)
        result = {
            "mode": "query_page_detail",
            "site_url": args.site_url,
            "query": args.query,
            "page": args.page,
            "data": data,
        }

    elif args.mode == "keyword":
        if not args.query:
            print("Error: --query required for keyword mode", file=sys.stderr)
            sys.exit(1)
        data = get_keyword(args.site_url, args.query, args.country,
                           args.start_date, args.end_date)
        result = {
            "mode": "keyword",
            "site_url": args.site_url,
            "query": args.query,
            "country": args.country,
            "date_range": {"start": args.start_date, "end": args.end_date},
            "data": data,
        }

    elif args.mode == "related_keywords":
        if not args.query:
            print("Error: --query required for related_keywords mode", file=sys.stderr)
            sys.exit(1)
        data = get_related_keywords(args.site_url, args.query, args.country,
                                    args.start_date, args.end_date)
        result = {
            "mode": "related_keywords",
            "site_url": args.site_url,
            "query": args.query,
            "country": args.country,
            "date_range": {"start": args.start_date, "end": args.end_date},
            "data": data,
        }

    elif args.mode == "links":
        data = get_link_counts(args.site_url)
        result = {
            "mode": "links",
            "site_url": args.site_url,
            "data": data,
        }

    elif args.mode == "crawl_stats":
        data = get_crawl_stats(args.site_url)
        result = {
            "mode": "crawl_stats",
            "site_url": args.site_url,
            "data": data,
        }

    elif args.mode == "crawl_issues":
        data = get_crawl_issues(args.site_url)
        result = {
            "mode": "crawl_issues",
            "site_url": args.site_url,
            "data": data,
        }

    output = json.dumps(result, indent=2, ensure_ascii=False)
    if args.output:
        os.makedirs(os.path.dirname(os.path.abspath(args.output)), exist_ok=True)
        with open(args.output, "w", encoding="utf-8") as f:
            f.write(output)
        print(f"Output written to {args.output}", file=sys.stderr)
    else:
        print(output)


if __name__ == "__main__":
    main()


path: scripts/geo_audit.py
content: #!/usr/bin/env python3
from __future__ import annotations

"""GEO (Generative Engine Optimization) audit tool.

Checks AI-readability signals: llms.txt, robots.txt AI crawler rules,
content depth, question-style headings, HowTo schema, and more.

Usage:
    # Full GEO audit on a single page
    python geo_audit.py --url "https://example.com/"

    # Audit multiple pages
    python geo_audit.py --url "https://example.com/" --pages "/about,/pricing,/docs"

    # Audit all pages from sitemap
    python geo_audit.py --url "https://example.com/" --sitemap

    # Output to file
    python geo_audit.py --url "https://example.com/" -o geo_report.json

Reads .env from: .skills-data/google-analytics-and-search-improve/.env
Env var: SITE_URL (fallback if --url not provided)
"""

import argparse
import json
import os
import re
import sys
import time
from pathlib import Path
from urllib.parse import urljoin

import utils  # noqa: F401  — triggers .env loading & warning suppression

try:
    import requests
except ImportError:
    print("Error: 'requests' package required. Install with: pip install requests", file=sys.stderr)
    sys.exit(1)


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------

USER_AGENT = (
    "Mozilla/5.0 (compatible; GEOAuditBot/1.0; +https://github.com/skills/geo-audit)"
)
REQUEST_TIMEOUT = 15

# Known AI crawlers and their User-Agent strings
AI_CRAWLERS = [
    "GPTBot",
    "ClaudeBot",
    "PerplexityBot",
    "Google-Extended",
    "Amazonbot",
    "anthropic-ai",
    "Bytespider",
    "CCBot",
    "cohere-ai",
]


def fetch(url: str) -> requests.Response:
    return requests.get(
        url,
        headers={"User-Agent": USER_AGENT, "Accept-Encoding": "gzip, br"},
        timeout=REQUEST_TIMEOUT,
        allow_redirects=True,
    )


def safe_fetch(url: str) -> requests.Response | None:
    try:
        return fetch(url)
    except Exception as exc:
        print(f"  ⚠️  Failed to fetch {url}: {exc}", file=sys.stderr)
        return None


def extract_sitemap_urls(sitemap_url: str) -> list[str]:
    resp = safe_fetch(sitemap_url)
    if not resp or resp.status_code != 200:
        return []
    content = resp.text
    urls = []
    if "<sitemapindex" in content.lower():
        sub_sitemaps = re.findall(r'<loc>\s*(.*?)\s*</loc>', content)
        for sub_url in sub_sitemaps:
            sub_resp = safe_fetch(sub_url)
            if sub_resp and sub_resp.status_code == 200:
                urls.extend(re.findall(r'<loc>\s*(.*?)\s*</loc>', sub_resp.text))
    else:
        urls = re.findall(r'<loc>\s*(.*?)\s*</loc>', content)
    return urls


# ---------------------------------------------------------------------------
# llms.txt audit
# ---------------------------------------------------------------------------

def audit_llms_txt(base_url: str) -> dict:
    """Check presence and quality of llms.txt and llms-full.txt."""
    result = {
        "llms_txt": {"exists": False, "url": f"{base_url}/llms.txt", "lines": 0, "preview": ""},
        "llms_full_txt": {"exists": False, "url": f"{base_url}/llms-full.txt", "lines": 0, "preview": ""},
        "robots_txt_reference": False,
        "issues": [],
        "checks": {},
    }

    # llms.txt
    resp = safe_fetch(f"{base_url}/llms.txt")
    if resp and resp.status_code == 200 and len(resp.text.strip()) > 10:
        result["llms_txt"]["exists"] = True
        result["llms_txt"]["lines"] = len(resp.text.strip().splitlines())
        result["llms_txt"]["preview"] = "\n".join(resp.text.strip().splitlines()[:10])
    else:
        result["issues"].append("Missing /llms.txt — critical for AI engine discovery (P0)")

    # llms-full.txt
    resp = safe_fetch(f"{base_url}/llms-full.txt")
    if resp and resp.status_code == 200 and len(resp.text.strip()) > 10:
        result["llms_full_txt"]["exists"] = True
        result["llms_full_txt"]["lines"] = len(resp.text.strip().splitlines())
        result["llms_full_txt"]["preview"] = "\n".join(resp.text.strip().splitlines()[:10])
    else:
        result["issues"].append("Missing /llms-full.txt — recommended for comprehensive AI indexing")

    # Check robots.txt for llms.txt reference
    robots_resp = safe_fetch(f"{base_url}/robots.txt")
    if robots_resp and robots_resp.status_code == 200:
        if "llms" in robots_resp.text.lower():
            result["robots_txt_reference"] = True
        else:
            result["issues"].append("robots.txt does not reference llms.txt (add: Llms-txt: URL)")

    result["checks"]["has_llms_txt"] = result["llms_txt"]["exists"]
    result["checks"]["has_llms_full_txt"] = result["llms_full_txt"]["exists"]
    result["checks"]["robots_references_llms"] = result["robots_txt_reference"]

    return result


# ---------------------------------------------------------------------------
# robots.txt AI crawler audit
# ---------------------------------------------------------------------------

def audit_robots_ai_crawlers(base_url: str) -> dict:
    """Analyze robots.txt for AI crawler directives."""
    result = {
        "url": f"{base_url}/robots.txt",
        "exists": False,
        "content": "",
        "crawlers": {},
        "issues": [],
        "checks": {},
    }

    resp = safe_fetch(f"{base_url}/robots.txt")
    if not resp or resp.status_code != 200:
        result["issues"].append("robots.txt not found")
        return result

    result["exists"] = True
    content = resp.text
    result["content"] = content

    # Parse robots.txt for AI crawler rules
    lines = content.splitlines()
    current_agent = None
    agent_rules = {}

    for line in lines:
        line = line.strip()
        if line.startswith("#") or not line:
            continue
        if line.lower().startswith("user-agent:"):
            current_agent = line.split(":", 1)[1].strip()
            if current_agent not in agent_rules:
                agent_rules[current_agent] = []
        elif current_agent and (line.lower().startswith("allow:") or line.lower().startswith("disallow:")):
            agent_rules[current_agent] = agent_rules.get(current_agent, [])
            agent_rules[current_agent].append(line)

    # Check each known AI crawler
    for crawler in AI_CRAWLERS:
        status = "not_mentioned"
        if crawler in agent_rules:
            rules = agent_rules[crawler]
            has_allow = any("allow:" in r.lower() and "disallow:" not in r.lower() for r in rules)
            has_disallow = any("disallow:" in r.lower() for r in rules)
            if has_disallow and not has_allow:
                status = "blocked"
            elif has_allow:
                status = "allowed"
            else:
                status = "configured"
        elif "*" in agent_rules:
            # Falls under wildcard
            rules = agent_rules["*"]
            has_disallow_all = any("disallow: /" == r.lower().strip() for r in rules)
            if has_disallow_all:
                status = "blocked_by_wildcard"
            else:
                status = "allowed_by_wildcard"
        else:
            status = "allowed_by_default"

        result["crawlers"][crawler] = status

    # Generate issues
    blocked = [c for c, s in result["crawlers"].items() if "blocked" in s]
    not_mentioned = [c for c, s in result["crawlers"].items() if s in ("not_mentioned", "allowed_by_default", "allowed_by_wildcard")]

    if blocked:
        result["issues"].append(f"AI crawlers BLOCKED: {', '.join(blocked)}")
    if not_mentioned:
        result["issues"].append(
            f"AI crawlers not explicitly configured (recommended to add Allow rules): {', '.join(not_mentioned)}"
        )

    result["checks"]["no_ai_crawlers_blocked"] = len(blocked) == 0
    result["checks"]["ai_crawlers_explicitly_allowed"] = len(not_mentioned) == 0

    return result


# ---------------------------------------------------------------------------
# Content depth audit
# ---------------------------------------------------------------------------

def audit_content_depth(url: str, html: str) -> dict:
    """Analyze page content depth: word count, structure, intro summary, FAQ."""
    result = {
        "url": url,
        "word_count": 0,
        "has_intro_summary": False,
        "has_faq_section": False,
        "has_howto_content": False,
        "heading_count": 0,
        "question_headings": 0,
        "question_heading_ratio": 0,
        "content_sections": [],
        "issues": [],
        "checks": {},
    }

    # Strip scripts and styles
    clean = re.sub(r'<script[^>]*>.*?</script>', '', html, flags=re.DOTALL | re.IGNORECASE)
    clean = re.sub(r'<style[^>]*>.*?</style>', '', clean, flags=re.DOTALL | re.IGNORECASE)
    # Remove nav, header, footer for main content estimation
    clean_main = re.sub(r'<(nav|header|footer)[^>]*>.*?</\1>', '', clean, flags=re.DOTALL | re.IGNORECASE)
    text = re.sub(r'<[^>]+>', ' ', clean_main)
    text = re.sub(r'\s+', ' ', text).strip()

    cjk_chars = sum(1 for ch in text if '\u4e00' <= ch <= '\u9fff'
                    or '\u3400' <= ch <= '\u4dbf'
                    or '\U00020000' <= ch <= '\U0002a6df')
    non_cjk_text = re.sub(r'[\u4e00-\u9fff\u3400-\u4dbf]', ' ', text)
    en_words = len(non_cjk_text.split())

    is_cjk_dominant = cjk_chars > en_words
    result["cjk_chars"] = cjk_chars
    result["en_words"] = en_words
    result["is_cjk_dominant"] = is_cjk_dominant
    result["word_count"] = cjk_chars if is_cjk_dominant else en_words

    # Check for intro summary: a meaningful text block before the first <h2>.
    # Language-agnostic — checks length, not specific keywords.
    intro_text = ""
    h1_match = re.search(r'</h1>', clean_main, re.IGNORECASE)
    h2_match = re.search(r'<h2[\s>]', clean_main, re.IGNORECASE)
    if h1_match:
        start = h1_match.end()
        end = h2_match.start() if h2_match and h2_match.start() > start else start + 3000
        intro_html = clean_main[start:end]
        intro_text = re.sub(r'<[^>]+>', ' ', intro_html)
        intro_text = re.sub(r'\s+', ' ', intro_text).strip()

    if is_cjk_dominant:
        intro_len = sum(1 for ch in intro_text if '\u4e00' <= ch <= '\u9fff'
                        or '\u3400' <= ch <= '\u4dbf')
        result["has_intro_summary"] = intro_len >= 50
    else:
        result["has_intro_summary"] = len(intro_text.split()) >= 30

    # Check for FAQ section
    faq_signals = ["faq", "frequently asked", "common questions", "q&a",
                    "常见问题", "问答", "疑问解答"]
    html_lower = html.lower()
    result["has_faq_section"] = any(s in html_lower for s in faq_signals)

    # Check for HowTo content
    howto_signals = ["step 1", "step 2", "how to", "getting started", "tutorial",
                     "第一步", "第二步", "步骤", "教程", "使用方法", "操作指南"]
    result["has_howto_content"] = any(s in html_lower for s in howto_signals)

    # Heading analysis
    headings = re.findall(r'<(h[1-6])[^>]*>(.*?)</\1>', html, re.IGNORECASE | re.DOTALL)
    result["heading_count"] = len(headings)

    question_count = 0
    for _, heading_text in headings:
        cleaned = re.sub(r'<[^>]+>', '', heading_text).strip()
        if cleaned.rstrip().endswith("?") or cleaned.rstrip().endswith("？"):
            question_count += 1

    result["question_headings"] = question_count
    result["question_heading_ratio"] = (
        round(question_count / len(headings), 2) if headings else 0
    )

    # Check JSON-LD for FAQPage and HowTo
    faq_schema = '"FAQPage"' in html or "'FAQPage'" in html
    howto_schema = '"HowTo"' in html or "'HowTo'" in html
    result["checks"]["has_faq_schema"] = faq_schema
    result["checks"]["has_howto_schema"] = howto_schema

    # Issues — thresholds differ by dominant language
    if is_cjk_dominant:
        min_thin, min_target = 600, 1600
        unit = "CJK chars"
        target_label = "1600-2400 CJK chars"
    else:
        min_thin, min_target = 300, 800
        unit = "words"
        target_label = "800-1200 words"

    result["checks"]["word_count_sufficient"] = result["word_count"] >= min_target
    result["checks"]["has_intro_summary"] = result["has_intro_summary"]
    result["checks"]["has_faq_section"] = result["has_faq_section"]
    result["checks"]["has_question_headings"] = question_count > 0

    if result["word_count"] < min_thin:
        result["issues"].append(f"Very thin content ({result['word_count']} {unit}) — target {target_label}")
    elif result["word_count"] < min_target:
        result["issues"].append(f"Below content depth target ({result['word_count']} {unit}) — target {target_label}")

    if not result["has_intro_summary"]:
        result["issues"].append("No intro summary paragraph detected between H1 and first H2")

    if not result["has_faq_section"]:
        result["issues"].append("No FAQ section detected — adding FAQ improves GEO citations")

    if len(headings) > 3 and question_count == 0:
        result["issues"].append("No question-style headings found — AI engines prefer question format H2/H3")

    if result["has_faq_section"] and not faq_schema:
        result["issues"].append("FAQ content found but no FAQPage schema — add JSON-LD")

    if result["has_howto_content"] and not howto_schema:
        result["issues"].append("HowTo content found but no HowTo schema — add JSON-LD")

    return result


# ---------------------------------------------------------------------------
# Full page GEO audit
# ---------------------------------------------------------------------------

def audit_page_geo(url: str) -> dict:
    """Run GEO-focused audit on a single page."""
    resp = safe_fetch(url)
    if not resp or resp.status_code != 200:
        return {
            "url": url,
            "status": resp.status_code if resp else "error",
            "issues": [f"Failed to fetch page (status: {resp.status_code if resp else 'network error'})"],
            "checks": {},
        }

    html = resp.text
    content_result = audit_content_depth(url, html)
    return content_result


# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(description="GEO (Generative Engine Optimization) audit tool")
    parser.add_argument("--url", default=os.environ.get("SITE_URL"),
                        help="Base URL of the site (or set SITE_URL env)")
    parser.add_argument("--sitemap", action="store_true",
                        help="Audit all pages from sitemap.xml")
    parser.add_argument("--pages", help="Comma-separated page paths to audit (e.g. /about,/pricing)")
    parser.add_argument("--max-pages", type=int, default=50,
                        help="Max pages to audit from sitemap (default: 50)")
    parser.add_argument("--output", "-o", help="Output file path (default: stdout)")

    args = parser.parse_args()

    if not args.url:
        print("Error: --url required or set SITE_URL", file=sys.stderr)
        sys.exit(1)

    base_url = args.url.rstrip("/")
    if not base_url.startswith("http"):
        base_url = f"https://{base_url}"

    # --- Site-level GEO checks ---
    print("Checking llms.txt...", file=sys.stderr)
    llms_result = audit_llms_txt(base_url)

    print("Checking robots.txt AI crawlers...", file=sys.stderr)
    robots_result = audit_robots_ai_crawlers(base_url)

    # --- Page-level content depth checks ---
    page_urls = []
    if args.pages:
        for path in args.pages.split(","):
            path = path.strip()
            if not path.startswith("/"):
                path = f"/{path}"
            page_urls.append(f"{base_url}{path}")
    elif args.sitemap:
        sitemap_url = f"{base_url}/sitemap.xml"
        print(f"Fetching sitemap: {sitemap_url}", file=sys.stderr)
        page_urls = extract_sitemap_urls(sitemap_url)
        if not page_urls:
            print("Warning: No URLs in sitemap, falling back to homepage", file=sys.stderr)
            page_urls = [f"{base_url}/"]
        else:
            print(f"Found {len(page_urls)} pages in sitemap", file=sys.stderr)
            if len(page_urls) > args.max_pages:
                print(f"Limiting to first {args.max_pages} pages", file=sys.stderr)
                page_urls = page_urls[:args.max_pages]
    else:
        page_urls = [f"{base_url}/"]

    print(f"Auditing content depth on {len(page_urls)} page(s)...", file=sys.stderr)
    page_results = []
    for i, url in enumerate(page_urls, 1):
        print(f"  [{i}/{len(page_urls)}] {url}", file=sys.stderr)
        page_results.append(audit_page_geo(url))
        if i < len(page_urls):
            time.sleep(0.5)

    # --- Aggregate ---
    all_issues = []
    for issue in llms_result["issues"]:
        all_issues.append({"source": "llms.txt", "issue": issue})
    for issue in robots_result["issues"]:
        all_issues.append({"source": "robots.txt", "issue": issue})
    for pr in page_results:
        for issue in pr.get("issues", []):
            all_issues.append({"source": pr["url"], "issue": issue})

    # Aggregate checks
    site_checks = {}
    site_checks.update(llms_result["checks"])
    site_checks.update(robots_result["checks"])

    page_check_keys = set()
    for pr in page_results:
        page_check_keys.update(pr.get("checks", {}).keys())
    page_checks = {}
    for key in sorted(page_check_keys):
        values = [pr["checks"].get(key) for pr in page_results if key in pr.get("checks", {})]
        passed = sum(1 for v in values if v)
        page_checks[key] = {"passed": passed, "total": len(values)}

    result = {
        "tool": "geo_audit",
        "base_url": base_url,
        "pages_audited": len(page_results),
        "summary": {
            "total_issues": len(all_issues),
            "site_level_checks": site_checks,
            "page_level_checks": page_checks,
        },
        "llms_txt": llms_result,
        "robots_ai_crawlers": robots_result,
        "issues": all_issues,
        "pages": page_results,
    }

    output = json.dumps(result, indent=2, ensure_ascii=False, default=str)
    if args.output:
        with open(args.output, "w", encoding="utf-8") as f:
            f.write(output)
        print(f"\nOutput written to {args.output}", file=sys.stderr)
    else:
        print(output)

    # Quick summary
    print(f"\n{'='*60}", file=sys.stderr)
    print(f"GEO Audit Summary: {base_url}", file=sys.stderr)
    print(f"Pages audited: {len(page_results)}", file=sys.stderr)
    print(f"Total issues: {len(all_issues)}", file=sys.stderr)
    print(f"llms.txt: {'✅' if llms_result['llms_txt']['exists'] else '❌'}", file=sys.stderr)
    print(f"llms-full.txt: {'✅' if llms_result['llms_full_txt']['exists'] else '❌'}", file=sys.stderr)

    blocked = [c for c, s in robots_result["crawlers"].items() if "blocked" in s]
    if blocked:
        print(f"Blocked AI crawlers: ❌ {', '.join(blocked)}", file=sys.stderr)
    else:
        print(f"AI crawlers: ✅ None blocked", file=sys.stderr)

    if page_results:
        any_cjk = any(pr.get("is_cjk_dominant", False) for pr in page_results)
        avg_score = sum(pr.get("word_count", 0) for pr in page_results) / len(page_results)
        if any_cjk:
            print(f"Avg content depth: {avg_score:.0f} CJK chars/page (target: 1600-2400)", file=sys.stderr)
        else:
            print(f"Avg content depth: {avg_score:.0f} words/page (target: 800-1200)", file=sys.stderr)
    print(f"{'='*60}", file=sys.stderr)


if __name__ == "__main__":
    main()


path: references/data-visualization-guide.md
content: # Data Visualization Guide

## Visualization Principles

1. **Chart-first reporting**: When presenting data findings, always generate a chart BEFORE writing the textual interpretation. The chart is the primary evidence; the text explains the insight.
2. **One chart, one insight**: Each chart should communicate a single clear message. Use the chart title to state the insight (e.g., "Mobile bounce rate is 2x desktop" instead of "Bounce rate by device").
3. **Goal-aligned visualization**: Highlight goal-relevant thresholds, targets, or benchmarks in charts using reference lines or color coding.
4. **Consistent style**: Use a consistent color palette and style across all charts in a single report.

## Chart Generation Method

Use Python with `matplotlib` to generate charts. Each analysis script should combine **data processing and chart generation in one script** — read raw JSON from `$DATA_DIR/data/`, process/aggregate the data, output structured results, and generate charts in the same execution. This ensures charts are always consistent with the analysis data.

**Standard pattern** (data analysis + chart generation in one script):

```python
import matplotlib
matplotlib.use('Agg')  # Non-interactive backend, MUST be set before importing pyplot
import matplotlib.pyplot as plt
import json
import os
import platform

# ── Configuration ──────────────────────────────────────────
DATA_DIR = os.environ.get("DATA_DIR", ".skills-data/google-analytics-and-search-improve")
CHARTS_DIR = os.path.join(DATA_DIR, "charts")
os.makedirs(CHARTS_DIR, exist_ok=True)

# ── CJK font support (Chinese / Japanese / Korean) ────────
_system = platform.system()
if _system == "Darwin":
    _cjk_fonts = ["Arial Unicode MS", "PingFang SC", "Heiti SC", "STHeiti"]
elif _system == "Linux":
    _cjk_fonts = ["WenQuanYi Micro Hei", "Noto Sans CJK SC", "DejaVu Sans"]
else:  # Windows
    _cjk_fonts = ["Microsoft YaHei", "SimHei", "SimSun"]

# ── Style defaults ─────────────────────────────────────────
plt.rcParams.update({
    "font.sans-serif": _cjk_fonts,
    "axes.unicode_minus": False,     # Fix minus sign rendering with CJK fonts
    "figure.figsize": (10, 6),
    "figure.dpi": 150,
    "axes.titlesize": 14,
    "axes.labelsize": 12,
    "xtick.labelsize": 10,
    "ytick.labelsize": 10,
    "axes.spines.top": False,
    "axes.spines.right": False,
    "figure.facecolor": "white",
    "axes.facecolor": "white",
    "savefig.bbox": "tight",
    "savefig.pad_inches": 0.3,
})

# Color palette (consistent across all charts)
COLORS = ["#4285F4", "#EA4335", "#FBBC04", "#34A853", "#FF6D01", "#46BDC6", "#7B61FF", "#F538A0"]
COLOR_GOOD = "#34A853"
COLOR_WARN = "#FBBC04"
COLOR_BAD = "#EA4335"
COLOR_NEUTRAL = "#4285F4"

# ── Load data ──────────────────────────────────────────────
with open(os.path.join(DATA_DIR, "data", "DATAFILE.json")) as f:
    data = json.load(f)

# ── Plot ───────────────────────────────────────────────────
fig, ax = plt.subplots()
# ... plotting code ...
ax.set_title("Insight-Driven Chart Title")
ax.set_xlabel("X Label")
ax.set_ylabel("Y Label")

output_path = os.path.join(CHARTS_DIR, "chart_name.png")
fig.savefig(output_path)
plt.close(fig)
print(f"Chart saved: {output_path}")
```

**Embed in Markdown report**:
```markdown
### Finding: [Insight Title]

![Chart description](../charts/chart_name.png)

[Textual interpretation of what the chart reveals and recommended action]
```

## Chart Type Selection Guide

| Data Pattern | Recommended Chart | When to Use |
|---|---|---|
| **Trend over time** | Line chart | GSC daily clicks/impressions, GA4 traffic trends, funnel trends |
| **Category comparison** | Horizontal bar chart | Top queries by clicks, pages by traffic, channels comparison |
| **Part of whole** | Stacked bar or pie chart (≤6 slices) | Traffic source distribution, device breakdown |
| **Funnel / conversion flow** | Funnel chart (horizontal stacked bars) | User journey drop-off, signup/purchase funnel |
| **Two metrics correlation** | Scatter plot | CTR vs position, bounce rate vs page load time |
| **Distribution** | Histogram or box plot | Session duration distribution, page load time spread |
| **Before/After or Gap** | Grouped bar chart | Expected vs actual behavior, mobile vs desktop comparison |
| **Performance scores** | Gauge or bullet chart | PSI scores, CWV metrics against thresholds |
| **Ranking with values** | Horizontal bar + value labels | Top 10 keywords, top pages by any metric |

## Required Charts per Phase

**Phase 2 (GSC Analysis)** — generate and embed:
- **Keyword intent distribution**: Stacked bar or pie chart showing query distribution across Awareness / Consideration / Decision / Retention stages
- **Top queries performance**: Horizontal bar chart of top 15 queries by clicks, with CTR annotated
- **CTR vs Position scatter**: Scatter plot of queries showing CTR vs average position, highlighting high-impression low-CTR outliers
- **Trend chart**: Line chart of daily clicks & impressions over the analysis period
- **Device/country breakdown**: Bar chart of clicks by device type and/or top countries

**Phase 3 (GA4 Analysis)** — generate and embed:
- **Traffic channel breakdown**: Pie or bar chart of sessions by source/medium
- **Top landing pages**: Horizontal bar chart with engagement rate or bounce rate overlay
- **Journey gap visualization**: Grouped bar chart comparing expected vs actual conversion rates at each journey stage
- **Device comparison**: Side-by-side bars comparing key metrics (bounce rate, engagement rate, conversion) across desktop/mobile/tablet
- **Trend chart**: Line chart of sessions/users over the analysis period

**Phase 3b (Funnel Analysis)** — generate and embed:
- **Funnel chart**: Horizontal funnel visualization showing step-by-step user counts and drop-off percentages
- **Funnel by device/channel**: Grouped funnel comparison across segments
- **Funnel trend**: Line chart of daily conversion rates for key funnel steps

**Phase 4 (Site Audit)** — generate and embed:
- **PSI scores radar/bar**: Bar chart of Performance, SEO, Accessibility, Best Practices scores (mobile & desktop)
- **Core Web Vitals**: Bar or gauge chart of LCP, FID/INP, CLS against Good/Needs Improvement/Poor thresholds

**Phase 6 (Final Report)** — generate and embed:
- **Executive summary dashboard**: A multi-panel figure (2×2 or 2×3 subplots) summarizing the most critical metrics at a glance
- **Goal achievement scorecard**: Bar chart showing goal achievement percentage for each journey stage
- **Priority distribution**: Horizontal bar chart showing count of issues by P0/P1/P2/P3 priority
- Reuse the most impactful charts from Phase 2-4 in the final report (reference the same image files)

## CJK Font Support

> **Note**: The standard pattern template above already includes CJK font auto-detection. You do NOT need to add it separately — it is built into the `plt.rcParams` block. The logic below is kept for reference only.

When the site content or data contains Chinese/Japanese/Korean characters, matplotlib needs a CJK-compatible font. The standard template handles this automatically via `platform.system()` detection:

- **macOS (Darwin)**: Arial Unicode MS → PingFang SC → Heiti SC → STHeiti
- **Linux**: WenQuanYi Micro Hei → Noto Sans CJK SC → DejaVu Sans
- **Windows**: Microsoft YaHei → SimHei → SimSun

Key `rcParams` that must be set:
```python
plt.rcParams["font.sans-serif"] = ["PingFang SC", ...]  # CJK-capable font list
plt.rcParams["axes.unicode_minus"] = False               # Fix minus sign rendering
```


path: references/report-template.md
content: # Final Report Template

Use this template for Phase 6: Generate Goal-Aligned Improvement Report. Organize output around the **website goals defined in Phase 0**, using the "Priority Matrix" (P0-P3) in [metrics-glossary.md](metrics-glossary.md).

---

```markdown
# Website Data Analysis & Improvement Plan

## Website Profile
- **Target Website**: [URL]
- **Website Type**: [e.g., SaaS Tool / E-commerce / Content Blog]
- **Primary Goal**: [e.g., Convert visitors to paid subscribers]
- **Intended User Journey**: [Journey defined in Phase 0]
- **Data Source**: API auto-collection / Manual CSV export / Browser audit only
- **Analysis Period**: [start_date] ~ [end_date]

## Executive Summary
[2-3 sentences: What the site wants to happen vs. what's actually happening, and the single biggest opportunity]

![Executive Summary Dashboard](../charts/report_executive_dashboard.png)

## Goal Achievement Status

### Primary Goal: [goal name]

![Goal Achievement Scorecard](../charts/report_goal_scorecard.png)

| Journey Stage | Expected Behavior | Actual Data | Gap | Severity |
|---|---|---|---|---|
| [e.g., Landing → Features] | Users explore features after landing | 68% bounce rate on landing page | Users leave immediately | 🔴 Critical |
| [e.g., Features → Signup] | Users proceed to signup | Only 2% click signup CTA | Very low progression | 🔴 Critical |
| [e.g., Signup → Onboarding] | Users complete onboarding | 45% complete (if data available) | Moderate drop-off | 🟡 Moderate |

### Secondary Goals
(Same format for each secondary goal)

## Data Overview

![Traffic Trend](../charts/ga4_traffic_trend.png)

| Metric | Current Value | Trend | Interpretation |
|--------|---------------|-------|----------------|
| GSC Total Impressions / Clicks / CTR / Position | ... | ... | [How this relates to goals] |
| GA4 Sessions / Users / Bounce Rate / Engagement Rate | ... | ... | [How this relates to goals] |
| PSI Performance Score (Mobile/Desktop) | ... | ... | [Impact on user journey] |

![Device Comparison](../charts/ga4_device_comparison.png)

## Key Findings: Where Users Diverge from the Intended Journey

![Journey Gap Analysis](../charts/ga4_journey_gap.png)

1. **[Finding]** — Data evidence → Impact on goal → Recommended fix
2. ...
(Ordered by impact on primary goal, not by analysis dimension)

## Improvement Plan

![Priority Distribution](../charts/report_priority_distribution.png)

### P0 Critical (Directly blocks primary goal)
1. **[Issue]** — Data evidence / Fix / Expected impact on goal

### P1 High (Significantly impacts goal achievement)
### P2 Medium (Supports goal but not on the critical path)
### P3 Low (Nice to have)

## What Data Is Missing
| Question I Can't Answer | Why It Matters | How to Get This Data |
|---|---|---|
| [e.g., Why do users leave the pricing page?] | [It's the biggest funnel drop-off] | [Add heatmap tracking / Set up exit-intent survey] |

## Next Steps & Strategy
### Immediate Actions (This Week)
1. [Specific, actionable item tied to P0 finding]

### Short-term Strategy (This Month)
1. [Specific item tied to P1 findings]

### Medium-term Strategy (Next Quarter)
1. [Items requiring more effort, tied to P2-P3]

### Data Collection Improvements
1. [Specific events/tracking to add for better analysis next time]

## Detailed Analysis
(Organized by the priority dimensions from Phase 0:
e.g., Conversion Funnel → Acquisition/SEO → Content Engagement → UX → Performance → Technical → SEO/GEO Checklist)

[Embed relevant charts from Phase 2-4 in each subsection. Reuse chart files — do not regenerate.]

## Execution Roadmap
| Phase | Timeline | Tasks | Expected Outcome | How to Verify |
|-------|----------|-------|------------------|---------------|
| Week 1-2 | P0 | ... | ... | [Specific metric to check] |
| Week 3-4 | P1 | ... | ... | [Specific metric to check] |
| Month 2+ | P2-P3 | ... | ... | [Specific metric to check] |
```

---

## Chart Requirements for Final Report

Generate the following Phase 6 charts (see [data-visualization-guide.md](data-visualization-guide.md) for chart generation patterns):

- **Executive summary dashboard** (`report_executive_dashboard.png`): A multi-panel figure (2×2 or 2×3 subplots) summarizing the most critical metrics at a glance
- **Goal achievement scorecard** (`report_goal_scorecard.png`): Bar chart showing goal achievement percentage for each journey stage
- **Priority distribution** (`report_priority_distribution.png`): Horizontal bar chart showing count of issues by P0/P1/P2/P3 priority

Reuse Phase 2-4 chart images in the Detailed Analysis section by referencing the same file paths.

Save the report to `$DATA_DIR/analysis/improvement-report.md`.


path: references/metrics-glossary.md
content: # Key Metrics & Analysis Dimensions

## Six Analysis Dimensions

### 1. SEO

**Primary data source**: GSC Search Analytics

| Metric | Healthy Range | Warning Signal |
|--------|---------------|----------------|
| Average CTR | > 3% (overall) | < 1% requires immediate optimization |
| Average Position | < 20 (first two pages) | > 30 low exposure value |
| Impressions Trend | Steady growth | Sustained decline |
| Index Coverage | > 90% | < 70% structural issues |

**Diagnostic points**:
- High impressions, low CTR → Title/description needs optimization
- Keywords ranked 4-10 → Best optimization targets (push into top 3)
- Pages with declining rankings → Content needs updating or competitors overtaking

### 2. Performance

**Primary data source**: PageSpeed Insights API + agent-browser profiling

| Metric | Good | Needs Improvement | Poor |
|--------|------|-------------------|------|
| LCP (Largest Contentful Paint) | < 2.5s | 2.5-4s | > 4s |
| INP (Interaction to Next Paint) | < 200ms | 200-500ms | > 500ms |
| CLS (Cumulative Layout Shift) | < 0.1 | 0.1-0.25 | > 0.25 |
| TTFB (Time to First Byte) | < 800ms | 800-1800ms | > 1800ms |

### 3. Content Strategy

**Primary data source**: GA4 top_pages + GSC search_analytics

**Diagnostic points**:
- **High-traffic low-engagement pages**: High pageViews but low engagementRate → Content quality doesn't match user expectations
- **High-ranking low-traffic pages**: Good rankings but low clicks → Title/description lacks appeal
- **Zero-traffic content**: Indexed but no impressions → Keyword strategy failure or poor content quality
- **Content decay**: Pages with sustained traffic decline → Needs updating or merging

### 4. User Experience (UX)

**Primary data source**: GA4 user_behavior + landing_pages

| Metric | Healthy Range | Warning Signal |
|--------|---------------|----------------|
| Bounce Rate | < 50% | > 70% (non-blog sites) |
| Engagement Rate | > 60% | < 40% |
| Average Session Duration | > 2 min | < 30s |
| Page Depth | > 2 | = 1 (single-page exit) |

**Diagnostic points**:
- Mobile bounce rate much higher than desktop → Poor mobile experience
- Specific pages with abnormally high bounce rate → Page content or design issues
- Very short session duration → Slow page load or irrelevant content

### 5. Conversion Rate Optimization

**Primary data source**: GA4 conversion_events + landing_pages

**Diagnostic points**:
- Large conversion rate variance across landing pages → A/B testing opportunity
- High-traffic low-conversion pages → CTA or user path issues
- Conversion funnel drop-off points → Identify where users are lost
- Cross-device conversion rate differences → Device-specific experience issues

### 6. Technical Issues

**Primary data source**: GSC URL Inspect + source code analysis + agent-browser

**Checklist**:
- [ ] robots.txt configured correctly
- [ ] sitemap.xml complete and accessible
- [ ] No 4xx/5xx error pages
- [ ] Pages have correct meta tags
- [ ] Structured data (JSON-LD) error-free
- [ ] Mobile-friendly (viewport meta)
- [ ] HTTPS configured correctly
- [ ] No mixed content warnings
- [ ] Images have alt attributes
- [ ] No broken internal links

## Priority Matrix

Classified by **Impact** x **Implementation Effort**:

| Priority | Impact | Effort | Typical Items |
|----------|--------|--------|---------------|
| P0 Critical | High | Low | Fix 4xx errors, add meta descriptions, fix indexing issues |
| P1 High | High | Medium | Optimize high-impression low-CTR pages, improve Core Web Vitals |
| P2 Medium | Medium | Medium | Content updates, landing page optimization, add structured data |
| P3 Low | Low/Medium | High | Large-scale refactoring, internationalization, new feature development |


path: references/data-collection-reference.md
content: # Data Collection Reference

Complete reference for data collection: authentication setup, script usage, API details, CSV export instructions, and custom query guidance.

---

## 1. Authentication Setup (Shared by GSC & GA4)

GSC and GA4 share the same Google Service Account. Create it once to use for both.

### Step 1: Create a Google Cloud Project and Enable APIs

1. Open [Google Cloud Console](https://console.cloud.google.com/)
2. Click the project selector (top-left) → "New Project" → name it (e.g., "my-site-analytics") → "Create"
3. Confirm you've switched to the new project, then open the [API Library](https://console.cloud.google.com/apis/library)
4. Search for **"Google Search Console API"** → click into it → click "Enable"
5. Search for **"Google Analytics Data API"** → click into it → click "Enable"
6. Search for **"PageSpeed Insights API"** → click into it → click "Enable" (used for site performance auditing; without it you'll get 429 quota errors)

### Step 2: Create a Service Account and Download JSON Key

1. Open the [Service Accounts page](https://console.cloud.google.com/iam-admin/serviceaccounts)
2. Click "+ Create Service Account" at the top
3. Enter a name (e.g., "analytics-reader") → click "Create and Continue"
4. Skip role selection (no project-level role needed) → "Continue" → "Done"
5. Find the newly created Service Account in the list, click its email address to enter the details page
6. Click the "Keys" tab → "Add Key" → "Create new key" → select **JSON** → "Create"
7. Your browser will automatically download a `.json` file — this is the key file
8. Note the Service Account email address (format: `[REDACTED_SECRET]`)

### Step 3: Place JSON Key in configs/

```bash
cp /path/to/downloaded/my-site-analytics-xxxx.json "$DATA_DIR/configs/"
```

All scripts (`gsc_query.py`, `ga4_query.py`, `ga4_funnel.py`) auto-discover the `*.json` key file from `$DATA_DIR/configs/` via the shared `utils.py` module — no need to configure the path in `.env` or set `GOOGLE_APPLICATION_CREDENTIALS`. If multiple JSON files exist, the first one (alphabetically) is used.

### Step 4: Authorize in Search Console (for GSC)

1. Open [Google Search Console](https://search.google.com/search-console/)
2. Select your site property
3. Click "Settings" at the bottom of the left menu → "Users and permissions"
4. Click "Add user" → paste the Service Account email → set permission to "Restricted" (read-only) → "Add"

### Step 5: Confirm GSC Property Type

GSC has two property types, and the `GSC_SITE_URL` value must match the actual type:

| GSC Property Type | GSC_SITE_URL Format | Example |
|-------------------|---------------------|---------|
| **Domain property** | `sc-domain:domain` | `sc-domain:example.com` |
| **URL-prefix property** | Full URL | `https://example.com` |

> How to check: In the [Search Console](https://search.google.com/search-console/) property selector (top-left), if it shows a bare domain name it's a Domain property (use `sc-domain:` prefix); if it shows a full URL it's a URL-prefix property. Using the wrong format will cause the API to return a 403 permission error.

### Step 6: Authorize Service Account in GA4

1. Open [Google Analytics](https://analytics.google.com/)
2. Click the gear icon (Admin) in the bottom-left
3. Under "Property", click "Property Access Management"
4. Click "+" (top-right) → "Add users"
5. Paste the Service Account email → set role to "Viewer" → "Add"

### Step 7: Get GA4 Property ID

1. In the Google Analytics Admin page, click "Property Settings" (or "Property Details") under "Property"
2. The "Property ID" is displayed in the top-right — a numeric string (e.g., `123456789`, no "UA-" prefix)

### Step 8: Write to .env

```bash
cat > "$DATA_DIR/.env" <<EOF
SITE_URL=provided by user
GSC_SITE_URL=provided by user (note sc-domain: or https:// format)
GA4_PROPERTY_ID=provided by user
BING_WEBMASTER_API_KEY=provided by user (optional, for Bing search data)
SOURCE_CODE_PATH=provided by user
PSI_API_KEY=
EOF
```

---

## 2. Bing Webmaster Authentication (Separate from Google)

### Setup

1. Open [Bing Webmaster Tools](https://www.bing.com/webmasters/) and sign in
2. Add your site if not already added, and complete site verification
3. Click the **gear icon** (Settings) in the top-right → **"API Access"**
4. Accept the Terms & Conditions → click **"Generate API Key"** → copy the key

> Only one API key per user account. If compromised, delete and regenerate.

Write to `.env`:
```
BING_WEBMASTER_API_KEY=your_api_key_here
```

The site URL is read from `SITE_URL` (shared with GSC). No separate Bing URL variable needed.

---

## 3. Install Dependencies

Create a Python 3.12 virtual environment via `uv` and install dependencies:

```bash
uv venv "$DATA_DIR/venv" --python 3.12
uv pip install -p "$DATA_DIR/venv" -r scripts/pyproject.toml
```

> **Note**: All Python script executions MUST activate the venv first. After activation, use `python` directly:
> ```bash
> source "$DATA_DIR/venv/bin/activate"
> python scripts/xxx.py ...
> ```

---

## 4. Data Collection Modes

### Mode A: API Auto-Collection

```bash
source "$DATA_DIR/venv/bin/activate"
set -a; source "$DATA_DIR/.env"; set +a

# GSC data
python scripts/gsc_query.py --dimensions query --limit 500 -o "$DATA_DIR/data/gsc_queries.json"
python scripts/gsc_query.py --dimensions page --limit 500 -o "$DATA_DIR/data/gsc_pages.json"
python scripts/gsc_query.py --dimensions device,country -o "$DATA_DIR/data/gsc_devices.json"
python scripts/gsc_query.py --dimensions date -o "$DATA_DIR/data/gsc_trends.json"
python scripts/gsc_query.py --mode sitemaps -o "$DATA_DIR/data/gsc_sitemaps.json"

# GA4 data
python scripts/ga4_query.py --preset traffic_overview -o "$DATA_DIR/data/ga4_traffic.json"
python scripts/ga4_query.py --preset top_pages --limit 100 -o "$DATA_DIR/data/ga4_pages.json"
python scripts/ga4_query.py --preset user_acquisition -o "$DATA_DIR/data/ga4_acquisition.json"
python scripts/ga4_query.py --preset device_breakdown -o "$DATA_DIR/data/ga4_devices.json"
python scripts/ga4_query.py --preset landing_pages --limit 50 -o "$DATA_DIR/data/ga4_landing.json"
python scripts/ga4_query.py --preset user_behavior --limit 100 -o "$DATA_DIR/data/ga4_behavior.json"
python scripts/ga4_query.py --preset conversion_events -o "$DATA_DIR/data/ga4_conversions.json"

# Optional: funnel exploration (if user has custom events)
# python scripts/ga4_funnel.py --steps "event1,event2,event3" -o "$DATA_DIR/data/ga4_funnel.json"

# Optional: Bing Webmaster data (if BING_WEBMASTER_API_KEY is configured)
# python scripts/bing_query.py --mode query_stats -o "$DATA_DIR/data/bing_queries.json"
# python scripts/bing_query.py --mode page_stats -o "$DATA_DIR/data/bing_pages.json"
# python scripts/bing_query.py --mode rank_traffic -o "$DATA_DIR/data/bing_traffic.json"
# python scripts/bing_query.py --mode links -o "$DATA_DIR/data/bing_links.json"
# python scripts/bing_query.py --mode crawl_stats -o "$DATA_DIR/data/bing_crawl.json"
```

### Mode B: Manual CSV Export

Send the following export instructions to the user, asking them to place files in `$DATA_DIR/data/`:

> **Export GSC data**:
> 1. Open [Google Search Console](https://search.google.com/search-console/) → Select your site
> 2. Click "Search results" (Performance) in the left menu
> 3. Set date range to last 3 months, click "Export" → "Download CSV"
> 4. Save the downloaded CSV as `$DATA_DIR/data/gsc_export.csv`
>
> **Export GA4 data (export the following reports)**:
> 1. Open [Google Analytics](https://analytics.google.com/) → Select your property
> 2. Export "Pages and screens" report:
>    - Left menu: "Reports" → "Engagement" → "Pages and screens"
>    - Click the share icon (top-right) → "Download file" → CSV
>    - Save as `$DATA_DIR/data/ga4_pages.csv`
> 3. Export "Traffic acquisition" report:
>    - Left menu: "Reports" → "Acquisition" → "Traffic acquisition"
>    - Export CSV → Save as `$DATA_DIR/data/ga4_acquisition.csv`
> 4. Export "Landing pages" report:
>    - Left menu: "Reports" → "Engagement" → "Landing pages"
>    - Export CSV → Save as `$DATA_DIR/data/ga4_landing.csv`
>
> Let me know when the export is complete, and I'll read the files to start analysis.

Also ask the user for:
- **Target website URL** (required, write to `SITE_URL` in `$DATA_DIR/.env`)
- **Source code path** (optional, write to `SOURCE_CODE_PATH`)

### Mode C: Browser Audit Only

Only ask the user for:
- **Target website URL** (required)
- **Source code path** (optional)

Write to `$DATA_DIR/.env` and skip directly to Phase 4 (site audit) and Phase 5 (source code review), skipping Phase 2-3.

### PageSpeed Insights Data Collection

```bash
PSI_BASE="https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=$SITE_URL&category=PERFORMANCE&category=SEO&category=ACCESSIBILITY&category=BEST_PRACTICES"
PSI_KEY_PARAM="${PSI_API_KEY:+&key=$PSI_API_KEY}"
curl -s "${PSI_BASE}&strategy=mobile${PSI_KEY_PARAM}" > "$DATA_DIR/data/psi_mobile.json"
curl -s "${PSI_BASE}&strategy=desktop${PSI_KEY_PARAM}" > "$DATA_DIR/data/psi_desktop.json"
```

> **PSI failure fallback**: If a 429 (quota exceeded) or other error is returned, check whether "PageSpeed Insights API" has been enabled in the Google Cloud project (see Step 1). When PSI data is missing, continue with subsequent phases and note the missing performance data in the report.

---

## 5. GSC Script Reference (gsc_query.py)

> All example commands below assume the venv has been activated: `source "$DATA_DIR/venv/bin/activate"`

Scripts auto-read `GSC_SITE_URL` from `$DATA_DIR/.env` and auto-discover the Service Account JSON key from `$DATA_DIR/configs/`.

### Search Analytics Queries

```bash
# Last 28 days grouped by query and page (default)
python scripts/gsc_query.py

# Specify dimensions and date range
python scripts/gsc_query.py --dimensions query --limit 500 \
    --start-date 2025-01-01 --end-date 2025-03-01

# Group by device and country
python scripts/gsc_query.py --dimensions device,country --limit 100

# View trends by date
python scripts/gsc_query.py --dimensions date

# Output to file
python scripts/gsc_query.py --dimensions query -o gsc_data.json
```

The `--site-url` CLI flag overrides `GSC_SITE_URL` from `.env`.

### Available Dimensions

| Dimension | Description |
|-----------|-------------|
| `query` | Search query term |
| `page` | Page URL |
| `country` | Country code |
| `device` | Device type (DESKTOP/MOBILE/TABLET) |
| `date` | Date |
| `searchAppearance` | Search result appearance type |

### Returned Metrics

Each row contains:
- `clicks` — Click count
- `impressions` — Impression count
- `ctr` — Click-through rate (clicks / impressions)
- `position` — Average ranking position

### Sitemap & URL Inspection

```bash
# Sitemap query
python scripts/gsc_query.py --mode sitemaps

# URL inspection (indexing status, crawl info, mobile usability)
python scripts/gsc_query.py --mode inspect --inspect-url "https://example.com/some-page"
```

### Custom GSC Queries for Targeted Analysis

Beyond the standard data collection, GSC can serve as a powerful auxiliary analysis tool for deeper investigation. When the standard aggregated data is insufficient, write targeted queries to drill down into specific page sections, track daily trends for particular queries, or cross-filter dimensions.

**When to use custom GSC queries**:
- Standard data reveals an area that needs deeper investigation (e.g., a page group with anomalous metrics)
- The user asks about a specific subset of pages or queries
- You need to filter, segment, or cross-reference dimensions that the standard data collection didn't cover

**How to build custom queries**: Use `gsc_query.py` for standard dimension/date queries. For advanced filtering (e.g., `dimensionFilterGroups`), write a custom script following the GSC Search Analytics API patterns. Key capabilities:
- **Dimension filtering**: Filter by page path patterns, query keywords, country, device
- **Cross-dimension filtering**: Filter on a dimension not in the `dimensions` list
- **Fresh data access**: Use `dataState: 'all'` to include preliminary data from last 1-2 days
- **Pagination**: Retrieve up to 25,000 rows per request; paginate with `startRow`
- **Multi-condition filters**: Combine multiple filters within a group using AND logic

**Common analysis scenarios**:
1. **Section-specific analysis**: Filter by page path to isolate a site section
2. **Query trend tracking**: Group by `query` + `date` with a page filter
3. **Long-tail keyword discovery**: Query with high `rowLimit`
4. **Country-specific page performance**: Filter by country and group by page
5. **Regex-based pattern matching**: Use `includingRegex` operator

Save custom query output to `$DATA_DIR/data/gsc_*.json`.

### Common GSC Analysis Scenarios

- **Find High-Impression Low-Click Keywords**: Query `search_analytics` grouped by query; find rows with high `impressions` but low `ctr`.
- **Find Pages with Declining Rankings**: Group by `date,page`; compare `position` changes across different time periods.
- **Find Unindexed Pages**: Use `inspect` mode to check the indexing status of key pages one by one.

---

## 6. GA4 Script Reference (ga4_query.py)

> All example commands below assume the venv has been activated: `source "$DATA_DIR/venv/bin/activate"`

Scripts auto-read `GA4_PROPERTY_ID` from `$DATA_DIR/.env` and auto-discover the Service Account JSON key from `$DATA_DIR/configs/`.

### Preset Query Templates

```bash
python scripts/ga4_query.py --preset traffic_overview       # Daily traffic trends
python scripts/ga4_query.py --preset top_pages --limit 50   # Top pages
python scripts/ga4_query.py --preset user_acquisition       # User acquisition sources
python scripts/ga4_query.py --preset device_breakdown        # Device distribution
python scripts/ga4_query.py --preset geo_distribution        # Geographic distribution
python scripts/ga4_query.py --preset landing_pages           # Landing pages
python scripts/ga4_query.py --preset user_behavior           # User behavior
python scripts/ga4_query.py --preset conversion_events       # Conversion events
python scripts/ga4_query.py --preset demographics_age        # Age distribution (requires Google Signals)
python scripts/ga4_query.py --preset demographics_gender     # Gender distribution (requires Google Signals)
python scripts/ga4_query.py --preset demographics_geo        # Country + city distribution
python scripts/ga4_query.py --preset demographics_language   # Language distribution
python scripts/ga4_query.py --preset demographics_interests  # Interest categories (requires Google Signals)
python scripts/ga4_query.py --preset new_vs_returning        # New vs returning users
```

The `--property-id` CLI flag overrides `GA4_PROPERTY_ID` from `.env`.

### Custom Queries

```bash
python scripts/ga4_query.py \
    --dimensions pagePath,deviceCategory \
    --metrics sessions,bounceRate,averageSessionDuration \
    --start-date 2025-01-01 --end-date 2025-03-01 \
    --order-by="-sessions" --limit 200
```

### Date Formats

Both absolute and relative dates are supported:
- Absolute: `2025-01-01`
- Relative: `today`, `yesterday`, `NdaysAgo` (e.g., `28daysAgo`)

### Common Dimensions

| Dimension | Description |
|-----------|-------------|
| `date` | Date |
| `pagePath` | Page path |
| `pageTitle` | Page title |
| `landingPage` | Landing page |
| `sessionDefaultChannelGroup` | Channel grouping |
| `sessionSource` / `sessionMedium` | Source / Medium |
| `deviceCategory` | Device category |
| `operatingSystem` / `browser` | Operating system / Browser |
| `country` / `city` | Country / City |
| `eventName` | Event name |

### Common Metrics

| Metric | Description |
|--------|-------------|
| `sessions` | Session count |
| `totalUsers` / `newUsers` | Total users / New users |
| `screenPageViews` | Page views |
| `bounceRate` | Bounce rate |
| `averageSessionDuration` | Average session duration (seconds) |
| `engagementRate` / `engagedSessions` | Engagement rate / Engaged sessions |
| `eventCount` | Event count |
| `conversions` | Conversion count |

### Preset Template Reference

| Template | Purpose |
|----------|---------|
| `traffic_overview` | Daily traffic trends; detect traffic anomalies |
| `top_pages` | Find most popular and underperforming pages |
| `user_acquisition` | Analyze user source channel effectiveness |
| `device_breakdown` | Device and browser distribution; detect compatibility issues |
| `geo_distribution` | Geographic distribution; optimize internationalization |
| `landing_pages` | Landing page performance; optimize entry experience |
| `user_behavior` | User behavior paths and engagement depth |
| `conversion_events` | Conversion event tracking and funnel analysis |
| `demographics_age` | User age distribution with engagement and conversion metrics |
| `demographics_gender` | User gender distribution with engagement and conversion metrics |
| `demographics_geo` | User geographic distribution (country + city) |
| `demographics_language` | User language distribution |
| `demographics_interests` | User interest categories (requires Google Signals) |
| `new_vs_returning` | New vs returning user comparison with engagement depth |

---

## 7. GA4 Funnel Exploration (ga4_funnel.py)

> All example commands below assume the venv has been activated: `source "$DATA_DIR/venv/bin/activate"`

### API Coverage

GA4's "Explore" section offers several exploration types. Their API availability:

| Exploration Type | API Support | Method | API Version | Script |
|---|---|---|---|---|
| **Free-form** | ✅ Fully supported | `runReport` | v1beta (stable) | `ga4_query.py` |
| **Funnel exploration** | ✅ Supported | `runFunnelReport` | v1alpha (early preview) | `ga4_funnel.py` |
| **Path exploration** | ❌ No API | — | — | — |
| **Segment overlap** | ❌ No API | — | — | — |
| **User explorer** | ❌ No API | — | — | — |
| **Cohort exploration** | ❌ No API | — | — | — |
| **User lifetime** | ❌ No API | — | — | — |

> **Note**: `runFunnelReport` is in **v1alpha** (early preview). Google may introduce breaking changes before it reaches beta. The API is fully functional but you should monitor the [Google Analytics API announcements](https://groups.google.com/forum/#!forum/google-analytics-api-notify) for updates.

Authentication: Uses the **same Service Account** as `ga4_query.py`. No additional setup needed.

### Quick Start

```bash
# Simple 3-step funnel
python scripts/ga4_funnel.py \
    --steps "page_view,signup,purchase" \
    -o "$DATA_DIR/data/ga4_funnel.json"

# With custom display names
python scripts/ga4_funnel.py \
    --steps "page_view,signup,purchase" \
    --step-names "View Page,Sign Up,Purchase" \
    -o "$DATA_DIR/data/ga4_funnel.json"
```

### CLI Options

| Flag | Description |
|------|-------------|
| `--property-id ID` | GA4 property ID (or set `GA4_PROPERTY_ID` env) |
| `--steps "e1,e2,e3"` | Comma-separated event names for funnel steps |
| `--step-names "n1,n2,n3"` | Comma-separated display names (must match `--steps` count) |
| `--config FILE` | Path to JSON config file for advanced funnel definitions |
| `--start-date DATE` | Start date (default: `28daysAgo`) |
| `--end-date DATE` | End date (default: `yesterday`) |
| `--open` | Open funnel — users can enter at any step (default: closed) |
| `--breakdown DIM` | Dimension to break down by (e.g. `deviceCategory`) |
| `--trended` | Show trended funnel (daily trend per step) |
| `-o FILE` | Output file path (default: stdout) |

### OR Logic in Steps

Use `|` within a step to match multiple events:

```bash
# Step 1 matches either first_open OR first_visit
python scripts/ga4_funnel.py \
    --steps "first_open|first_visit,page_view,purchase" \
    --step-names "First Touch,View Page,Purchase"
```

### Open vs Closed Funnels

| Type | Behavior | Flag |
|------|----------|------|
| **Closed** (default) | Users must enter at step 1. Only users who complete step 1 are tracked. | (none) |
| **Open** | Users can enter the funnel at any step. Each step is evaluated independently. | `--open` |

### Breakdown Dimension

```bash
# See funnel performance by device type
python scripts/ga4_funnel.py \
    --steps "page_view,add_to_cart,purchase" \
    --breakdown deviceCategory

# By traffic source
python scripts/ga4_funnel.py \
    --steps "page_view,signup" \
    --breakdown sessionDefaultChannelGroup
```

### Trended Funnel

```bash
python scripts/ga4_funnel.py \
    --steps "page_view,purchase" \
    --trended --start-date 30daysAgo
```

### Advanced: JSON Config File

For complex funnels with field filters, timing constraints, or mixed conditions:

```json
{
  "steps": [
    {
      "name": "Organic First Visit",
      "events": ["first_open", "first_visit"],
      "field_filter": {
        "field_name": "firstUserMedium",
        "value": "organic",
        "match_type": "CONTAINS"
      }
    },
    {
      "name": "View Product",
      "events": ["view_item"]
    },
    {
      "name": "Add to Cart",
      "events": ["add_to_cart"],
      "within_duration": "1800s"
    },
    {
      "name": "Purchase",
      "events": ["purchase", "in_app_purchase"],
      "directly_followed_by": true
    }
  ],
  "open": false,
  "breakdown": "deviceCategory",
  "trended": false
}
```

```bash
python scripts/ga4_funnel.py --config funnel_config.json -o funnel_report.json
```

**Config step fields**:

| Field | Type | Description |
|-------|------|-------------|
| `name` | string | Step display name (required) |
| `events` | string[] | Event names — matched with OR logic |
| `field_filter` | object | Dimension field filter: `{field_name, value, match_type}` |
| `field_filter.match_type` | string | `EXACT`, `BEGINS_WITH`, `ENDS_WITH`, `CONTAINS`, `FULL_REGEXP`, `PARTIAL_REGEXP` |
| `directly_followed_by` | bool | If `true`, this step must immediately follow the previous one |
| `within_duration` | string | Max time from prior step, e.g. `"300s"` (5 minutes) |

**Config top-level fields** (optional, CLI flags override):

| Field | Type | Description |
|-------|------|-------------|
| `open` | bool | Open funnel |
| `breakdown` | string | Breakdown dimension |
| `trended` | bool | Trended funnel |

### Output Format

```json
{
  "funnel_table": {
    "dimensions": ["funnelStepId", "funnelStepName", ...],
    "metrics": ["activeUsers", "funnelStepCompletionRate", "funnelStepAbandonments", "funnelStepAbandonmentRate"],
    "rows": [
      {
        "funnelStepId": "0",
        "funnelStepName": "View Page",
        "activeUsers": 5000,
        "funnelStepCompletionRate": 0.35,
        "funnelStepAbandonments": 3250,
        "funnelStepAbandonmentRate": 0.65
      }
    ]
  },
  "funnel_visualization": {
    "dimensions": ["funnelStepId", "funnelStepName"],
    "metrics": ["activeUsers"],
    "rows": [...]
  },
  "query": {
    "property_id": "123456789",
    "date_range": {"start": "28daysAgo", "end": "yesterday"},
    "is_open_funnel": false,
    "breakdown": "deviceCategory",
    "trended": false,
    "step_count": 3,
    "steps": ["View Page", "Sign Up", "Purchase"]
  }
}
```

**Key metrics in `funnel_table`**:

| Metric | Description |
|--------|-------------|
| `activeUsers` | Number of users who reached this step |
| `funnelStepCompletionRate` | % of users from this step who advanced to the next step |
| `funnelStepAbandonments` | Number of users who dropped off at this step |
| `funnelStepAbandonmentRate` | % of users who dropped off at this step |

When `--breakdown` is used, rows include the breakdown dimension value. The special value `RESERVED_TOTAL` indicates the total row across all dimension values.

### Sampling

Funnel reports may use data sampling. If the response includes a `"sampling"` key, the data is sampled:

```json
{
  "sampling": [
    {
      "samples_read_count": 500000,
      "sampling_space_size": 5000000
    }
  ]
}
```

This means 500,000 out of 5,000,000 events were sampled (10% sample rate). Results are extrapolated from this sample.

### Path Exploration — No API

Google has **not** released any API for path exploration. Alternatives:
1. **GA4 web UI**: Use the path exploration directly in the GA4 Explore section
2. **BigQuery export**: If your GA4 property is linked to BigQuery, query raw event-level data to reconstruct user paths with SQL
3. **Approximate with ga4_query.py**: Query `pagePath` + `eventName` dimensions to get page-level event distributions (not true sequential paths)

### Common Dimensions for Breakdown

| Dimension | Description |
|-----------|-------------|
| `deviceCategory` | Device type (desktop, mobile, tablet) |
| `country` / `city` | Geographic location |
| `sessionDefaultChannelGroup` | Traffic channel (Organic, Direct, Social, etc.) |
| `sessionSource` / `sessionMedium` | Traffic source and medium |
| `operatingSystem` / `browser` | OS and browser |
| `newVsReturning` | New vs returning users |

---

## 8. Bing Webmaster Script Reference (bing_query.py)

> All example commands below assume the venv has been activated: `source "$DATA_DIR/venv/bin/activate"`

| Item | Details |
|------|---------|
| **Base URL** | `https://ssl.bing.com/webmaster/api.svc/json/` |
| **Authentication** | API Key (passed as `apikey` query parameter) or OAuth 2.0 Bearer token |
| **Response format** | JSON |
| **Data retention** | **6 months** (shorter than GSC's ~16 months — set up regular data collection) |
| **Rate limits** | ~40-50 calls/day per API key for stats endpoints |

Scripts auto-read `BING_WEBMASTER_API_KEY` and `SITE_URL` from `$DATA_DIR/.env`.

### Search Traffic Queries

```bash
# Top queries with traffic stats (default)
python scripts/bing_query.py

# Top pages by traffic
python scripts/bing_query.py --mode page_stats

# Overall rank & traffic stats (daily impressions/clicks)
python scripts/bing_query.py --mode rank_traffic

# Stats for a specific query
python scripts/bing_query.py --mode query_detail --query "online 3d model converter"

# Stats for a specific page
python scripts/bing_query.py --mode page_detail --page "https://example.com/tools/converter"

# Stats for a specific query + page combination
python scripts/bing_query.py --mode query_page_detail --query "3d converter" --page "https://example.com/tools/converter"

# Keyword research (with country and date range)
python scripts/bing_query.py --mode keyword --query "image compressor" --country us \
    --start-date 2026-01-01 --end-date 2026-03-01

# Related keywords
python scripts/bing_query.py --mode related_keywords --query "image compressor" --country us \
    --start-date 2026-01-01 --end-date 2026-03-01

# Inbound link analysis
python scripts/bing_query.py --mode links

# Crawl stats
python scripts/bing_query.py --mode crawl_stats

# Output to file
python scripts/bing_query.py --mode query_stats -o "$DATA_DIR/data/bing_queries.json"
```

The `--site-url` CLI flag overrides `SITE_URL` from `.env`.

### Available Modes

| Mode | API Method | Description |
|------|-----------|-------------|
| `query_stats` | `GetQueryStats` | Top queries with impressions, clicks, position |
| `page_stats` | `GetPageStats` | Top pages with traffic stats |
| `rank_traffic` | `GetRankAndTrafficStats` | Daily impressions & clicks overview |
| `query_detail` | `GetQueryTrafficStats` | Detailed stats for a specific query |
| `page_detail` | `GetPageQueryStats` | All queries driving traffic to a specific page |
| `query_page_detail` | `GetQueryPageDetailStats` | Stats for a specific query + page combination |
| `keyword` | `GetKeyword` | Keyword impression data for a country & date range |
| `related_keywords` | `GetRelatedKeywords` | Related keyword suggestions with impressions |
| `links` | `GetLinkCounts` | Pages with inbound links and link counts |
| `crawl_stats` | `GetCrawlStats` | Crawl frequency, errors, and patterns |

### Returned Metrics

Each traffic stats row contains:
- `Impressions` — Impression count on Bing search
- `Clicks` — Click count from Bing search
- `AvgClickPosition` — Average ranking position (where available)
- `Date` — Date of the data point (for time-series endpoints)

Keyword endpoints return:
- `Impressions` — Search volume / impression count for the keyword
- `BroadImpressions` — Broad match impression count
- `Query` — The keyword text

### Comparison with GSC API

| Feature | GSC API | Bing Webmaster API |
|---------|---------|-------------------|
| Auth method | Service Account (OAuth2) | API Key or OAuth 2.0 |
| Dimension filtering | `dimensionFilterGroups` with operators | Per-endpoint (query/page/query+page) |
| Custom date range | `startDate` / `endDate` on all queries | Available on keyword endpoints; traffic stats return all available data |
| Data retention | ~16 months | **6 months** |
| Keyword research | Not available | ✅ `GetKeyword`, `GetRelatedKeywords` |
| Backlink analysis | Not available | ✅ `GetLinkCounts`, `GetUrlLinks` |
| Crawl stats | Limited (via URL Inspection) | ✅ `GetCrawlStats`, `GetCrawlIssues` |
| URL submission | Not via same API | ✅ `SubmitUrl`, `SubmitUrlBatch` |
| Rate limit | ~25,000 rows/request | ~40-50 calls/day |

### Common Bing Analysis Scenarios

- **Find Top Performing Queries on Bing**: Use `query_stats` mode. Sort by impressions or clicks.
- **Analyze a Specific Page's Search Performance**: Use `page_detail` mode with `--page`.
- **Keyword Research & Discovery**: Use `keyword` and `related_keywords` modes — a capability GSC doesn't offer.
- **Monitor Crawl Health**: Use `crawl_stats` mode. Cross-reference with crawl issues.
- **Compare Google vs Bing Performance**: Run the same analysis period on both GSC and Bing data to identify keywords/pages where rankings differ.


path: references/user-persona-analysis-reference.md
content: # User Persona Analysis Reference

Data-driven user persona analysis methodology for websites and applications. Combines GA4 behavioral data, GSC search intent data, and live-site observation to build evidence-backed user profiles that directly guide product iteration and marketing optimization.

---

## 1. What Is Data-Driven User Persona Analysis?

A **data-driven user persona** is a fictional but evidence-backed representation of a user type, built from quantitative behavioral data and qualitative insights. Unlike traditional personas based on assumptions, data-driven personas are:

- **Verifiable**: Every attribute is backed by analytics data
- **Measurable**: Key metrics track each persona's behavior over time
- **Actionable**: Each persona maps to concrete product/marketing decisions
- **Iterative**: Personas are refreshed as new data arrives (recommended quarterly)

---

## 2. Persona Analysis Framework

### 2.1 The Five-Dimension Model

Build each persona across five dimensions, from outer (observable) to inner (inferred):

```
┌───────────────────────────────────────────────────┐
│  1. WHO — Demographics & Environment              │
│  ┌───────────────────────────────────────────────┐│
│  │  2. HOW — Behavioral Patterns                 ││
│  │  ┌───────────────────────────────────────────┐││
│  │  │  3. WHERE FROM — Acquisition & Context    │││
│  │  │  ┌───────────────────────────────────────┐│││
│  │  │  │  4. WHAT — Content & Feature Affinity ││││
│  │  │  │  ┌───────────────────────────────────┐││││
│  │  │  │  │  5. WHY — Intent & Goals          │││││
│  │  │  │  └───────────────────────────────────┘││││
│  │  │  └───────────────────────────────────────┘│││
│  │  └───────────────────────────────────────────┘││
│  └───────────────────────────────────────────────┘│
└───────────────────────────────────────────────────┘
```

| Dimension | Key Questions | Primary Data Source |
|-----------|---------------|---------------------|
| **1. WHO** | Age, gender, location, language, device, OS | GA4 demographics + device reports |
| **2. HOW** | Session patterns, engagement depth, visit frequency, journey paths | GA4 behavior + funnel data |
| **3. WHERE FROM** | Traffic channels, search queries, referral sources, first-touch attribution | GA4 acquisition + GSC queries |
| **4. WHAT** | Top pages visited, content preferences, feature usage, search terms | GA4 pages + events + site search |
| **5. WHY** | User intent stage, conversion behavior, goal completion | GSC intent mapping + GA4 conversions |

### 2.2 Analysis Phases

```
Phase A: Data Collection & Enrichment
  ↓
Phase B: Segmentation & Clustering
  ↓
Phase C: Persona Construction
  ↓
Phase D: Validation & Insights
  ↓
Phase E: Application & Iteration Recommendations
```

---

## 3. Phase A: Data Collection & Enrichment

### 3.1 GA4 Data Collection

Collect the following data using `ga4_query.py` presets and custom queries:

```bash
source "$DATA_DIR/venv/bin/activate"
set -a; source "$DATA_DIR/.env"; set +a

# ── Demographics ──────────────────────────────────────────
python scripts/ga4_query.py --preset demographics_age -o "$DATA_DIR/data/ga4_persona_age.json"
python scripts/ga4_query.py --preset demographics_gender -o "$DATA_DIR/data/ga4_persona_gender.json"
python scripts/ga4_query.py --preset demographics_geo -o "$DATA_DIR/data/ga4_persona_geo.json"
python scripts/ga4_query.py --preset demographics_language -o "$DATA_DIR/data/ga4_persona_language.json"
python scripts/ga4_query.py --preset demographics_interests -o "$DATA_DIR/data/ga4_persona_interests.json"

# ── Behavioral Patterns ───────────────────────────────────
python scripts/ga4_query.py --preset device_breakdown -o "$DATA_DIR/data/ga4_persona_devices.json"
python scripts/ga4_query.py --preset user_behavior --limit 200 -o "$DATA_DIR/data/ga4_persona_behavior.json"
python scripts/ga4_query.py --preset landing_pages --limit 100 -o "$DATA_DIR/data/ga4_persona_landing.json"
python scripts/ga4_query.py --preset top_pages --limit 200 -o "$DATA_DIR/data/ga4_persona_pages.json"

# ── Acquisition ───────────────────────────────────────────
python scripts/ga4_query.py --preset user_acquisition -o "$DATA_DIR/data/ga4_persona_acquisition.json"
python scripts/ga4_query.py --preset traffic_overview -o "$DATA_DIR/data/ga4_persona_traffic.json"

# ── New vs Returning ──────────────────────────────────────
python scripts/ga4_query.py --preset new_vs_returning -o "$DATA_DIR/data/ga4_persona_new_returning.json"

# ── Engagement Depth (custom) ─────────────────────────────
python scripts/ga4_query.py \
    --dimensions newVsReturning,deviceCategory \
    --metrics sessions,totalUsers,bounceRate,averageSessionDuration,engagementRate,screenPageViewsPerSession \
    -o "$DATA_DIR/data/ga4_persona_engagement_segments.json"

# ── Hourly / Day-of-Week Patterns (custom) ────────────────
python scripts/ga4_query.py \
    --dimensions dayOfWeekName,hour \
    --metrics sessions,totalUsers \
    --order-by="-sessions" \
    -o "$DATA_DIR/data/ga4_persona_time_patterns.json"

# ── Conversion by Segment (custom) ────────────────────────
python scripts/ga4_query.py \
    --dimensions sessionDefaultChannelGroup,deviceCategory \
    --metrics sessions,conversions,engagementRate \
    --order-by="-sessions" \
    -o "$DATA_DIR/data/ga4_persona_conversion_segments.json"
```

> **Note on Demographics data**: Age, Gender, and Interests dimensions (`userAgeBracket`, `userGender`, `brandingInterest`) require **Google Signals** to be enabled in the GA4 property. If not enabled, these queries return empty results. In that case, note the gap and proceed with behavioral data only. See [data-collection-reference.md](data-collection-reference.md) for auth setup.

### 3.2 GSC Search Intent Data

```bash
# Top queries reveal what users are looking for
python scripts/gsc_query.py --dimensions query --limit 500 -o "$DATA_DIR/data/gsc_persona_queries.json"

# Query + page mapping shows intent-to-content alignment
python scripts/gsc_query.py --dimensions query,page --limit 500 -o "$DATA_DIR/data/gsc_persona_query_pages.json"

# Device distribution in search
python scripts/gsc_query.py --dimensions device --limit 100 -o "$DATA_DIR/data/gsc_persona_devices.json"

# Country distribution in search
python scripts/gsc_query.py --dimensions country --limit 100 -o "$DATA_DIR/data/gsc_persona_countries.json"
```

### 3.3 Supplementary Data (If Available)

| Data Source | What It Reveals | How to Collect |
|-------------|-----------------|----------------|
| **Bing Webmaster** | Keyword research, related queries | `bing_query.py --mode keyword` / `--mode related_keywords` |
| **Site search logs** | On-site user intent | GA4 `searchTerm` dimension |
| **Funnel data** | Conversion journey patterns | `ga4_funnel.py` (see [data-collection-reference.md](data-collection-reference.md) §7) |
| **Heatmaps / Session recordings** | Click patterns, attention zones | Third-party tools (Hotjar, Clarity) |
| **User surveys / Interviews** | Motivations, pain points, goals | Qualitative research methods |
| **CRM / Support tickets** | Common questions, user frustrations | Internal data export |

---

## 4. Phase B: Segmentation & Clustering

### 4.1 Segmentation Strategies

Apply multiple segmentation lenses to identify distinct user groups:

#### Strategy 1: Behavioral Segmentation (Primary)

Segment by engagement level using GA4 metrics:

| Segment | Definition | Key Indicators |
|---------|------------|----------------|
| **Power Users** | High frequency + high engagement + repeat visits | `sessionsPerUser` > 3, `engagementRate` > 80%, `newVsReturning` = returning |
| **Engaged Explorers** | Moderate frequency, deep content consumption | `screenPageViewsPerSession` > 4, `averageSessionDuration` > 3 min |
| **Quick Scanners** | Brief visits, shallow engagement | `bounceRate` > 70%, `averageSessionDuration` < 30s |
| **One-Time Visitors** | Single visit, never return | `newVsReturning` = new, `sessions` = 1 |
| **Converting Users** | Completed key conversion events | `conversions` > 0, `keyEvents` > 0 |

#### Strategy 2: Acquisition Channel Segmentation

| Segment | Channel | Typical Behavior |
|---------|---------|------------------|
| **Organic Seekers** | Organic Search (Google/Bing) | Intent-driven, goal-oriented, query patterns reveal needs |
| **Social Browsers** | Social Media | Discovery-driven, shorter sessions, content-focused |
| **Direct Loyalists** | Direct | Repeat visitors, brand-aware, deeper engagement |
| **Referral Explorers** | Referral | Curiosity-driven, evaluate against known alternatives |
| **Paid Converters** | Paid Search / Display | High intent, expect immediate value delivery |

#### Strategy 3: Demographics + Device Cross-Segmentation

| Lens | Dimensions | Use Case |
|------|-----------|----------|
| Age × Device | `userAgeBracket` × `deviceCategory` | Younger users on mobile vs older on desktop |
| Geo × Language | `country` × `language` | Localization priorities |
| Device × Channel | `deviceCategory` × `sessionDefaultChannelGroup` | Mobile-social vs desktop-organic patterns |
| Gender × Interests | `userGender` × `brandingInterest` | Content and marketing personalization |

#### Strategy 4: Intent-Based Segmentation (from GSC)

Classify search queries by user journey stage:

| Intent Stage | Query Pattern Examples | User Need |
|--------------|----------------------|-----------|
| **Awareness** | "what is...", "how to...", informational | Learning, exploring |
| **Consideration** | "[product] vs [alternative]", "best [category]", comparison | Evaluating options |
| **Decision** | "[brand] pricing", "[product] signup", transactional | Ready to act |
| **Retention** | "[brand] login", "[brand] support", navigational | Existing user needs |

### 4.2 Segmentation Analysis Script

Write a Python script in `$DATA_DIR/scripts/` to process the collected data and produce segmented profiles. The script should:

1. Load all `ga4_persona_*.json` and `gsc_persona_*.json` from `$DATA_DIR/data/`
2. Cross-tabulate dimensions (e.g., device × channel × new_vs_returning)
3. Calculate per-segment metrics (engagement rate, bounce rate, conversion rate, avg session duration)
4. Identify 3-5 distinct user clusters based on behavioral patterns
5. Output structured JSON to `$DATA_DIR/data/persona_segments.json`
6. Generate persona charts to `$DATA_DIR/charts/persona_*.png`

---

## 5. Phase C: Persona Construction

### 5.1 Persona Card Template

For each identified persona (recommend 3-5 personas), construct a card with:

```markdown
## Persona: [Name] — "[Tagline]"

### Quick Profile
| Attribute | Value |
|-----------|-------|
| **Archetype** | [e.g., Power User / Casual Browser / Intent Seeker] |
| **% of Total Users** | [e.g., 35%] |
| **% of Conversions** | [e.g., 60%] |
| **Primary Channel** | [e.g., Organic Search] |
| **Primary Device** | [e.g., Desktop] |

### Demographics (if available)
| Attribute | Value |
|-----------|-------|
| **Age Range** | [e.g., 25-34] |
| **Gender** | [e.g., 65% Male] |
| **Top Locations** | [e.g., US, UK, India] |
| **Languages** | [e.g., English, Chinese] |
| **Interests** | [e.g., Technology, Business] |

### Behavioral Fingerprint
| Metric | Value | vs Site Average |
|--------|-------|-----------------|
| **Sessions / User** | [value] | [+/-% vs avg] |
| **Avg Session Duration** | [value] | [+/-% vs avg] |
| **Pages / Session** | [value] | [+/-% vs avg] |
| **Bounce Rate** | [value] | [+/-% vs avg] |
| **Engagement Rate** | [value] | [+/-% vs avg] |
| **Conversion Rate** | [value] | [+/-% vs avg] |

### Journey Pattern
- **Entry Point**: [typical landing pages]
- **Content Preferences**: [top pages / content types consumed]
- **Search Queries**: [representative GSC queries that bring this persona]
- **Visit Pattern**: [e.g., weekday mornings, mobile-first, 2-3 visits before converting]

### Needs & Pain Points (Inferred)
1. [Need inferred from behavior data + search queries]
2. [Pain point inferred from high bounce / exit pages]

### Product Iteration Opportunities
1. [Specific recommendation tied to this persona's behavior]
2. [Content/feature gap identified from their journey]
```

### 5.2 Key Principles

1. **Name each persona** — use memorable names (e.g., "Alex the Analyst", "Sarah the Searcher"), not "Persona 1"
2. **Data-backed attributes only** — every attribute must trace to specific data evidence
3. **Decision-relevant** — only include attributes that influence product or marketing decisions; remove noise
4. **Quantify differences** — show how each persona differs from site averages and from other personas
5. **Link to actions** — each persona must map to at least 2-3 concrete product/marketing recommendations

---

## 6. Phase D: Validation & Insights

### 6.1 Cross-Validation Methods

| Method | Description | Data Source |
|--------|-------------|-------------|
| **Behavioral consistency** | Check if persona segments show consistent patterns across time periods | Compare 30-day windows |
| **Channel-behavior alignment** | Verify that channel-based segments align with behavioral segments | Cross-tab acquisition × behavior |
| **Conversion correlation** | Confirm that high-engagement personas actually convert more | Conversion rate per segment |
| **Search intent match** | Verify GSC intent stages align with GA4 journey behavior | Map queries → landing pages → next actions |
| **Cohort stability** | Check if persona distributions remain stable over time | Monthly segment size tracking |

### 6.2 Key Insights to Extract

For each persona, answer these questions:

**Acquisition Efficiency**:
- Which channels bring the most users of this type?
- What's the cost-efficiency of acquiring this persona? (if ad data available)
- What search queries attract this persona?

**Engagement Quality**:
- Which content keeps this persona engaged?
- Where do they drop off in the journey?
- What's their typical visit cadence?

**Conversion Potential**:
- What's this persona's conversion rate?
- What % of revenue/conversions does this persona drive?
- What's the gap between their current and potential conversion rate?

**Retention Signals**:
- Does this persona return? How often?
- What triggers return visits?
- What are the early warning signs of churn for this persona?

---

## 7. Phase E: Application & Iteration Recommendations

### 7.1 Product Iteration Matrix

Map each persona to specific product decisions:

| Persona | UX Priority | Content Strategy | Feature Priority | Marketing Channel |
|---------|-------------|-----------------|------------------|-------------------|
| [Name A] | Mobile-first optimization | In-depth tutorials | Advanced features | SEO + Content marketing |
| [Name B] | Fast load, clear CTA | Quick-start guides | Core features, simple UI | Paid search |
| [Name C] | Browsing-friendly layout | Visual, shareable content | Social sharing, bookmarks | Social media |

### 7.2 Actionable Recommendations Checklist

For each persona, generate recommendations in these categories:

#### Content & Messaging
- [ ] Content gaps: topics this persona searches for but can't find on the site
- [ ] Content format: does this persona prefer long-form, video, interactive, or quick-reference?
- [ ] Messaging tone: formal vs casual, technical vs beginner-friendly
- [ ] Localization needs: language/regional preferences

#### UX & Design
- [ ] Device optimization: mobile-specific issues for mobile-heavy personas
- [ ] Navigation: are key pages easily reachable for this persona's entry point?
- [ ] Page speed: impact on bounce rate for this persona's device/network
- [ ] CTA placement: aligned with this persona's scrolling and engagement patterns

#### Conversion Optimization
- [ ] Landing page alignment: do landing pages match this persona's search intent?
- [ ] Conversion path: simplify the path from this persona's entry to conversion
- [ ] Trust signals: what does this persona need to see before converting?
- [ ] Friction points: specific pages where this persona drops off

#### Acquisition Strategy
- [ ] Channel investment: increase spend on channels that bring high-value personas
- [ ] Keyword strategy: target queries that attract converting personas
- [ ] Content marketing: create content for underserved persona segments
- [ ] Retargeting: build audience lists based on persona behavior patterns

---

## 8. Visualization Requirements

Generate the following charts for persona analysis (follow patterns in [data-visualization-guide.md](data-visualization-guide.md)):

### Required Charts

| Chart | Type | Description | Output File |
|-------|------|-------------|-------------|
| **Demographics overview** | Multi-panel (pie + bar) | Age, gender, top countries distribution | `persona_demographics.png` |
| **Device & technology** | Stacked bar / pie | Device type, OS, browser distribution | `persona_devices.png` |
| **Channel distribution** | Pie / horizontal bar | Traffic sources per persona | `persona_channels.png` |
| **Engagement comparison** | Grouped bar | Key metrics (bounce rate, engagement rate, session duration) by persona | `persona_engagement.png` |
| **Content affinity heatmap** | Heatmap / grouped bar | Top content categories per persona | `persona_content.png` |
| **Journey flow** | Sankey or funnel | Entry → key pages → conversion/exit per persona | `persona_journey.png` |
| **Time patterns** | Heatmap (day × hour) | Visit patterns by day-of-week and hour | `persona_time_patterns.png` |
| **New vs Returning** | Stacked bar | New/returning ratio with engagement overlay | `persona_new_returning.png` |
| **Persona summary** | Radar / spider chart | Multi-metric comparison across all personas | `persona_radar.png` |
| **Interests** | Horizontal bar | Top interest categories (if Google Signals enabled) | `persona_interests.png` |

### Chart Embedding in Report

```markdown
![Demographics Overview](../charts/persona_demographics.png)
![Persona Engagement Comparison](../charts/persona_engagement.png)
![User Journey by Persona](../charts/persona_journey.png)
```

---

## 9. Output & Report

Save the persona analysis report to `$DATA_DIR/analysis/user-persona-report.md`.

### Report Structure

```markdown
# User Persona Analysis Report

## Overview
- **Website**: [URL]
- **Analysis Period**: [start_date] ~ [end_date]
- **Total Users Analyzed**: [count]
- **Personas Identified**: [count]
- **Data Sources**: [GA4 demographics / GA4 behavior / GSC queries / ...]

## Key Findings Summary
[3-5 bullet points: most impactful persona insights]

## Site-Wide User Profile
[Aggregate demographics, device, channel, behavior overview — the "average user"]

![Demographics Overview](../charts/persona_demographics.png)
![Device Distribution](../charts/persona_devices.png)
![Time Patterns](../charts/persona_time_patterns.png)

## Persona Profiles
### Persona 1: [Name] — "[Tagline]"
[Full persona card using template from §5.1]

### Persona 2: [Name] — "[Tagline]"
...

## Cross-Persona Comparison

![Engagement Comparison](../charts/persona_engagement.png)
![Persona Radar](../charts/persona_radar.png)

| Metric | [Persona A] | [Persona B] | [Persona C] | Site Average |
|--------|-------------|-------------|-------------|--------------|
| % of Users | ... | ... | ... | 100% |
| % of Conversions | ... | ... | ... | 100% |
| Bounce Rate | ... | ... | ... | ... |
| Engagement Rate | ... | ... | ... | ... |
| Avg Session Duration | ... | ... | ... | ... |
| Pages / Session | ... | ... | ... | ... |

## Product Iteration Recommendations
[Organized by persona, using the matrix from §7.1]

## Content Strategy Recommendations
[Content gaps, format preferences, localization needs per persona]

## Data Gaps & Next Steps
| Missing Data | Impact | How to Collect |
|---|---|---|
| [e.g., Google Signals not enabled] | Cannot determine age/gender | Enable in GA4 Admin |
| [e.g., No custom events] | Cannot track micro-conversions | Add event tracking |
| [e.g., No user survey data] | Cannot validate intent assumptions | Run user survey |

## Appendix: Raw Segment Data
[Link to or embed key data tables for reference]
```

---

## 10. GA4 API Dimensions & Metrics Quick Reference

### Demographics Dimensions

| API Name | Description | Requires Google Signals |
|----------|-------------|------------------------|
| `userAgeBracket` | Age range (18-24, 25-34, etc.) | ✅ Yes |
| `userGender` | Gender (male, female) | ✅ Yes |
| `brandingInterest` | Interest categories | ✅ Yes |
| `country` | Country | No |
| `city` | City | No |
| `region` | Region | No |
| `continent` | Continent | No |
| `language` | Browser language | No |
| `languageCode` | Language code (e.g., en-us) | No |

### Behavioral Dimensions

| API Name | Description |
|----------|-------------|
| `newVsReturning` | New or returning user |
| `deviceCategory` | Desktop / mobile / tablet |
| `operatingSystem` | OS name |
| `browser` | Browser name |
| `screenResolution` | Screen resolution |
| `dayOfWeekName` | Day of week |
| `hour` | Hour of day (0-23) |

### Key Metrics for Persona Analysis

| API Name | Description | Use In Persona |
|----------|-------------|----------------|
| `totalUsers` | Total unique users | Segment sizing |
| `newUsers` | First-time users | New user ratio |
| `sessions` | Total sessions | Visit frequency |
| `sessionsPerUser` | Sessions per user | Loyalty indicator |
| `bounceRate` | Bounce rate | Engagement quality |
| `engagementRate` | Engagement rate | Content relevance |
| `averageSessionDuration` | Avg session duration (sec) | Depth of interest |
| `screenPageViewsPerSession` | Pages per session | Exploration depth |
| `conversions` | Conversion count | Business value |
| `eventCount` | Total events | Activity level |
| `userEngagementDuration` | Total engagement time (sec) | Time invested |

### Custom User Properties

If the site tracks custom user attributes (e.g., membership level, plan type), query them via:

```bash
python scripts/ga4_query.py \
    --dimensions "customUser:membership_level" \
    --metrics sessions,totalUsers,engagementRate,conversions \
    -o "$DATA_DIR/data/ga4_persona_custom.json"
```

> Custom user properties must be registered in GA4 Admin > Custom definitions before querying via API.


path: references/SEO-GEO-Optimization-Checklist.md
content: # SEO & GEO Optimization Checklist

> **Purpose**: Reference document for the AI agent to audit any website's SEO (Search Engine Optimization) and GEO (Generative Engine Optimization) readiness.
> **Core principle**: SEO optimizes rankings; GEO optimizes AI citations. They are complementary, not competing.
> **Variable**: All commands below use `$SITE_URL` — the target website URL from `.env`.

---

## Table of Contents

- [1. Structured Data (JSON-LD)](#1-structured-data-json-ld)
- [2. AI Readability (GEO Core)](#2-ai-readability-geo-core)
- [3. Page Content Depth](#3-page-content-depth)
- [4. Technical SEO Basics](#4-technical-seo-basics)
- [5. Page Performance](#5-page-performance)
- [6. Off-page Authority Building](#6-off-page-authority-building)
- [7. Detection Commands](#7-detection-commands)
- [8. Priority Overview](#8-priority-overview)

---

## 1. Structured Data (JSON-LD)

### 1.1 Recommended JSON-LD Types

Select the types applicable to the target site. Not every site needs all types — choose based on site category.

| @type | Applicable Pages | Purpose | Check Points |
|-------|-----------------|---------|-------------|
| **WebSite** | All pages | Site-wide info with search box | Localize for each language version |
| **Organization** | All pages | Organization info (name, logo, contact) | `inLanguage` must match page language |
| **WebPage** | All pages | Current page info | Include `speakable` (voice search) and `dateModified` |
| **BreadcrumbList** | All pages | Breadcrumb navigation | Multi-level paths, e.g. `Home > Category > Page` |
| **SoftwareApplication** | Tool/product pages | Tool/product description | Each page must have unique content, not templated |
| **FAQPage** | Content/tool pages | FAQ section | 3-5 targeted Q&As matching user search intent per page |
| **ItemList** | Category/listing pages | List items in a category | Helps search engines understand page hierarchy |
| **HowTo** | Tutorial/tool pages | Step-by-step guide | 3-5 steps; can trigger Google rich snippets |
| **Article** | Blog/article pages | Article metadata | Include author, datePublished, dateModified |
| **Product** | E-commerce pages | Product info | Include price, availability, reviews |
| **LocalBusiness** | Local business sites | Location info | Include address, hours, phone |

### 1.2 Checklist

- [ ] **Coverage**: All indexable pages have at least WebSite + WebPage + BreadcrumbList JSON-LD
- [ ] **SSR output**: JSON-LD is rendered server-side in the HTML (some crawlers don't execute JS)
- [ ] **Content uniqueness**: Each page has distinct `name`, `description`, and FAQ content
- [ ] **Localization**: Multi-language sites have localized JSON-LD for each language version
- [ ] **Language tag**: `inLanguage` field is correct (e.g. `en-US`, `zh-CN`)
- [ ] **URL consistency**: `url` in JSON-LD matches the canonical URL

### 1.3 Common Pitfalls

| Problem | Symptom | Solution |
|---------|---------|---------|
| SSR empty shell | Page returns `<meta http-equiv="refresh">` self-redirect with no real content | Check SSG/SSR pre-rendering config (Nuxt/Next/Astro), ensure routes are correctly built |
| Trailing slash mismatch | `/page` returns 301 → `/page/`, causing crawl issues | Unify URL format; use `curl -sL` (follow redirects) when testing |
| Unlocalized JSON-LD | Non-English pages still have English WebSite/Organization | Configure localized JSON-LD blocks for each language in i18n setup |
| Duplicate content | Multiple pages share identical JSON-LD name/description | Generate page-specific structured data from actual page content |

---

## 2. AI Readability (GEO Core)

### 2.1 llms.txt File (P0 — Must Do)

> `llms.txt` is a standard proposed by Jeremy Howard in 2024, adopted by Anthropic, Mintlify and others. It provides AI crawlers with a curated site map.

- [ ] **`/llms.txt`**: Summary version — site overview + categorized page list with links and one-line descriptions
- [ ] **`/llms-full.txt`**: Full version — each page with URL + detailed description + use cases + technical details
- [ ] **robots.txt reference**: Add `Llms-txt: https://yoursite.com/llms.txt` at the end of robots.txt

**llms.txt template**:

```markdown
# Site Name

> One-line description of the site's purpose and core value

- Key feature 1
- Key feature 2
- Site URL

## Category 1
- [Page Name](URL): One-line description

## Category 2
- [Page Name](URL): One-line description

## Optional
- [Secondary Content](URL): Description
```

**llms-full.txt template**:

```markdown
# Site Name — Complete Reference

> Detailed description

## Overview
- Total pages/tools, privacy model, tech stack, supported languages, etc.

## Category (N items)

### Page/Tool Name
- **URL**: https://...
- **Description**: Detailed description (include technical details)
- **Use cases**: Usage scenario list

## Technical Details
- Tech stack, processing methods, i18n, SEO config, etc.

## Contact
- Contact info
```

### 2.2 robots.txt AI Crawler Configuration

- [ ] Explicitly allow major AI crawlers (global + China):

```
# --- Global AI Crawlers ---
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

# --- Chinese AI Crawlers ---
User-agent: Bytespider
Allow: /

User-agent: Doubaobot
Allow: /

User-agent: ERNIEBot
Allow: /

User-agent: Baiduspider
Allow: /

User-agent: DeepSeekBot
Allow: /

User-agent: QwenBot
Allow: /

User-agent: TongyiBot
Allow: /
```

| Crawler | Company | AI Product |
|---------|---------|------------|
| GPTBot / ChatGPT-User | OpenAI | ChatGPT, GPT Search |
| ClaudeBot / anthropic-ai | Anthropic | Claude |
| PerplexityBot | Perplexity | Perplexity AI Search |
| Google-Extended | Google | Gemini, AI Overviews |
| Bytespider / Doubaobot | ByteDance | Doubao (豆包) |
| ERNIEBot / Baiduspider | Baidu | ERNIE Bot (文心一言) |
| DeepSeekBot | DeepSeek | DeepSeek |
| QwenBot / TongyiBot | Alibaba | Qwen / Tongyi Qianwen (通义千问) |

- [ ] Avoid over-restriction: A single `Allow: /` per bot is sufficient
- [ ] Note: ByteSpider has aggressive crawl rates — if bandwidth is a concern, consider adding `Crawl-delay` or rate-limiting at the server level rather than blocking entirely

### 2.3 H2/H3 Heading Format (Question-style)

> AI engines pattern-match headings against user queries. Question-style headings are cited far more often than declarative ones.

- [ ] Convert declarative headings to question format:

| ❌ Declarative | ✅ Question-style |
|---------------|------------------|
| `Features` | `What Makes This Tool Different?` |
| `Getting Started` | `How to Get Started with [Product]?` |
| `Pricing` | `How Much Does [Product] Cost?` |
| `Related Tools` | `What Other Tools Are Available?` |

### 2.4 HowTo Schema + Visible Step Content

- [ ] Pages with tutorials or workflows should have `HowTo` JSON-LD (3-5 steps)
- [ ] Steps must also be **visually present** on the page (not just in JSON-LD)

```json
{
  "@context": "https://schema.org",
  "@type": "HowTo",
  "name": "How to [Action]",
  "step": [
    {"@type": "HowToStep", "position": 1, "name": "Step 1 Title", "text": "Step 1 description..."},
    {"@type": "HowToStep", "position": 2, "name": "Step 2 Title", "text": "Step 2 description..."},
    {"@type": "HowToStep", "position": 3, "name": "Step 3 Title", "text": "Step 3 description..."}
  ]
}
```

---

## 3. Page Content Depth

### 3.1 Word Count Targets

> 2026 GEO research consistently shows: AI engines prefer reference-grade content. Content with statistics and cited sources gets 30-40% more AI citations.

- [ ] Target per key page (excluding navigation and footer):

| Language | Metric | Target | Notes |
|----------|--------|--------|-------|
| English | Words (space-separated) | **800-1200** | Standard GEO recommendation |
| Chinese | Characters (CJK codepoints) | **1600-2400** | ~2 Chinese chars ≈ 1 English word in information density |

> **Why separate thresholds?** Chinese text has no spaces between words. Counting by `split()` on Chinese content returns near-zero, severely underestimating actual content depth. Always use **CJK character count** for Chinese pages.

- [ ] Detection method: see [Detection Commands](#7-detection-commands) — the script auto-detects language and reports both metrics

### 3.2 Recommended Content Structure

| Content Block | Description | Target Words |
|--------------|-------------|:----------:|
| **Intro Summary** | Top of page (between H1 and first H2) — directly answers "what is this, what does it do" | 100 |
| **How to Use** | 3-5 step guide, paired with HowTo Schema | 200 |
| **Technical Details** | Technical principles (algorithms, architecture, privacy model) | 200 |
| **Use Cases / Scenarios** | Usage scenarios: "ideal for designers", "perfect for developers" | 150 |
| **Comparison** | Competitive comparison (features, pros/cons) | 150 |
| **FAQ** | 3-5 targeted questions matching user search intent | 200 |

### 3.3 Key Principles

- [ ] **First 200 words must directly answer the core question** (intro-summary-first principle)
- [ ] One deep guide > ten shallow pages
- [ ] Include specific data and cited sources (e.g. "reduces size by 70%", "based on WebAssembly")
- [ ] Avoid pure feature lists — add explanatory content

---

## 4. Technical SEO Basics

### 4.1 Meta Tags

- [ ] **title**: 50-60 characters, include core keyword, avoid truncation
- [ ] **description**: 150-160 characters, include call to action
- [ ] **canonical**: Every page has a unique canonical URL
- [ ] **hreflang**: Multi-language pages correctly configured (`<link rel="alternate" hreflang="zh" href="...">`)

### 4.2 Open Graph & Social

- [ ] **og:title**: Unique per page
- [ ] **og:description**: Unique per page
- [ ] **og:image**: Each page uses a distinct social share image (with page title and brief description)
- [ ] **Twitter Card**: Correctly set `twitter:card`, `twitter:title`, etc.

### 4.3 Sitemap

- [ ] sitemap.xml includes all indexable pages (separate by language if multi-language)
- [ ] Each URL has a `<lastmod>` date
- [ ] Sitemap declared in robots.txt

### 4.4 URL Structure

- [ ] Consistent trailing slash strategy (recommend always with `/`)
- [ ] URLs use lowercase English with hyphens
- [ ] Avoid deep nesting (recommend no more than 3 levels)

### 4.5 Security & Trust

- [ ] **HSTS Header**: `Strict-Transport-Security: max-age=31536000; includeSubDomains`
- [ ] **HTTPS**: Enforced site-wide
- [ ] These factors affect E-E-A-T trust scores

### 4.6 Content Freshness

- [ ] Page displays a **visible** `Last Updated` date (not just in JSON-LD)
- [ ] `dateModified` in JSON-LD matches the actual update date

### 4.7 IndexNow

- [ ] Integrate IndexNow protocol (supported by Bing/Yandex)
- [ ] Automatically notify search engines on content updates for faster indexing

---

## 5. Page Performance

### 5.1 Loading Speed

> 2026 GEO research: Pages with FCP < 0.4s average 6.7 AI citations; pages > 1.13s average only 2.1. Fast pages are cited 3x more.

- [ ] **FCP (First Contentful Paint)**: Target < 0.4s
- [ ] **TLS optimization**: Enable OCSP Stapling, stable TLS Session Ticket reuse
- [ ] **Compression**: Enable Brotli compression (better than gzip)
- [ ] **CDN**: Use CDN for static assets (e.g. CloudFlare, Fastly)
- [ ] **HTML size**: Keep under 100KB, consider code splitting

### 5.2 Detection Method

```bash
# Page load time
curl -sL -o /dev/null -w "Total: %{time_total}s\nTLS: %{time_appconnect}s\nFCP proxy: %{time_starttransfer}s\n" "$SITE_URL/"

# Compression check
curl -sI -H "Accept-Encoding: gzip, br" "$SITE_URL/" | grep -i 'content-encoding'
```

---

## 6. Off-page Authority Building

> Research shows brands are cited by AI engines via third-party domains 6.5x more often than via their own domain.

### 6.1 Platform Presence

- [ ] **Product Hunt**: Publish a product page
- [ ] **AlternativeTo**: Create an entry benchmarked against competitors
- [ ] **Reddit**: Share in relevant subreddits (r/webdev, r/SideProject, etc.)
- [ ] **GitHub**: Open-source part of the code, link back in README
- [ ] **Wikidata**: Create a product entity
- [ ] **Industry directories**: Submit to relevant niche directories

### 6.2 Comparison Content Pages

> AI search queries average 23 words — much longer than traditional search — and often contain comparison intent.

- [ ] Create `vs` comparison pages (e.g. `/product-vs-competitor/`)
- [ ] Include feature comparison tables and pros/cons analysis

### 6.3 Multi-modal Optimization

- [ ] Add tutorial videos to key pages (YouTube embed + text transcript)
- [ ] All images have descriptive alt text
- [ ] Add `VideoObject` Schema when videos are present

---

## 7. Detection Commands

All audit scripts live in `scripts/` and accept the same core arguments. They use `$SITE_URL` from `.env` or `--url`.

> All example commands below assume the venv has been activated: `source "$DATA_DIR/venv/bin/activate"`

### 7.1 SEO Audit (`seo_audit.py`)

Covers: JSON-LD coverage, meta tags, headings, og:image, canonical, hreflang, SSR empty-shell detection, sitemap, HowTo/FAQ schema.

```bash
# Homepage only
python scripts/seo_audit.py --url "$SITE_URL"

# Specific pages
python scripts/seo_audit.py --url "$SITE_URL" --pages "/,/about,/pricing"

# Full-site audit via sitemap (JSON-LD + meta per page)
python scripts/seo_audit.py --url "$SITE_URL" --sitemap

# Save results to file
python scripts/seo_audit.py --url "$SITE_URL" --sitemap -o data/seo_audit.json
```

### 7.2 GEO Audit (`geo_audit.py`)

Covers: llms.txt / llms-full.txt, robots.txt AI crawler rules, content depth (auto-detects CJK vs English, see [3.1](#31-word-count-targets)), intro summary detection, FAQ/HowTo sections, question-style headings.

```bash
# Homepage only
python scripts/geo_audit.py --url "$SITE_URL"

# Specific pages
python scripts/geo_audit.py --url "$SITE_URL" --pages "/,/blog/my-post"

# Full-site audit via sitemap (max 50 pages)
python scripts/geo_audit.py --url "$SITE_URL" --sitemap

# Save results to file
python scripts/geo_audit.py --url "$SITE_URL" --sitemap -o data/geo_audit.json
```

### 7.3 Performance & Security Audit (`perf_audit.py`)

Covers: load time (TTFB, total), compression (Brotli/gzip), HTML size, HSTS, HTTPS/TLS version, cache headers, CSP, X-Frame-Options, CDN detection.

```bash
# Homepage only
python scripts/perf_audit.py --url "$SITE_URL"

# Specific pages
python scripts/perf_audit.py --url "$SITE_URL" --pages "/,/about"

# Full-site via sitemap (max 20 pages)
python scripts/perf_audit.py --url "$SITE_URL" --sitemap

# Save results to file
python scripts/perf_audit.py --url "$SITE_URL" --sitemap -o data/perf_audit.json
```

### 7.4 Run All Audits

```bash
python scripts/seo_audit.py  --url "$SITE_URL" --sitemap -o data/seo_audit.json
python scripts/geo_audit.py  --url "$SITE_URL" --sitemap -o data/geo_audit.json
python scripts/perf_audit.py --url "$SITE_URL" --sitemap -o data/perf_audit.json
```

---

## 8. Priority Overview

When generating the SEO/GEO audit section of the improvement report, use this priority matrix to classify findings:

```
Priority    Item                                    Impact    Effort   Est. Time
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
P0 🔴   JSON-LD site-wide coverage + SSR output      ★★★★★   ★★★     1-2 days
P0 🔴   Add llms.txt + llms-full.txt                 ★★★★★   ★☆☆     1 hour
P0 🔴   Increase page content depth                  ★★★★★   ★★★     2-3 days
P0 🔴   Convert H2/H3 to question format             ★★★★☆   ★☆☆     2 hours
P1 🟡   Add HowTo Schema                             ★★★★☆   ★★☆     Half day
P1 🟡   robots.txt AI crawler rules                   ★★★☆☆   ★☆☆     10 min
P1 🟡   Page load speed optimization                  ★★★★☆   ★★★     1-2 days
P1 🟡   og:image uniqueness per page                  ★★★☆☆   ★★☆     Half day
P2 🟢   Visible Last Updated date                     ★★★☆☆   ★☆☆     1 hour
P2 🟢   Title length optimization (50-60 chars)       ★★☆☆☆   ★☆☆     5 min
P2 🟢   HSTS Header                                   ★★☆☆☆   ★☆☆     5 min
P2 🟢   robots.txt reference to llms.txt              ★★☆☆☆   ★☆☆     1 min
P3 🔵   Off-page authority building                    ★★★★★   ★★★     Ongoing
P3 🔵   IndexNow integration                          ★★★☆☆   ★★☆     Half day
P3 🔵   Comparison content pages                       ★★★★☆   ★★★     Ongoing
P3 🔵   Multi-modal optimization (video + transcript)  ★★★☆☆   ★★★     Ongoing
```

---

## References

- [llms.txt Standard Proposal](https://llmstxt.org/) — Jeremy Howard, 2024
- [Google Structured Data Docs](https://developers.google.com/search/docs/appearance/structured-data)
- [Schema.org](https://schema.org/) — Structured data vocabulary
- [IndexNow Protocol](https://www.indexnow.org/)
- GEO Research: *"GEO: Generative Engine Optimization"* (2024)


path: references/website-reconnaissance-reference.md
content: # Website Reconnaissance Reference

Detailed commands and classification tables for Phase 0: Website Reconnaissance & Goal Definition.

## Browser Commands for Site Exploration

### Take screenshots

```bash
agent-browser open "$SITE_URL" && agent-browser wait --load networkidle
agent-browser screenshot --full homepage_overview.png
```

Save screenshots to `$DATA_DIR/tmp/`.

### Extract page metadata

```bash
agent-browser eval --stdin <<'EVALEOF'
JSON.stringify({
  title: document.title,
  meta_desc: document.querySelector('meta[name="description"]')?.content,
  h1: Array.from(document.querySelectorAll('h1')).map(h => h.textContent.trim()),
  h2s: Array.from(document.querySelectorAll('h2')).slice(0, 10).map(h => h.textContent.trim()),
  nav_links: Array.from(document.querySelectorAll('nav a, header a')).slice(0, 20).map(a => ({text: a.textContent.trim(), href: a.href})),
  cta_buttons: Array.from(document.querySelectorAll('a[class*="btn"], button[class*="btn"], a[class*="cta"], button[class*="cta"], [role="button"]')).slice(0, 10).map(el => ({text: el.textContent.trim(), href: el.href || ''})),
  footer_links: Array.from(document.querySelectorAll('footer a')).slice(0, 15).map(a => ({text: a.textContent.trim(), href: a.href})),
  forms: Array.from(document.querySelectorAll('form')).map(f => ({action: f.action, id: f.id, inputs: Array.from(f.querySelectorAll('input,select,textarea')).map(i => i.name || i.type)})),
  has_pricing: !!document.body.innerText.match(/pric(e|ing)|plan|subscribe|buy|purchase|付费|价格|套餐/i),
  has_login: !!document.querySelector('a[href*="login"], a[href*="signin"], a[href*="signup"], a[href*="register"]'),
  has_ecommerce: !!document.querySelector('[class*="cart"], [class*="product"], [class*="shop"]'),
  has_blog: !!document.querySelector('a[href*="blog"], a[href*="article"], a[href*="news"]'),
})
EVALEOF
```

### Extract front-end metadata (when no source code available)

```bash
agent-browser eval --stdin <<'EVALEOF'
JSON.stringify({
  title: document.title,
  meta_desc: document.querySelector('meta[name="description"]')?.content,
  h1: Array.from(document.querySelectorAll('h1')).map(h => h.textContent),
  has_jsonld: document.querySelectorAll('script[type="application/ld+json"]').length,
  images_no_alt: document.querySelectorAll('img:not([alt])').length,
  viewport: document.querySelector('meta[name="viewport"]')?.content,
  canonical: document.querySelector('link[rel="canonical"]')?.href,
})
EVALEOF
```

## Website Type Classification

| Website Type | Key Signals | Primary Goal | Core Metrics |
|---|---|---|---|
| **Content/Blog** | Blog section, articles, tutorials | Reader engagement & return visits | Page views, time on page, pages/session, return visitor rate |
| **SaaS/Tool** | Login/signup, pricing page, app features | User registration → activation → retention | Signup rate, trial-to-paid, feature adoption |
| **E-commerce** | Product pages, cart, checkout | Purchase conversion | Add-to-cart rate, cart abandonment, purchase conversion, AOV |
| **Lead Generation** | Contact forms, demo requests, whitepapers | Form submissions / demo bookings | Form completion rate, lead quality, cost per lead |
| **Portfolio/Showcase** | Project galleries, about page, contact | Contact or inquiry | Contact form submissions, inquiry rate |
| **Documentation/Docs** | API docs, guides, reference pages | Help users find answers | Search usage, page depth, time on page, exit rate from docs |
| **Community/Forum** | User posts, comments, threads | User engagement & content creation | Posts per user, comment rate, return rate |
| **Landing Page** | Single page, strong CTA, no navigation depth | Single conversion action | CTA click rate, conversion rate, bounce rate |

## Analysis Dimension Ranking

Not all analysis dimensions are equally important for every website type. Based on the goal profile, select and **rank** the applicable dimensions:

| Analysis Dimension | When It Matters Most |
|---|---|
| **Acquisition / SEO** | Sites that depend on organic search traffic (content, SaaS, e-commerce) |
| **Conversion Funnel** | Sites with clear multi-step user journeys (SaaS signup, e-commerce purchase) |
| **Content Engagement** | Content-heavy sites (blogs, docs, tutorials) |
| **User Experience (UX)** | All sites, but especially mobile-first sites or sites with high bounce rates |
| **Performance** | All sites, but critical for e-commerce and landing pages where speed = money |
| **Retention / Return Visits** | SaaS, community, and content sites where repeat visits matter |
| **Technical SEO / GEO** | All sites that need search visibility |

Output a ranked list of dimensions for this specific site, e.g.:
```
Priority analysis dimensions for [site]:
1. Conversion Funnel (SaaS → signup → trial → paid is the core business flow)
2. Acquisition / SEO (organic search is the main traffic channel)
3. Content Engagement (blog is used for top-of-funnel acquisition)
4. UX (mobile bounce rate appears high from initial visit)
5. Performance (site loads well, lower priority)
```

## Goal Profile Presentation Template

> Based on my visit to your site, here's what I understand:
>
> **Website Type**: [e.g., SaaS Tool]
> **Primary Goal**: [e.g., Get users to sign up for free trial and convert to paid]
> **Secondary Goals**: [e.g., Build brand awareness through blog content, provide documentation for retention]
>
> **Intended User Journey** (the path you want users to follow):
> ```
> [e.g., Search/Ad → Landing Page → Explore Features → Sign Up → Onboarding → First Value → Upgrade]
> ```
>
> **Key Conversion Points** I'll focus on:
> 1. [e.g., Landing page → Sign up page (acquisition)]
> 2. [e.g., Sign up → Complete onboarding (activation)]
> 3. [e.g., Free trial → Paid plan (monetization)]
>
> Does this match your understanding? Please correct or add anything I missed.
> Also, if you have specific KPIs or targets (e.g., "we want to increase signup rate from 3% to 5%"), please share them.

## Information Request Template

> To give you the most actionable analysis, it would be very helpful if you could tell me:
>
> 1. **Your main traffic sources** — Is most traffic from organic search, paid ads, social media, or direct?
> 2. **Key GA4 events you've set up** — Do you track specific events like `signup`, `purchase`, `add_to_cart`, `form_submit`? (This determines whether I can do funnel analysis)
> 3. **Any known pain points** — Are there specific pages or flows where you suspect users are dropping off?
> 4. **Business context** — Any recent changes (redesign, new content, campaign launches) that might affect the data?
> 5. **Target audience** — Who are your primary users? (This helps interpret geographic and device data)
>
> Don't worry if you can't answer all of these — I'll work with what's available and flag areas where more data would help.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)