Skill: hasdata/hasdata-api (version 1.0.0)

--- name: hasdata description: Fetch real-time web data via the hasdata CLI. Use when the user wants search results, news, fact-checks, product or seller info, current prices, reviews, real-estate listings or sold comps, vacation rentals, local-business contact details, job postings, salary research, search trends, images, flights, social profiles, or to scrape any URL (HTML / markdown / AI-extracted JSON). Also use when the user asks to summarize a web page, ground a prompt with current information, verify a URL is live or render a JavaScript-heavy page, monitor a price over time, find a phone number or address for a business, build a competitor map, identify recent sold comparables, gather employer reviews, fan out a list of items to per-item details, or check what's being said online about a topic right now. Backed by Google, Bing, Amazon, Shopify, Zillow, Redfin, Airbnb, Yelp, YellowPages, Indeed, Glassdoor, Instagram, Google Maps / Trends / News / Images / Flights / Events APIs. homepage: https://github.com/HasData/hasdata-cli metadata: clawdbot: emoji: "🔎" primaryEnv: HASDATA_API_KEY requires: bins: [hasdata] --- # hasdata Use the `hasdata` CLI for real-time web data. One subcommand per API — flags, enums, defaults are derived from the live schema at `api.hasdata.com/apis`. ## Prerequisites - `command -v hasdata` — if missing, install with `curl -sSL https://raw.githubusercontent.com/HasData/hasdata-cli/main/install.sh | sh`. - One-time setup: the user runs `hasdata configure`, pastes their API key, and it's saved to `~/.hasdata/config.yaml` (mode 0600). Every future call picks it up automatically. - If a call fails with `no API key configured`, the user hasn't run `hasdata configure` yet — tell them to. **Never invent a key.** ## Quick start ```bash hasdata <api> --flag value [--flag value ...] --raw | jq . ``` Always pass `--raw` when piping to `jq` (skips pretty-print and TTY detection). Use `--pretty` only for human-readable terminal output. ## Picking the right subcommand | User intent | Subcommand | | --- | --- | | Web search ("what does Google say about…") | `google-serp` (full features) or `google-serp-light` (cheap, single page) | | Latest news | `google-news` | | AI Mode SERP | `google-ai-mode` | | Shopping / product prices | `google-shopping` (broad), `amazon-search` / `amazon-product` (Amazon), `shopify-products` (Shopify) | | Immersive product page | `google-immersive-product` | | Maps / places / reviews | `google-maps`, `google-maps-place`, `google-maps-reviews`, `google-maps-photos` | | Yelp / YellowPages local data | `yelp-search`, `yelp-place`, `yellowpages-search`, `yellowpages-place` | | Real-estate listings | `zillow-listing`, `redfin-listing`, `airbnb-listing` | | Real-estate single property deep dive | `zillow-property`, `redfin-property`, `airbnb-property` | | Jobs | `indeed-listing`, `indeed-job`, `glassdoor-listing`, `glassdoor-job` | | Bing search | `bing-serp` | | Trends | `google-trends` | | Images | `google-images` | | Flights | `google-flights` | | Short videos | `google-short-videos` | | Events | `google-events` | | Instagram profile | `instagram-profile` | | Amazon seller | `amazon-seller`, `amazon-seller-products` | | **Scrape a specific URL** | `web-scraping` — supports JS rendering, proxies, markdown output, AI extraction, screenshots | For exact flags of a subcommand, run `hasdata <api> --help` or read the matching file in `references/`. ## Non-obvious triggers (when to reach for hasdata even if the user doesn't say "scrape") The user often won't ask for a SERP API or a scraper directly. Map these intents to the skill: - **"Is this still true?" / "What's the latest on X?" / "Has Y happened yet?"** — LLM training data is stale. Run `google-serp` or `google-news` to ground the answer. - **"Summarize this article" / "TL;DR this URL"** — Use `web-scraping --output-format markdown` and feed the markdown into the summary prompt. Beats copy-paste because it strips ads, nav, scripts. - **"Verify this link" / "Is this site real?"** — `web-scraping --url X --no-block-resources` returns status + screenshot. Or `google-serp --q "site:example.com"`. - **"What does X say about itself?"** — Pull the company's own homepage with `web-scraping --output-format markdown`, then summarize. - **"Find me alternatives to X"** — `google-serp --q "X alternatives"` or `google-shopping --q "X competitors"`. - **"What's the going rate for X?"** — `google-shopping` (broad) or `amazon-search` (Amazon-specific) with `jq` to extract the price distribution. - **"Phone number / address for X"** — `google-maps-place` or `yelp-place`. Don't guess from training data. - **"Are people happy with X service?" / "Is X reputable?"** — `google-maps-reviews --place-id ... --sort lowest` for negative samples; `glassdoor-job` for employer rep. - **"What's the salary range for Y role?"** — `indeed-listing` filtered by role + location, then `jq` over `.jobs[].salary`. - **"Find me homes/apartments matching X criteria"** — `zillow-listing` / `redfin-listing` / `airbnb-listing` with the corresponding filters. - **"Recent sold comps near X"** — `zillow-listing --type sold --keyword "X" --days-on-zillow 12m`. - **"Track this product's price"** — Loop `amazon-product --asin X` on a schedule; persist `.price` to a file. - **"What's trending around X?"** — `google-trends --q "X"` for relative interest; `google-news --q "X"` for headlines. - **"Find businesses near me that do X"** — `google-maps --q "X" --ll "@LAT,LNG,12z"` then fan out `google-maps-place` for contacts. - **"How does this look in country Y?"** — `--gl Y` on SERP commands, `--proxy-country Y` on `web-scraping`. Useful for geo-targeted SEO checks, geo-blocked content. - **"Pull structured data from this page"** — `web-scraping --ai-extract-rules-json '{"price": {"type": "number"}, ...}'`. Works on arbitrary pages without writing CSS selectors. - **"List of items → per-item details"** — Pattern: search command produces IDs/URLs, pipe through `xargs` into the matching `*-property` / `*-product` / `*-place` deep-dive command. - **"Find this person's role / employer / LinkedIn / followers"** — `google-serp --q '"Person Name" linkedin'` first. The organic-result title is typically `Name — Role at Company | LinkedIn` and the snippet carries location, headline, connection count. SERP often answers the whole question without ever opening the profile page. - **"What is company X doing? Where's their HQ? Who works there?"** — `google-serp --q "$COMPANY"` returns a `.knowledge_graph` block with founder, HQ, founded year, parent, employee range — pre-extracted. `google-news --q "$COMPANY"` for recent activity. Specific facts via targeted SERP: `--q '"$COMPANY" headquarters'`, `--q '"$COMPANY" funding'`, `--q 'site:linkedin.com/company "$COMPANY"'`. - **"Find emails for company X" / "personal email for person Y"** — start with SERP: `--q '"@example.com"'` or `--q '"[REDACTED_SECRET]"'` often surfaces actual emails indexed by Google. Pattern-guess + SERP-verify for individuals. Disclose unverified guesses to the user. - **"Enrich this CSV of leads"** — per row: `google-serp` for LinkedIn, role, employer; another SERP to verify email or pattern. Stay in SERP unless a specific field is missing. - **Reverse-lookup (email / phone / domain → identity)** — `google-serp` with the literal value in quotes (`--q '"[REDACTED_SECRET]"'`, `--q '"+1 555 123 4567"'`, `--q '"acme corp" site:example.com'`) almost always surfaces the matching person or business. **SERP-first principle**: for any data-enrichment intent (people, companies, emails, products, places), reach for `google-serp` / `google-news` / `google-shopping` / `google-maps` first. They return Google's already-extracted structured fields (`.knowledge_graph`, `.organic_results[].snippet`, `.local_results[]`, etc.) and bypass anti-bot. Only escalate to `web-scraping` when SERP doesn't surface the specific field you need — it's the last resort, not the default. See `references/enrichment.md`. If a user request matches one of the above and you don't invoke hasdata, you're probably hallucinating a stale answer. ## Universal flag patterns - **Kebab-case** flag names. The CLI maps them back to the original camelCase before sending to the API. - **Booleans defaulting to `true`** have a paired negation: `--no-block-ads`, `--no-screenshot`, `--no-js-rendering`, `--no-extract-emails`, `--no-block-resources`. Setting both `--block-ads` and `--no-block-ads` errors. - **Anything ending in `-json`** accepts: - inline JSON: `--extract-rules-json '{"title":"h1"}'` - file: `--extract-rules-json @rules.json` - stdin: `cat rules.json | hasdata web-scraping ... --extract-rules-json -` - **Repeatable key=value** flags split on the first `=` (so values containing `=` survive): `--headers User-Agent=foo --headers Cookie=session=abc`. Pair with `--headers-json` for a JSON base; kv items override per key. - **List flags** accept either repeats or comma-joined: `--lr lang_en --lr lang_fr` or `--lr lang_en,lang_fr`. Serialized as `key[]=value` for GET endpoints. - **Enum flags** validate client-side. If you guess wrong, the error lists the allowed values — read the message and retry. ## Global flags (apply to every subcommand) | Flag | Effect | | --- | --- | | `--raw` | Write response bytes as-is (use this when piping to `jq`) | | `--pretty` | Pretty-print JSON (default when stdout is a TTY) | | `-o, --output FILE` | Write response to file instead of stdout (works for binary like screenshots) | | `--verbose` | Log outgoing URL and `X-RateLimit-*` headers to stderr | | `--api-key KEY` | Override env var (rarely needed) | | `--timeout DURATION` | Per-request timeout (default 2m) | | `--retries N` | Max retries on 429/5xx (default 2) | ## Output contract Responses are JSON. Pipe through `jq` for extraction: ```bash hasdata google-serp --q "espresso machine" --num 10 --raw \ | jq -c '.organic_results[] | {title, link, snippet}' ``` For real-estate / e-commerce results, the array shape is API-specific — read a single response with `--pretty` first to learn the schema, then write the `jq` filter. ## Exit codes (script-safe) | Code | Meaning | | --- | --- | | 0 | success | | 1 | user / CLI-input error (missing required flag, bad enum value, missing API key) | | 2 | network error | | 3 | API returned 4xx (auth, quota, validation) | | 4 | API returned 5xx | ## References - [`references/enrichment.md`](references/enrichment.md) — **person and company enrichment** (LinkedIn lookup, emails, HQ/funding/news, CSV-row enrichment, reverse-lookup) — the highest-leverage cross-API workflows - [`references/search.md`](references/search.md) — Google SERP / Bing / News / Trends flag catalog - [`references/web-scraping.md`](references/web-scraping.md) — `web-scraping` flags, JS scenarios, AI extraction - [`references/real-estate.md`](references/real-estate.md) — Zillow / Redfin / Airbnb filters and bracketed params - [`references/ecommerce.md`](references/ecommerce.md) — Amazon / Shopify - [`references/local-business.md`](references/local-business.md) — Maps / Yelp / YellowPages - [`references/jobs.md`](references/jobs.md) — Indeed / Glassdoor - [`references/all-commands.md`](references/all-commands.md) — full subcommand index with credit costs

path: README.md
content: # hasdata — openclaw skill

Real-time web data for openclaw agents via the `hasdata` CLI. Covers Google SERP, Maps, Amazon, Zillow, Redfin, Airbnb, Yelp, Indeed, Bing, arbitrary URL scraping, and more.

## Install

```sh
clawhub install hasdata
```

The skill itself is just instructions — it shells out to the `hasdata` binary, which the agent installs on first use:

```sh
curl -sSL https://raw.githubusercontent.com/HasData/hasdata-cli/main/install.sh | sh
```

## Configure

One-time setup:

```sh
hasdata configure
```

Prompts for the API key and saves it to `~/.hasdata/config.yaml`. Every future call picks it up automatically — no env var to re-export.

Get a key from <https://hasdata.com>.

## What's in here

```
openclaw-skill/
├── SKILL.md                  # entry point — frontmatter + decision matrix + flag patterns
├── README.md                 # this file
└── references/
    ├── enrichment.md         # person + company enrichment (LinkedIn, emails, HQ, news)
    ├── search.md             # google-serp, bing-serp, news, trends, etc.
    ├── web-scraping.md       # JS scenarios, AI extraction, markdown output
    ├── real-estate.md        # zillow / redfin / airbnb (incl. bracketed filters)
    ├── ecommerce.md          # amazon / shopify
    ├── local-business.md     # maps / yelp / yellowpages
    ├── jobs.md               # indeed / glassdoor
    └── all-commands.md       # full subcommand index with credit costs
```

`SKILL.md` is loaded into the agent's context when the skill triggers; the `references/` files are progressive-disclosure — pulled in only when the agent needs deep flag info for a specific vertical.

## Source CLI

Built on top of <https://github.com/HasData/hasdata-cli>. The skill's content is hand-authored against the CLI's `--help` output; when the CLI ships new APIs, this skill should be regenerated.


path: package.sh
content: #!/usr/bin/env bash
# package.sh — pack this openclaw skill into a zip ready for distribution or
# manual upload. README.md is human-facing only and is excluded from the
# bundle. The script itself, OS junk, prior zips, and VCS metadata are also
# excluded.
#
# Usage:
#   ./package.sh                  # writes hasdata-openclaw-skill.zip alongside SKILL.md
#   ./package.sh /tmp/out.zip     # custom output path

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
OUT="${1:-$SCRIPT_DIR/hasdata-openclaw-skill.zip}"

# Resolve OUT to an absolute path so the cd below doesn't break a relative arg.
case "$OUT" in
  /*) ;;
  *) OUT="$(pwd)/$OUT" ;;
esac

cd "$SCRIPT_DIR"
rm -f "$OUT"

zip -r "$OUT" . \
  -x "README.md" \
  -x "$SCRIPT_NAME" \
  -x ".DS_Store" -x "*/.DS_Store" \
  -x ".git/*" \
  -x "*.zip" \
  > /dev/null

echo "wrote: $OUT"
echo "size:  $(du -h "$OUT" | awk '{print $1}')"
echo
echo "contents:"
unzip -l "$OUT" | sed '1,2d;$d'


path: references/search.md
content: # Search references

Subcommands: `google-serp`, `google-serp-light`, `google-ai-mode`, `google-news`, `google-shopping`, `bing-serp`, `google-trends`, `google-images`, `google-events`, `google-short-videos`, `google-immersive-product`, `google-flights`.

Run `hasdata <api> --help` for the live, authoritative flag set. Below are the commonly-used flags and example invocations.

---

## google-serp (10 credits)

```bash
hasdata google-serp --q "QUERY" [--gl COUNTRY] [--hl LANG] [--num 10] [--start 0] --raw | jq .
```

Common flags:
- `--q TEXT` (required) — search query
- `--gl us|gb|ca|de|fr|...` — country code; affects results
- `--hl en|es|fr|de|...` — UI language
- `--num 10..100` — results per page
- `--start 0|10|20...` — pagination offset (multiples of 10)
- `--location "Austin,Texas,United States"` — Google canonical location
- `--device-type desktop|mobile|tablet`
- `--tbm isch|vid|nws|shop|lcl` — search type
- `--safe active|off`
- `--lr lang_en --lr lang_fr` — restrict to language(s)
- `--domain google.com|google.co.uk|...`
- `--tbs cdr:1,cd_min:10/17/2018,cd_max:3/8/2021` — advanced search filters

Useful response fields (via `jq`):
- `.organic_results[] | {title, link, snippet}` — main results
- `.ai_overview` — AI Overview block (when present)
- `.answer_box`, `.knowledge_graph`, `.related_searches`, `.people_also_ask`
- `.local_results`, `.shopping_results`, `.news_results`

Example — top-10 organic for prompt grounding:
```bash
hasdata google-serp --q "$Q" --num 10 --raw \
  | jq -r '.organic_results[] | "- \(.title): \(.snippet)"'
```

## google-serp-light (5 credits)

Same flags as `google-serp` but cheaper and returns a single page. Use when the user wants quick results and doesn't need PAA/AI Overview/local sections.

## google-ai-mode (5 credits)

Returns Google's AI Mode answer for a query. Same `--q` / `--gl` / `--hl` semantics.

## google-news (10 credits)

```bash
hasdata google-news --q "QUERY" [--gl us] [--hl en] --raw | jq '.news_results[]'
```

Per-article fields: `title`, `link`, `source.name`, `date`, `snippet`, `thumbnail`.

## google-shopping (10 credits)

```bash
hasdata google-shopping --q "PRODUCT" [--gl us] --raw | jq '.shopping_results[]'
```

Per-result: `title`, `link`, `price`, `extracted_price`, `source`, `rating`, `reviews`, `delivery`.

## bing-serp (10 credits)

```bash
hasdata bing-serp --q "QUERY" [--cc us] [--setlang en] --raw | jq '.organic_results[]'
```

Use when the user explicitly asks for Bing or wants a non-Google second opinion.

## google-trends (5 credits)

```bash
hasdata google-trends --q "TERM" [--geo US] [--cat 0] [--time "today 12-m"] --raw | jq .
```

Multiple terms for comparison: `--q "term1,term2,term3"` (comma-separated, max 5).

## google-images (5 credits)

```bash
hasdata google-images --q "QUERY" --raw | jq '.images_results[] | {title, original, source}'
```

## google-events (5 credits)

```bash
hasdata google-events --q "concerts in austin" [--gl us] --raw | jq '.events_results[]'
```

## google-flights (15 credits)

```bash
hasdata google-flights \
  --departure-id "JFK" --arrival-id "LAX" \
  --outbound-date 2026-06-15 --return-date 2026-06-22 \
  --currency USD --raw | jq .
```

Round-trip vs. one-way controlled by presence/absence of `--return-date`.

## google-short-videos (10 credits) / google-immersive-product (5 credits)

Less common — run `--help` to see flags when needed.

---

## Non-obvious use cases

- **Fact-check a claim before answering** — instead of relying on training data, run `google-serp --q "EXACT CLAIM"` and check whether top results corroborate or contradict.
- **Resolve "what is the URL for X?"** — agents often hallucinate URLs. `google-serp --q "X official site" --num 3` and pick the result with the matching domain.
- **Source-find a quote** — `google-serp --q "\"the exact quoted text here\""` (escape the inner quotes). Returns the page that originated it.
- **Compare same query across regions** — same `--q`, different `--gl us|gb|de|fr` to see how SERP differs by geo. Useful for international SEO and "what do users in country Y see".
- **Time-bound search** — `--tbs qdr:d` (past day), `qdr:w` (week), `qdr:m` (month), `qdr:y` (year). Combine with `google-news` for fresh-only news. Or `--tbs cdr:1,cd_min:M/D/YYYY,cd_max:M/D/YYYY` for an explicit window.
- **Site-restricted search** — `--q "site:example.com TOPIC"` to search within one domain (better than scraping the site's own search box).
- **"What's been written about X recently?"** — `google-news --q "X" --gl us` then `jq` over `.news_results[] | select(.date | test("hours? ago|day ago"))`.
- **Discover competitors** — `google-serp --q "best alternatives to PRODUCT"` followed by extracting brand names from the top results' titles.
- **Find documentation links** — `google-serp --q "LIBRARY official docs"` instead of guessing a URL pattern.
- **Translate search intent** — `--hl de --gl de --q "the English query"` shows German-language ranking; useful for multi-locale SEO checks.
- **Trends over time** — `google-trends --q "term1,term2"` (comma-separated, ≤5 terms) for relative interest curves. Better than guessing "is X popular now" from training data.
- **Image-driven research** — `google-images --q "X"` to grab source URLs; pipe top results into `web-scraping` to read context.
- **Flights without a travel API** — `google-flights` with `--departure-id`, `--arrival-id`, dates, and `--currency` for ad-hoc fare checks.
- **Map-distance trick** — for "things near X", use `google-maps --ll "@LAT,LNG,Zz"` not a SERP query — much higher signal for proximity intent.
- **Fact-check pricing** — when training data has stale prices for SaaS or subscription services, search and read the pricing page rather than answering from memory.

## Picking the right SERP variant

- Use `google-serp-light` when only top organic results are needed (skips PAA / AI overview / local sections).
- Use `google-serp` when you need the full SERP feature set (PAA, AI overview, knowledge graph, local pack).
- Cache results client-side when running the same query repeatedly — the CLI does not cache.


path: references/jobs.md
content: # Jobs reference

Subcommands: `indeed-listing`, `indeed-job`, `glassdoor-listing`, `glassdoor-job`. Indeed: 5 credits. Glassdoor: 10 credits.

`*-listing` searches by query/location; `*-job` deep-dives a specific posting by URL or ID.

---

## indeed-listing

```bash
hasdata indeed-listing \
  --query "backend engineer" --location "Remote" \
  [--days-since-posted 7] [--job-type fulltime|parttime|contract|temporary|internship] \
  [--remote] [--page 1] \
  --raw | jq '.jobs[] | {title, company, location, posted, salary, url}'
```

Other flags worth knowing:
- `--salary-min N` / `--salary-max N`
- `--experience-level entry|mid|senior`
- `--sort relevance|date`
- `--country us|gb|ca|de|fr|...`

## indeed-job

```bash
hasdata indeed-job --url "https://www.indeed.com/viewjob?jk=..." --raw | jq .
```

Or `--job-key JK`. Returns full description, company info, benefits, hiring insights, similar jobs.

## glassdoor-listing

```bash
hasdata glassdoor-listing --keyword "data scientist" --location "Boston" --raw \
  | jq '.jobs[] | {title, employer, salary_estimate, rating}'
```

Glassdoor includes employer ratings and salary estimates per result. Use this when the user cares about employer reputation alongside the role.

## glassdoor-job

```bash
hasdata glassdoor-job --url "https://www.glassdoor.com/Job/jobs.htm?...JV=..." --raw | jq .
```

Returns full posting plus employer rating breakdown, recent reviews, salary estimate range, interview difficulty.

---

## Non-obvious use cases

- **Salary-negotiation research** — `indeed-listing --query ROLE --location CITY` then `jq '[.jobs[].salary | select(.)] | sort_by(.min)'` to build a defensible range before a comp conversation.
- **"Should I move for this job?"** — same role across 3–5 cities; compare median salaries to local cost of living.
- **Hiring-pattern intel on a competitor** — `indeed-listing --query "company:NAME"` (or use `glassdoor-listing` filtered by employer) returns recent postings; surfaces what teams are growing, what stack they're on, what locations they're in.
- **"Are they really hiring or is this a ghost job?"** — `indeed-job --url X --raw | jq '.posted_at'`. Postings older than 60 days that haven't been refreshed are often ghosts.
- **Stack popularity by region** — `indeed-listing --query "Rust" --location "Berlin"` vs `--location "San Francisco"` to compare absolute postings and salary deltas for a specific tech.
- **Career-pivot research** — `indeed-listing --query "TARGET ROLE" --raw | jq -r '.jobs[].description'` then summarize the most-required skills with an LLM. Reveals the actual gap, not what bootcamps claim.
- **Employer reputation deep-dive** — `glassdoor-listing` returns ratings inline; `glassdoor-job` returns recent reviews + interview difficulty. Use before applying or accepting.
- **Remote-job filter** — `--remote` on `indeed-listing` cuts to fully-remote postings; use `--country gb` etc. to find role markets that are remote-friendly outside the US.
- **Visa-friendly employer detection** — search `indeed-listing --query "ROLE H1B sponsorship"` or `--query "ROLE relocation"` — listings that mention these are more likely to support visa transfers.
- **Internship-only filter** — `--job-type internship` on `indeed-listing` for early-career searches.
- **Salary-band reverse engineering** — Glassdoor's `salary_estimate` is an estimate; cross-check by sampling 5–10 indeed postings for the same role/title in the same city and computing your own median.
- **"Who's leaving Company X?"** — `glassdoor-listing --keyword "previously at: COMPANY"` is a stretch query but sometimes surfaces postings where ex-employees describe their transition.
- **Detect layoffs before the news** — sudden surge in `indeed-listing --query "ex-COMPANY"` postings in a region often precedes the announcement.
- **Negotiation prep — interview difficulty** — `glassdoor-job --url X --raw | jq '.interview_difficulty, .interview_experiences[]'` shows how previous candidates rated the process.

## Common patterns

```bash
# Salary distribution for a role
hasdata indeed-listing --query "senior python developer" --location "New York, NY" \
  --raw | jq '[.jobs[].salary | select(. != null)] | unique'

# Compare same role on both platforms
for src in indeed-listing glassdoor-listing; do
  echo "=== $src ==="
  hasdata "$src" --query "platform engineer" --location "SF" --raw \
    | jq '.jobs[:5][] | {title, company: (.company // .employer), location}'
done
```

Start with Indeed for breadth; escalate to Glassdoor when employer ratings or interview-difficulty data are needed.


path: references/ecommerce.md
content: # E-commerce reference

Subcommands: `amazon-search`, `amazon-product`, `amazon-seller`, `amazon-seller-products`, `shopify-products`, `shopify-collections`. 5 credits each.

---

## amazon-search

```bash
hasdata amazon-search --q "wireless earbuds" [--domain amazon.com] --raw | jq '.results[]'
```

- `--q TEXT` (required)
- `--domain amazon.com|amazon.co.uk|amazon.de|amazon.in|amazon.co.jp|amazon.fr|amazon.it|amazon.es|amazon.com.br|amazon.com.mx|amazon.ca|amazon.com.au`
- `--page N` — pagination
- `--sort featured|price-asc-rank|price-desc-rank|review-rank|date-desc-rank`
- `--node ID` — restrict to category node
- `--customer-reviews` — only items with reviews

Per-result fields: `asin`, `title`, `price`, `link`, `image`, `rating`, `reviews_count`, `prime`.

## amazon-product

```bash
hasdata amazon-product --asin B08N5WRWNW [--domain amazon.com] --raw | jq .
```

- `--asin ASIN` (required)
- `--domain` as above
- `--include-reviews` — include first page of reviews
- `--include-html` — return raw HTML alongside parsed data

Returns: title, price, availability, features, descriptions, variants, images, ratings, top reviews, A+ content.

## amazon-seller

```bash
hasdata amazon-seller --seller-id A1234567890ABC [--domain amazon.com] --raw
```

Seller profile: name, ratings, review count, returns policy, shipping policy, "About" content.

## amazon-seller-products

```bash
hasdata amazon-seller-products --seller-id A1234567890ABC [--domain amazon.com] [--page 1] --raw
```

List of products from a specific seller — useful for competitor analysis or storefront crawling.

---

## shopify-products

```bash
hasdata shopify-products --url "https://store.example.com" [--page 1] --raw | jq '.products[]'
```

Works on any Shopify store (it queries the public `/products.json`-style endpoint). Returns title, vendor, product_type, variants[] (with prices, SKUs, stock), images, tags, handle.

## shopify-collections

```bash
hasdata shopify-collections --url "https://store.example.com" --raw | jq '.collections[]'
```

For drilling into a specific collection, append `/collections/SLUG` to the URL or use the collection handle returned here.

---

## Non-obvious use cases

- **Cross-marketplace price arbitrage** — same `--asin` across `--domain amazon.com|amazon.co.uk|amazon.de` shows currency-normalized regional differences; useful for grey-market resellers and buyers shipping internationally.
- **"Is this product still available?"** — `amazon-product --asin X --raw | jq '.availability'`. Avoids hallucinating an answer based on stale training data.
- **Variant matrix dump** — `amazon-product --asin X --raw | jq '.variants[] | {asin, color, size, price}'` returns the full color/size/etc. lattice with current prices.
- **Counterfeit-listing detection** — `amazon-search --q "BRAND PRODUCT" --raw` then check `.results[].seller_name` for non-authorized sellers; pivot to `amazon-seller` to inspect their other listings.
- **Storefront catalog** — `amazon-seller-products --seller-id X --page 1..N` paginates a seller's full catalog; useful for competitor analysis or due diligence on a vendor.
- **"What were the bestsellers in category X?"** — `amazon-search --q "CATEGORY KEYWORD" --sort featured` returns Amazon's own ranking (`--sort review-rank` for review-weighted).
- **Lead enrichment for a Shopify store** — `shopify-products --url store.example.com --raw` exposes vendor names, tags, product types, SKUs — useful for competitor product-line audits.
- **Stock check** — `shopify-products` returns each variant's `available` boolean; can power "notify me when X is back" without scraping the cart UI.
- **Price-drop monitoring** — schedule `amazon-product --asin X` daily; persist `.price` to a file; alert when delta > N%.
- **A/B-test detection** — same `--asin` from two different `--proxy-country` settings (via web-scraping fallback if amazon-product doesn't support it) sometimes shows different price/title due to A/B tests.
- **Gift-card / coupon discovery** — `google-shopping --q "PRODUCT"` often surfaces resellers offering rebates Amazon doesn't show.
- **Product-image extraction for a moodboard** — `amazon-product --asin X --raw | jq -r '.images[]'` returns CDN URLs you can download separately.
- **Compare reviews summary** — `amazon-product --asin X --include-reviews --raw | jq '.reviews[] | {rating, title, body}'` gives a quick sentiment sample without scraping the review page.
- **Build a Shopify product feed for ads** — `shopify-products --url X --raw | jq -c '.products[] | {id, title, price: .variants[0].price, url: ("https://" + $store + "/products/" + .handle)}'`.

## Common patterns

```bash
# Price tracking — pull current price for a known ASIN
hasdata amazon-product --asin "$ASIN" --raw | jq '.price'

# Product discovery → details fan-out
hasdata amazon-search --q "$Q" --raw \
  | jq -r '.results[].asin' \
  | head -5 \
  | xargs -I{} hasdata amazon-product --asin {} --raw

# Compare across marketplaces
for d in amazon.com amazon.co.uk amazon.de; do
  echo "=== $d ==="
  hasdata amazon-product --asin "$ASIN" --domain "$d" --raw \
    | jq '{currency: .currency, price: .price}'
done
```



path: references/real-estate.md
content: # Real-estate reference

Subcommands: `zillow-listing`, `zillow-property`, `redfin-listing`, `redfin-property`, `airbnb-listing`, `airbnb-property` — 5 credits each.

`*-listing` is for filtered searches; `*-property` is a single-property deep dive.

---

## zillow-listing

```bash
hasdata zillow-listing --keyword "Austin, TX" --type forSale [filters] --raw | jq '.results[]'
```

Required:
- `--keyword "City, ST"` (default: `New York, NY`)
- `--type forSale|forRent|sold` (default: `forSale`)

Price / size (bracketed pairs, kept as floats):
- `--price-min N --price-max N`
- `--beds-min N --beds-max N`
- `--baths-min N --baths-max N`
- `--square-feet-min N --square-feet-max N`
- `--lot-size-min N --lot-size-max N`
- `--year-built-min N --year-built-max N`
- `--hoa N` — max HOA fee
- `--parking-spots-min N`

Array filters (enum-validated, lowercase camelCase values):
- `--home-types house|townhome|multiFamily|condo|lot|apartment|manufactured` (repeatable)
- `--pets allowsLargeDogs|allowsSmallDogs|allowsCats` (repeatable)
- `--other-amenities ac|pool|waterfront|onsiteParking|inUnitLaundry|acceptZillowApplications|incomeRestricted|apartmentCommunity` (repeatable)
- `--views city|mountain|park|water` (repeatable)
- `--basement finished|unfinished` (repeatable)
- `--property-status comingSoon|acceptingBackupOffers|pendingAndUnderContract` (repeatable)
- `--listing-publish-options ownerPosted|agentListed|newConstruction|foreclosures|auctions|foreclosed|preForeclosures` (repeatable)
- `--tours open|3d` (repeatable)

Booleans:
- `--must-have-garage` — only listings with a garage
- `--single-story-only`
- `--hide55plus-communities`

Other:
- `--listing-type byAgent|byOwner`
- `--days-on-zillow 1|7|14|30|90|6m|12m|24m|36m`
- `--keywords "open floor plan"` — refinement keywords (matches in description)
- `--move-in-date 2026-06-01`
- `--page N` — pagination
- `--sort verifiedSource|homesForYou|priceHighToLow|priceLowToHigh|paymentHighToLow|paymentLowToHigh|newest|bedrooms|bathrooms|squareFeet|lotSize`

### Examples

```bash
# Family home, mid-market, sorted cheapest first
hasdata zillow-listing \
  --keyword "Austin, TX" --type forSale \
  --price-min 400000 --price-max 900000 \
  --beds-min 3 --beds-max 5 --baths-min 2 \
  --home-types house --home-types townhome \
  --sort priceLowToHigh --raw | jq '.results[] | {address, price, beds, baths}'

# Pet-friendly rental
hasdata zillow-listing \
  --keyword "Seattle, WA" --type forRent \
  --price-max 4000 \
  --pets allowsSmallDogs --pets allowsCats \
  --parking-spots-min 1 --must-have-garage \
  --raw

# Recently sold comps
hasdata zillow-listing \
  --keyword "Miami, FL" --type sold \
  --square-feet-min 1500 --square-feet-max 4000 \
  --year-built-min 2000 --year-built-max 2020 \
  --days-on-zillow 12m --sort newest --raw
```

Bracketed query params (`price[max]`, `homeTypes[]`, `yearBuilt[min]`) are handled by the CLI — pass the kebab-case flags shown above, not the raw API names.

## zillow-property

```bash
hasdata zillow-property --url "https://www.zillow.com/homedetails/.../123_zpid/" --raw | jq .
```

Or with `--zpid <ID>`. Returns full property details (photos, history, schools, taxes, walk-score, etc.).

## redfin-listing

Similar shape to `zillow-listing` but Redfin's enums differ. Run `hasdata redfin-listing --help` for the exact list. Common pattern:

```bash
hasdata redfin-listing --location "San Francisco, CA" --status forSale \
  --min-price 800000 --max-price 1500000 \
  --min-beds 2 --raw
```

## redfin-property

```bash
hasdata redfin-property --url "https://www.redfin.com/CA/San-Francisco/.../home/12345" --raw
```

## airbnb-listing

```bash
hasdata airbnb-listing --location "Lisbon, Portugal" \
  --check-in 2026-06-15 --check-out 2026-06-22 \
  --adults 2 --price-max 200 --raw
```

Run `--help` for the full filter set (room type, amenities, instant book, etc.).

## airbnb-property

```bash
hasdata airbnb-property --url "https://www.airbnb.com/rooms/12345678" --raw
```

---

## Non-obvious use cases

- **Investment screening** — combine `--type sold` + `--days-on-zillow 12m` + `--year-built-min` + `--lot-size-min` to surface flip / value-add candidates. Then `xargs` into `zillow-property` for ARV analysis.
- **Tax-appeal comps** — `--type sold --keyword "ZIP CODE" --days-on-zillow 12m` filtered to your home's beds/baths/sqft band gives recent sales the assessor used; export to CSV with `jq -r '.results[] | [.address, .price, .beds, .baths, .squareFootage, .soldDate] | @csv'`.
- **Appraiser comp pull** — same trick, narrower square-footage and same year-built band.
- **Motivated-seller signal** — `--type forSale --days-on-zillow 90` returns listings that have lingered. Often willing to negotiate.
- **Pre-relocation neighborhood scan** — run the same `--type forRent` filter across 5–10 neighborhoods, dump rent distributions with `jq '.results[].price'`, eyeball cost differences before booking visits.
- **STR-vs-LTR feasibility** — pair `airbnb-listing` for nightly rates with `zillow-listing --type forSale` for purchase price in the same area; compute gross yield client-side.
- **HOA filter** — `--hoa N` caps fee; useful for buyers who want max payment ceilings.
- **School-driven house hunt** — `zillow-property` returns school ratings; filter `zillow-listing` results down by walking each property and keeping those with rating ≥ X.
- **Open-houses this weekend** — Zillow tags open-house listings; check `.results[].openHouseTimes` for upcoming slots.
- **3D-tour / virtual-tour-only filter** — `--tours 3d` → only listings with virtual tours. Useful for remote / international buyers.
- **Pet-friendly rentals at scale** — `--pets allowsLargeDogs --pets allowsCats` for multi-pet households. Pairs well with `--keyword` for specific neighborhoods.
- **Foreclosure and pre-foreclosure leads** — `--listing-publish-options foreclosures --listing-publish-options preForeclosures`.
- **Non-traditional listing types** — `--listing-type byOwner` for FSBO; `--listing-type byAgent` for agent-listed (default mix).
- **Move-in date constraint** — `--move-in-date YYYY-MM-DD` for rental searches with a hard timing requirement.
- **Bulk address verification** — pipe a list of property URLs through `zillow-property` to confirm they resolve and pull the canonical address Zillow uses.
- **Airbnb price-arbitrage check** — same dates, same area, two `airbnb-listing` calls with different `--adults` counts to surface listings that don't scale per-person price. Sometimes the difference is the deal.
- **Verify a Redfin/Zillow listing is real** — `redfin-property --url X --raw | jq .status` to confirm it hasn't been pulled.



path: references/all-commands.md
content: # All commands

Authoritative source: `hasdata --help`. This file is a snapshot — when in doubt, run `hasdata <api> --help` directly.

## Search

| Command | Cost | Notes |
| --- | --- | --- |
| `google-serp` | 10 | Full Google SERP — organic, ads, knowledge graph, PAA, AI overview |
| `google-serp-light` | 5 | Cheap single-page SERP |
| `google-ai-mode` | 5 | Google AI Mode answer |
| `google-news` | 10 | Google News results |
| `google-shopping` | 10 | Google Shopping results |
| `google-immersive-product` | 5 | Immersive product page details |
| `google-events` | 5 | Google Events |
| `google-short-videos` | 10 | Short videos panel |
| `google-trends` | 5 | Search trends data |
| `google-images` | 5 | Image search |
| `google-flights` | 15 | Flight search |
| `bing-serp` | 10 | Bing SERP |

## Maps & local

| Command | Cost | Notes |
| --- | --- | --- |
| `google-maps` | 5 | Maps search |
| `google-maps-place` | 5 | Single place by place_id |
| `google-maps-reviews` | 5 | Place reviews |
| `google-maps-contributor-reviews` | 5 | Reviews by contributor |
| `google-maps-photos` | 5 | Place photos |
| `yelp-search` | 5 | Yelp business search |
| `yelp-place` | 5 | Single Yelp business |
| `yellowpages-search` | 10 | YellowPages search |
| `yellowpages-place` | 10 | Single YellowPages listing |

## E-commerce

| Command | Cost | Notes |
| --- | --- | --- |
| `amazon-search` | 5 | Amazon search results |
| `amazon-product` | 5 | Amazon product by ASIN |
| `amazon-seller` | 5 | Amazon seller profile |
| `amazon-seller-products` | 5 | Seller's product catalog |
| `shopify-products` | 5 | Any Shopify store's products |
| `shopify-collections` | 5 | Shopify collections |

## Real estate

| Command | Cost | Notes |
| --- | --- | --- |
| `zillow-listing` | 5 | Zillow filtered search |
| `zillow-property` | 5 | Zillow single property |
| `redfin-listing` | 5 | Redfin filtered search |
| `redfin-property` | 5 | Redfin single property |
| `airbnb-listing` | 5 | Airbnb filtered search |
| `airbnb-property` | 5 | Airbnb single listing |

## Jobs

| Command | Cost | Notes |
| --- | --- | --- |
| `indeed-listing` | 5 | Indeed search |
| `indeed-job` | 5 | Single Indeed posting |
| `glassdoor-listing` | 10 | Glassdoor search (with ratings) |
| `glassdoor-job` | 10 | Single Glassdoor posting |

## Web

| Command | Cost | Notes |
| --- | --- | --- |
| `web-scraping` | 10 | Arbitrary URL — JS rendering, AI extraction, markdown output, screenshots |

## Social

| Command | Cost | Notes |
| --- | --- | --- |
| `instagram-profile` | 10 | Instagram profile by handle |

## Utility (no cost)

| Command | Notes |
| --- | --- |
| `configure` | Interactive setup; writes ~/.hasdata/config.yaml |
| `version` | Print version |
| `update` | Self-update from GitHub Releases |
| `completion {bash\|zsh\|fish\|powershell}` | Generate shell completion |

## Hidden / deprecated

`amazon-reviews` is hidden — Amazon now requires login for review scraping. Don't suggest it.


path: references/enrichment.md
content: # Data enrichment

Enriching a name, email, domain, or row of a CSV with public data. The highest-leverage workflow this skill enables — turning a sparse list (just names) into a structured dataset (name + role + employer + LinkedIn + emails + company HQ + recent activity).

## SERP first. Web-scraping is the last resort.

`google-serp` is your primary enrichment tool. Reasons:

- **Google has already extracted the structured fields you want.** `.knowledge_graph` carries HQ, founder, founded year, parent company, employees, industry. `.organic_results[]` titles and snippets carry person → role → employer mappings (LinkedIn titles are literally `Name — Role at Company`). `.local_results[]` carry phone/address/hours.
- **It bypasses anti-bot.** Many target sites (LinkedIn, Crunchbase, Glassdoor) gate or rate-limit direct access. Google's snippets reflect the same data without you needing to render their JS or get a 403.
- **It's the catch-all reverse index.** Quoted queries (`--q '"[REDACTED_SECRET]"'`, `--q '"+1 555 123 4567"'`, `--q '"Acme Corp"'`) work as a universal lookup over the entire indexed web.

Use `google-serp` (or `google-news` for recency, `google-maps` for places, `google-shopping` for products) **first**. Only fall through to `web-scraping` when:
- A specific field you need isn't in any SERP snippet, AND
- The target page renders that field server-side or via JS that the scraper can handle, AND
- The user explicitly needs it (don't fan out to N web-scraping calls when SERP would have answered N − 0 of them).

The patterns below show the full chain so you understand when to escalate. Most rows in a real CSV stop after step 1 or 2.

---

## Person enrichment

### Step 1 — SERP for role, employer, LinkedIn URL

```bash
hasdata google-serp --q '"Jane Doe" linkedin' --num 5 --raw \
  | jq -c '.organic_results[] | select(.link | contains("linkedin.com/in/")) |
           {title, snippet, link}'
```

The result usually looks like:

```json
{
  "title": "Jane Doe — Senior Engineer at Acme Corp | LinkedIn",
  "snippet": "San Francisco, CA · 500+ connections · Engineering @ Acme. Previously...",
  "link": "https://www.linkedin.com/in/janedoe"
}
```

You now have role, employer, location, LinkedIn URL, and a connection-count hint — without scraping anything. **Stop here unless a specific extra field is required.**

### Step 2 — Refine with targeted SERP queries

If step 1 didn't carry what you need, ask Google more specifically:

```bash
# Disambiguate by company
hasdata google-serp --q '"Jane Doe" "Acme Corp"' --num 10 --raw \
  | jq -c '.organic_results[] | {title, snippet, link}'

# Other social profiles
hasdata google-serp --q '"Jane Doe" site:twitter.com OR site:x.com' --num 3 --raw
hasdata google-serp --q '"Jane Doe" site:github.com' --num 3 --raw

# Past employers / bio paragraphs
hasdata google-serp --q '"Jane Doe" bio OR background OR experience' --num 5 --raw \
  | jq -r '.organic_results[].snippet'
```

### Step 3 — Web-scraping (only if SERP came up short)

When SERP snippets truncated the field you need, or the user explicitly wants the full profile content:

```bash
hasdata web-scraping --url "https://www.linkedin.com/in/janedoe" \
  --output-format markdown --no-screenshot --no-block-resources \
  --raw | jq -r .markdown
```

Or for structured fields, AI extraction:

```bash
hasdata web-scraping --url "https://www.linkedin.com/in/janedoe" \
  --ai-extract-rules-json '{
    "headline":   {"type": "string"},
    "location":   {"type": "string"},
    "company":    {"type": "string"},
    "role":       {"type": "string"},
    "followers":  {"type": "number"},
    "experience": {"type": "list", "output": {
      "company":  {"type": "string"},
      "role":     {"type": "string"},
      "duration": {"type": "string"}
    }}
  }' --raw | jq .
```

LinkedIn sometimes blocks the public preview; if it does, fall back to step 2 (combining SERP snippets) — it's almost always enough.

### Email lookup

Triangulate, don't promise. SERP first, scraping last.

```bash
# 1. Has Google already indexed the email anywhere?
hasdata google-serp --q '"[REDACTED_SECRET]"' --num 10 --raw \
  | jq -c '.organic_results[] | {title, snippet, link}'

# 2. What email format does the company use? Look for any indexed @company.com address.
hasdata google-serp --q 'site:acme.com "@acme.com"' --num 10 --raw \
  | jq -r '.organic_results[].snippet' \
  | grep -oE '[A-Za-z0-9._-]+@acme\.com' | sort -u

# 3. Pattern-guess + SERP-verify
for guess in "jane.doe" "jdoe" "jane" "j.doe" "janed"; do
  count=$(hasdata google-serp --q "\"$[REDACTED_SECRET]\"" --num 1 --raw \
            | jq -r '.organic_results | length')
  [ "$count" -gt 0 ] && echo "$[REDACTED_SECRET]  (appears in SERP)"
done

# 4. Last resort — scrape the company's contact / about / team pages for emails
hasdata web-scraping --url "https://acme.com/about" --extract-emails --raw \
  | jq -r '.emails // [] | .[]'
```

Always tell the user when an email is a pattern-guess vs. confirmed via SERP/scrape.

---

## Company enrichment

### Step 1 — SERP knowledge_graph

```bash
hasdata google-serp --q "Acme Corp" --num 5 --raw | jq '.knowledge_graph // {}'
```

`.knowledge_graph` typically contains: founder, founded (year), headquarters, parent_organization, ceo, employees (range), revenue, stock_price, industry, products. **For the majority of company enrichment requests, this single call is the entire answer.**

### Step 2 — Targeted SERP for specific fields

```bash
# Headquarters
hasdata google-serp --q '"Acme Corp" headquarters' --num 5 --raw \
  | jq -r '.organic_results[].snippet'

# Funding / acquisition signals
hasdata google-serp --q '"Acme Corp" raises OR acquires OR acquired OR ipo OR funding' --num 10 --raw \
  | jq -c '.organic_results[] | {title, snippet, link}'

# Recent news
hasdata google-news --q "Acme Corp" --gl us --raw \
  | jq -c '.news_results[] | {title, source: .source.name, date, link}'

# LinkedIn company page
hasdata google-serp --q '"Acme Corp" site:linkedin.com/company' --num 3 --raw \
  | jq -c '.organic_results[] | {title, snippet, link}'

# Employee profiles in a specific function/region
hasdata google-serp \
  --q 'site:linkedin.com/in "Acme Corp" engineer' --gl us --num 25 --raw \
  | jq -r '.organic_results[] | "\(.title)\t\(.link)"'
```

### Step 3 — Web-scraping (only when SERP can't fill a specific field)

```bash
# Company About page → AI-extract structured fields
hasdata web-scraping --url "https://acme.com/about" \
  --ai-extract-rules-json '{
    "name":         {"type": "string"},
    "founded":      {"type": "number"},
    "headquarters": {"type": "string"},
    "employees":    {"type": "string"},
    "industry":     {"type": "string"},
    "description":  {"type": "string"},
    "products":     {"type": "list"}
  }' --raw | jq .
```

Reach for this only when the user wants something SERP can't provide (e.g. mission statement verbatim, full product taxonomy, leadership team page parsed into rows).

---

## CSV row enrichment

For a list of N rows, fan out one or two SERP calls per row. Keep web-scraping out of the loop unless a specific row needs it.

```bash
# Input: people.csv with one column "name"
while IFS=, read -r name; do
  result=$(hasdata google-serp --q "\"$name\" linkedin" --num 1 --raw)
  linkedin=$(echo "$result" | jq -r '.organic_results[0].link // ""')
  title=$(echo "$result"    | jq -r '.organic_results[0].title // ""')
  snippet=$(echo "$result"  | jq -r '.organic_results[0].snippet // ""')
  printf '%s\t%s\t%s\t%s\n' "$name" "$title" "$snippet" "$linkedin"
done < people.csv > enriched.tsv
```

That's it. One SERP call per row, role/employer/LinkedIn extracted from the title and snippet. Add a second SERP call only if a row's first result didn't match (`select(.title | test("\(name)"; "i"))` filtering for confidence).

---

## Reverse-lookup

Always SERP-first with the literal value quoted.

```bash
# Email → identity
hasdata google-serp --q '"[REDACTED_SECRET]"' --num 10 --raw \
  | jq -c '.organic_results[] | {title, snippet, link}'

# Phone → owner / business
hasdata google-serp --q '"+1 555 123 4567"' --num 10 --raw
# Combine with yelp-search / yellowpages-search if it's a business number.

# Domain → company
hasdata google-serp --q "site:example.com" --num 5 --raw \
  | jq '.organic_results[0].title'
hasdata google-serp --q "Acme Corp" --num 5 --raw | jq '.knowledge_graph // {}'
```

Only scrape the domain (`web-scraping --url "https://example.com"`) when you specifically need the homepage's body text.

---

## Tips for reliable enrichment

- **Always quote names and other multi-token strings** — `"Jane Doe"` matches the exact person; `Jane Doe` matches noise.
- **Use `site:` aggressively** — `site:linkedin.com/in/`, `site:linkedin.com/company/`, `site:github.com`, `site:crunchbase.com`. Google's `site:` is the cheapest way to scope an enrichment search.
- **Read the `.knowledge_graph`** before doing anything else for a company. If it's populated, you're often done.
- **AI-extract over CSS selectors** when you do need to scrape — LinkedIn / Crunchbase / About-page markup changes constantly; AI extraction with field names + descriptions survives layout churn.
- **Cross-source verify** — never enrich from a single source. If LinkedIn's title says "Acme Corp" and a `--q '"Jane Doe" "Acme Corp"'` SERP corroborates with multiple results, confidence is high.
- **Mark guesses** — pattern-guessed emails, inferred locations, single-source roles should be flagged to the user as unverified.


path: references/web-scraping.md
content: # web-scraping reference

Subcommand: `web-scraping` (10 credits/call). Single endpoint for arbitrary URL scraping with JS rendering, proxies, AI extraction, screenshots, markdown conversion.

> **`web-scraping` is the last resort, not the default.** Before reaching for it, ask: would `google-serp` (or `google-news` / `google-shopping` / `google-maps`) already have the field I need? Google's `.knowledge_graph`, `.organic_results[].snippet`, `.local_results[]` carry pre-extracted facts that bypass anti-bot, JS rendering, and selector-maintenance overhead. Only invoke `web-scraping` when:
>
> - the user gave you a specific URL to read, OR
> - SERP came up short for a specific field, OR
> - the target page renders content that doesn't show up in any SERP snippet.
>
> See `references/enrichment.md` for the SERP-first patterns.

```bash
hasdata web-scraping --url "URL" [flags] --raw | jq .
```

## Required

- `--url URL` — target page

## Output formats

- `--output-format html|text|markdown|json` (repeatable)
  - One format → that format directly (e.g. `--output-format markdown` returns markdown text under `.markdown`)
  - Multiple → JSON response with one key per format
  - `--output-format json` combined with others → wraps everything in JSON

```bash
# LLM-friendly markdown for prompt context
hasdata web-scraping --url "$URL" --output-format markdown --raw | jq -r .markdown
```

## Proxy & rendering

- `--proxy-type datacenter|residential` (default datacenter)
- `--proxy-country US|UK|DE|IE|FR|IT|SE|BR|CA|JP|SG|IN|ID` (default US)
- `--js-rendering` / `--no-js-rendering` (default on) — full headless browser
- `--block-ads` / `--no-block-ads` (default on)
- `--block-resources` / `--no-block-resources` (default on) — blocks images/CSS for speed
- `--screenshot` / `--no-screenshot` (default on; the result includes a screenshot URL)
- `--remove-base64-images` — strip inline base64 images from response
- `--extract-emails` / `--no-extract-emails` (default on)
- `--extract-links` (default off)

## Wait controls

- `--wait MS` — fixed wait after page load
- `--wait-for "CSS_SELECTOR"` — wait until selector appears

## Custom JS scenario (complex array — JSON only)

```bash
hasdata web-scraping --url "$URL" \
  --js-scenario-json '[
    {"wait": 2000},
    {"click": ".load-more"},
    {"waitFor": ".item"},
    {"scrollY": 1000},
    {"fill": ["input#q", "espresso"]}
  ]' --raw
```

Supported actions: `evaluate`, `click`, `wait`, `waitFor`, `waitForAndClick`, `scrollX`, `scrollY`, `fill`. Executed sequentially.

Accepts raw JSON, `@file.json`, or `-` (stdin).

## Headers (kvSlice + JSON escape)

```bash
# Repeatable kv form (splits on first `=`)
hasdata web-scraping --url "$URL" \
  --headers "User-Agent=hasdata-cli" \
  --headers "Accept-Language=en-US,en;q=0.9" \
  --headers "Cookie=session=abc=def" \
  --raw

# JSON base + kv overrides
hasdata web-scraping --url "$URL" \
  --headers-json '{"User-Agent":"base","X-Common":"shared"}' \
  --headers "User-Agent=override" \
  --raw
```

## CSS-selector data extraction (kvSlice or JSON)

```bash
# Lightweight kv form: --extract-rules KEY=SELECTOR
hasdata web-scraping --url "https://quotes.toscrape.com" \
  --extract-rules "quote=.quote .text" \
  --extract-rules "author=.quote .author" \
  --raw | jq .

# JSON form for complex selectors / attributes
hasdata web-scraping --url "$URL" \
  --extract-rules-json '{"title":"h1","links":"a @href","price":".price-now"}' \
  --raw
```

`@href`, `@src`, etc. extract attributes. Without `@`, extracts text content.

## AI extraction (LLM-driven)

```bash
hasdata web-scraping --url "$URL" \
  --ai-extract-rules-json '{
    "headline": {"type": "string", "description": "the main story headline"},
    "comments_count": {"type": "number"},
    "is_paid_content": {"type": "boolean"},
    "tags": {"type": "list", "description": "topic tags"},
    "author": {"type": "item", "output": {
      "name": {"type": "string"},
      "verified": {"type": "boolean"}
    }}
  }' --raw | jq .
```

Supported types: `string`, `number`, `boolean`, `list`, `item` (nested object — defines its shape under `output`).

## Tag filtering

- `--include-only-tags "main,article"` (comma-joined CSS selectors) — keep only matching elements
- `--exclude-tags script --exclude-tags style` (repeatable) — remove elements

```bash
hasdata web-scraping --url "$URL" \
  --output-format markdown \
  --include-only-tags "article,main" \
  --exclude-tags script --exclude-tags style --exclude-tags nav \
  --raw | jq -r .markdown
```

## URL blocklist

```bash
--block-urls-json '["**.googletagmanager.com/**","**.doubleclick.net/**"]'
```

Glob patterns block specific subresource URLs from loading.

## Saving binary output

The `web-scraping` response is JSON, but if `--output-format` is set to a single non-JSON format, the wrapped result is still JSON. Use `jq -r .markdown > file.md` to extract text. For screenshots specifically, the response contains a screenshot URL — fetch it separately with `curl`.

## Non-obvious use cases

- **Page-to-prompt grounding** — `--output-format markdown` produces clean LLM-ready text from any URL. Strip nav/ads with `--exclude-tags script --exclude-tags style --exclude-tags nav`. Beats fetch + regex.
- **JavaScript-rendered SPAs that `curl` can't read** — default `--js-rendering` uses a real browser, so React/Vue/Angular pages return their hydrated DOM, not the empty shell.
- **Bypass rudimentary anti-bot** — `--proxy-type residential` rotates through residential IPs; use only when datacenter (default) gets blocked.
- **Geo-targeted content** — `--proxy-country DE` to see what users in Germany see (different prices, currencies, A/B variants, or geo-blocked content).
- **Quick "is this page real" check** — `--screenshot` (default on) returns a screenshot URL in the response; verify visually without manually opening the URL.
- **Universal price extractor** — `--ai-extract-rules-json '{"price":{"type":"number"},"currency":{"type":"string"},"in_stock":{"type":"boolean"}}'` works on arbitrary retailer pages without writing a CSS selector. Cheaper than maintaining per-site selectors when the user only needs occasional spot-checks.
- **Cookie-walled / login-walled "view-source" peek** — `--headers Cookie=session=...` injects auth cookies if the user has them. Use sparingly and only with explicit user permission.
- **Convert paginated lists to a clean record set** — combine `--js-scenario-json` (click "Load more" 5×) with `--ai-extract-rules-json` (pull the list shape). Lets you scrape paginated SPAs with one CLI call instead of N.
- **Headless screenshot of a layout** — set `--js-rendering`, `--no-block-resources` (so CSS loads), and capture the screenshot URL from the response. Useful for "render this URL and show me what it looks like".
- **Markdown for RAG ingestion** — pipe `.markdown` from many URLs into a JSONL corpus; embed and store. The CLI handles JS, ads, images so you don't need a custom pipeline.
- **Fallback for any other API** — when no purpose-built API exists for a vertical (e.g. niche directories, government pages, less-popular real-estate sites), `web-scraping` is the catch-all.
- **Detect content changes** — schedule `web-scraping --url X --output-format markdown` and diff the output across runs to flag pricing-page or terms-of-service changes.
- **Read PDFs / non-HTML resources** — `--output-format text` works on text-extractable PDFs accessible via URL (the underlying renderer handles them).
- **AI extraction for forms or tables** — pages with structured data in HTML tables are easy: `--ai-extract-rules-json '{"rows":{"type":"list","output":{"name":{"type":"string"},"value":{"type":"number"}}}}'`. The model fills in nested rows.

## Common patterns

```bash
# Full-page markdown for RAG
hasdata web-scraping --url "$URL" \
  --output-format markdown --no-screenshot --no-block-resources \
  --raw | jq -r .markdown >> corpus.md

# JS-heavy SPA: wait + scroll
hasdata web-scraping --url "$URL" \
  --js-scenario-json '[{"wait":2000},{"scrollY":2000},{"wait":1500}]' \
  --wait-for ".item" \
  --output-format html --raw | jq -r .html

# Extract structured data from an arbitrary page
hasdata web-scraping --url "$URL" \
  --ai-extract-rules-json '{"price":{"type":"number"},"in_stock":{"type":"boolean"}}' \
  --raw | jq .
```


path: references/local-business.md
content: # Local-business / maps reference

Subcommands: `google-maps`, `google-maps-place`, `google-maps-reviews`, `google-maps-contributor-reviews`, `google-maps-photos`, `yelp-search`, `yelp-place`, `yellowpages-search`, `yellowpages-place`. 5 credits each.

---

## google-maps

```bash
hasdata google-maps --q "coffee" --ll "@30.2672,-97.7431,14z" --raw | jq '.local_results[]'
```

Required: `--q TEXT`. Other useful flags:
- `--ll "@LAT,LNG,ZOOMz"` — center point + zoom (Google's `ll` parameter)
- `--gl us|gb|...` — country
- `--hl en|es|...` — language
- `--type search|place` — search vs. specific place lookup
- `--data` / `--cid` / `--fid` — Google identifiers (advanced)

Per-result: `title`, `place_id`, `rating`, `reviews`, `address`, `phone`, `website`, `types[]`, `gps_coordinates`.

## google-maps-place

```bash
hasdata google-maps-place --place-id "ChIJ..." --raw | jq .
```

Single place: full details, hours, popular times, attributes.

## google-maps-reviews

```bash
hasdata google-maps-reviews --place-id "ChIJ..." [--sort newest|highest|lowest] --raw \
  | jq '.reviews[] | {author, rating, date, snippet}'
```

For pagination, the response includes a `next_page_token` — pass via `--next-page-token`.

## google-maps-contributor-reviews

Reviews authored by a specific Google contributor (by `--contributor-id`). Useful for local-guide analysis.

## google-maps-photos

```bash
hasdata google-maps-photos --place-id "ChIJ..." --raw
```

Returns photo URLs by category (interior, exterior, food, etc.).

---

## yelp-search

```bash
hasdata yelp-search --query "italian" --location "Brooklyn, NY" [--page 1] --raw \
  | jq '.businesses[] | {name, rating, review_count, price, categories}'
```

## yelp-place

```bash
hasdata yelp-place --url "https://www.yelp.com/biz/SLUG" --raw | jq .
```

Single business: full details, hours, top reviews, photos, attributes.

## yellowpages-search

```bash
hasdata yellowpages-search --search-terms "plumber" --geo-location-terms "Atlanta, GA" --raw
```

## yellowpages-place

```bash
hasdata yellowpages-place --url "https://www.yellowpages.com/atlanta-ga/mip/..." --raw
```

---

## Non-obvious use cases

- **Sales-lead generation** — `google-maps --q "INDUSTRY" --ll "@LAT,LNG,12z"` to enumerate competitors/prospects, then `xargs` into `google-maps-place` for phone/email/website.
- **Reputation monitoring** — `google-maps-reviews --place-id X --sort lowest` returns the worst reviews first; great for surfacing crisis signals fast. Run weekly to detect new 1-star drops.
- **"Is this business open?"** — `google-maps-place --place-id X --raw | jq '.hours'` for current hours; also surfaces `permanently_closed` status.
- **Verify an address user gave you** — `google-maps --q "BUSINESS NAME, CITY" --raw | jq '.local_results[0].address'`. Don't trust user-provided addresses for high-stakes actions (mailing, payments).
- **Phone-number / email lookup** — `google-maps-place` includes phone and website; `web-scraping --url WEBSITE --extract-emails` (default-on) returns emails parsed from the homepage. Combine for full contact detail.
- **Competitive-density mapping** — `google-maps --q "coffee shop" --ll "@LAT,LNG,Zz"` at varying zoom levels; aggregate `.local_results[]` into a CSV with addresses + ratings to find under-served zones.
- **Service-area validation** — chain `yelp-search --location "CITY"` for a few cities to confirm a business with the same name covers them.
- **Find recently opened businesses** — `google-maps-place` returns `description.years_in_business` and posts/updates timestamps; sort.
- **Negative-review sample for product analysis** — `yelp-place --url X --raw | jq '.reviews[] | select(.rating <= 2)'`; useful as input to a "what do customers complain about" summary.
- **Photo-mining for visual docs** — `google-maps-photos --place-id X` returns categorized photo URLs (interior, food, exterior). Useful when the user needs an image and doesn't want to rely on `google-images`.
- **Local-guide credibility** — `google-maps-contributor-reviews --contributor-id X` shows everything a specific reviewer wrote, useful for filtering out shilling/fake reviewers when their pattern is suspicious.
- **YellowPages for B2B niches** — service-business categories (plumbers, electricians, lawyers) are often better indexed by YellowPages than Yelp; try both when one comes up empty.
- **Cross-platform reputation diff** — same business name + city via `yelp-search` and `google-maps` to compare ratings across platforms (gap often signals fake reviews on one).

## Common patterns

```bash
# Build a directory: search → fan out to per-place details
hasdata google-maps --q "yoga studios" --ll "@30.27,-97.74,12z" --raw \
  | jq -r '.local_results[].place_id' \
  | head -10 \
  | xargs -I{} hasdata google-maps-place --place-id {} --raw

# Sentiment / review analysis
hasdata google-maps-reviews --place-id "$PID" --sort lowest --raw \
  | jq '.reviews[] | {rating, snippet}'
```


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (CRITICAL)