npm package: @olostep/mastra-tools
Description: Olostep is a Web search, scraping and crawling API — an API to search, extract and structure web data for Mastra.
Author: Olostep
Latest version: 1.0.2

# @olostep/mastra-tools

Olostep is a Web search, scraping and crawling API — an API to search, extract and structure web data. This package adds an Olostep Integration to Mastra so your agents and workflows can search, scrape, crawl and map websites with a few lines of code.

Navigation
- Installation
- What you can build
- APIs (parameters, responses, use cases)
- Quick examples
- Parsers
- Support & Resources

## Installation

```bash
npm install @olostep/mastra-tools
```

Register the Integration in your Mastra setup:
```ts
import { Mastra } from '@mastra/core';
import { createOlostepIntegration } from '@olostep/mastra-tools';

const olostep = createOlostepIntegration();
olostep.registerApis(); // exposes the 4 Olostep APIs to Mastra

export const mastra = new Mastra({
  config: {
    integrations: [olostep],
  },
});
```

## What you can build

- Research agents that search, extract, and summarize web content
- E‑commerce price/stock monitors across thousands of product pages
- SEO workflows that map sites, scrape content, and analyze topics
- Competitive intelligence dashboards powered by live web data
- Knowledge ingestion pipelines for RAG

## Available APIs

All APIs accept your Olostep API key (temporary schema parameter) and return Olostep’s structured responses.

### 1) scrapeWebsite
Extract content from a single URL. Supports HTML, Markdown, JSON, and text. Optional wait to render JS content.

Parameters:
- apiKey (string, required): Your Olostep API key
- url_to_scrape (string, required): URL including http/https
- formats (array, default ['markdown']): 'html' | 'markdown' | 'json' | 'text'
- country (string, optional): 2‑letter country code (e.g. 'US')
- wait_before_scraping (number, optional): milliseconds (0–10000)
- parser (string, optional): e.g. '@olostep/amazon-product'

Response highlights:
- object: 'scrape'
- id, created
- result.markdown_content / html_content / json_content / text_content
- result.*_hosted_url (hosted assets), screenshot_hosted_url

Use cases:
- Page monitoring, product extraction, article ingestion

### 2) batchScrape
Scrape many URLs in parallel (up to 100k).

Parameters:
- apiKey (string, required)
- batch_array (array, required): [{ url, custom_id? }, ...]
- formats (array, default ['markdown'])
- country, wait_before_scraping, parser (optional)

Response highlights:
- object: 'batch'
- batch_id, status

Use cases:
- Catalog scraping, bulk enrichment, multi‑source ingestion

### 3) createCrawl
Autonomously discover and scrape pages by following links.

Parameters:
- apiKey (string, required)
- start_url (string, required)
- max_pages (number, default 10)
- follow_links (boolean, default true)
- formats (array, default ['markdown'])
- country, parser (optional)

Response highlights:
- object: 'crawl'
- id, status, created

Use cases:
- Docs/blog ingestion, knowledge base building, site archiving

### 4) createMap
Extract all URLs from a site for discovery and planning.

Parameters:
- apiKey (string, required)
- url (string, required)
- search_query (string, optional)
- top_n (number, optional)
- include_urls / exclude_urls (array/string, optional)

Response highlights:
- object: 'map'
- id, url, total_urls, urls[]

Use cases:
- Sitemap building, pre‑batch discovery, SEO site audits

## Quick examples

Call an API from Mastra:
```ts
const result = await mastra.callApi({
  integrationName: 'olostep',
  api: 'scrapeWebsite',
  payload: {
    data: {
      apiKey: process.env.OLOSTEP_API_KEY!,
      url_to_scrape: 'https://example.com',
      formats: ['markdown'],
      country: 'US',
    }
  }
});
```

Agent that can use Olostep:
```ts
import { Agent } from '@mastra/core';

const researchAgent = new Agent({
  name: 'research-assistant',
  instructions: `
    You can search, extract and structure web data with Olostep.
    Use scrapeWebsite for single pages, createMap to discover pages,
    and batchScrape for bulk extraction. Summarize findings.
  `,
  model: 'openai/gpt-4',
});
```

## Parsers (optional)

Use specialized parsers for popular sites to get structured fields automatically:
- @olostep/amazon-product
- @olostep/linkedin-profile
- @olostep/linkedin-company
- @olostep/google-search
- @olostep/google-maps
- @olostep/instagram-profile

Example:
```ts
await mastra.callApi({
  integrationName: 'olostep',
  api: 'scrapeWebsite',
  payload: { data: {
    apiKey: process.env.OLOSTEP_API_KEY!,
    url_to_scrape: 'https://www.amazon.com/dp/PRODUCT_ID',
    formats: ['json'],
    parser: '@olostep/amazon-product',
  }}
});
```

## Security
- API key via environment variables
- No secrets in code

## Support & Resources
- Docs: https://docs.olostep.com
- Dashboard (API key): https://olostep.com/dashboard
- Support: https://olostep.com/support