npm package: @gitmaxd/convex-firecrawl-agent
Description: A convex firecrawl agent component for Convex.
Latest version: 0.2.1

# Convex Firecrawl Agent

[![npm version](https://img.shields.io/npm/v/@gitmaxd/convex-firecrawl-agent)](https://www.npmjs.com/package/@gitmaxd/convex-firecrawl-agent)
[![npm downloads](https://img.shields.io/npm/dm/@gitmaxd/convex-firecrawl-agent)](https://www.npmjs.com/package/@gitmaxd/convex-firecrawl-agent)
[![CI](https://github.com/gitmaxd/convex-firecrawl-agent/actions/workflows/test.yml/badge.svg)](https://github.com/gitmaxd/convex-firecrawl-agent/actions/workflows/test.yml)
[![License: Apache-2.0](https://img.shields.io/github/license/gitmaxd/convex-firecrawl-agent)](./LICENSE)

Publishable Convex Component that wraps Firecrawl's async `/v2/agent` API as **durable, reactive job resources**.

<p align="center">
  <a href="./docs/assets/readme-screenshot.webp">
    <img
      src="./docs/assets/readme-screenshot.webp"
      alt="Screenshot of the Convex Firecrawl Agent demo UI"
      width="900"
    />
  </a>
</p>

Turn long-running agent workflows into something you can treat like any other Convex data model: create once, subscribe reactively, and render status/results without writing your own polling + retry + storage plumbing.

## What you get

- **Durable jobs:** Create once; observe status and results via reactive queries.
- **Webhook-first architecture:** Real-time status updates via Firecrawl webhooks, with bounded reconciliation fallback for reliability.
- **Retry with backoff:** Kickoff failures (429, 5xx, network errors) retry up to 5 times with exponential backoff.
- **Cancellation:** Cancel locally and (best-effort) remotely. If Firecrawl already completed, the job ends `completed` (completion wins).
- **Cleanup:** `deleteJob` removes terminal jobs (and their events, plus any stored result file) to keep UIs tidy and support retention policies.
- **Large result handling:** Results larger than **750 KiB** are stored in Convex file storage with an inline preview; use `getResultDownloadUrl` when `resultStorageId` is present.
- **Observability:** Optional per-job event timeline for debugging and UI timelines.

Found a bug or have a feature request? Please open an issue: https://github.com/gitmaxd/convex-firecrawl-agent/issues

## Model selection

The component supports Firecrawl's agent models via the optional `model` parameter:

| Model | Cost | Best for |
|-------|------|----------|
| `spark-1-mini` | **60% cheaper** | Most tasks (default) |
| `spark-1-pro` | Standard | Complex multi-domain research |

Start with the default (`spark-1-mini`) for cost efficiency. Use `spark-1-pro` when accuracy is critical or tasks require advanced reasoning across multiple sources.

See [Firecrawl model docs](https://docs.firecrawl.dev/features/models) for detailed comparison.

## Demo

See the [`example/`](./example/) directory for a complete working demo (Convex backend + Vite React UI) that demonstrates the full job lifecycle: create → observe → cancel → delete, with live event streaming and result viewing.

## Install

```sh
npm i @gitmaxd/convex-firecrawl-agent
```

## Prerequisites

- **Convex project:** If you don't have one yet, run `npx convex dev` to create a new project (you'll be prompted to log in or create a free account).
- **Firecrawl API key:** Get one at [firecrawl.dev](https://firecrawl.dev), then set it in your Convex deployment:
  ```sh
  npx convex env set FIRECRAWL_API_KEY "<your-key>"
  ```

## Add to your Convex app

```ts
// convex/convex.config.ts
import { defineApp } from "convex/server";
import convexFirecrawlAgent from "@gitmaxd/convex-firecrawl-agent/convex.config.js";

const app = defineApp();
app.use(convexFirecrawlAgent);

export default app;
```

## Usage

### Recommended pattern: expose an app-level API

Your Convex app should not call component functions directly from the client. Instead, define app functions that:

- authenticate the caller
- map identity → `userId`
- delegate to the component

Use `exposeApi` to generate app-level queries/mutations with your auth mapping:

```ts
// convex/firecrawlAgent.ts
import { components } from "./_generated/api";
import { exposeApi } from "@gitmaxd/convex-firecrawl-agent";

export const {
  createJob,
  getJob,
  listJobs,
  cancelJob,
  deleteJob,
  listJobEvents,
} = exposeApi(components.convexFirecrawlAgent, {
  auth: async (ctx) => {
    const userId = (await ctx.auth.getUserIdentity())?.subject;
    if (!userId) throw new Error("Unauthorized");
    return userId;
  },
  includeEvents: true,
  // Optional override for tests or custom limits:
  // resultStorageThresholdBytes: 750 * 1024,
});
```

### Job lifecycle (create → observe → cancel → cleanup)

From your client (via your app mutations/queries) or from other Convex functions:

```ts
// create
const jobId = await ctx.runMutation(api.firecrawlAgent.createJob, {
  targetId: "doc:123",
  prompt: "Extract a short summary",
  urls: ["https://news.convex.dev"],
});

// observe (reactive when used in a client query)
const job = await ctx.runQuery(api.firecrawlAgent.getJob, { jobId });

// cancel (optional)
await ctx.runMutation(api.firecrawlAgent.cancelJob, { jobId });

// cleanup (recommended for terminal jobs)
await ctx.runMutation(api.firecrawlAgent.deleteJob, { jobId });
```

## Guardrails & limits

To keep job documents durable and Firecrawl requests valid, the component enforces a few guardrails up front (the mutation throws and the job is **not** created if these fail):

- **`prompt` length:** maximum **10,000 characters**. If exceeded: `Prompt exceeds maximum length of 10000`.
- **`schema` must be JSON-serializable:** the schema is stored on the job and forwarded as JSON, so it must round-trip through `JSON.stringify(...)` (no `BigInt`, functions, or circular references). If invalid: `Schema must be JSON-serializable`.
- **Large-result storage threshold must be valid:** `resultStorageThresholdBytes` (and the `RESULT_STORAGE_THRESHOLD_BYTES` env override used by the helper wrappers) must be a **finite number ≥ 0**.
  - If you pass an invalid `resultStorageThresholdBytes`, the wrapper throws: `resultStorageThresholdBytes must be a finite number >= 0`.
  - If `RESULT_STORAGE_THRESHOLD_BYTES` is set but invalid, it is ignored (treated as unset).
  - `0` is valid and forces results to always be stored in file storage.
- **`maxCredits` must be valid:** when provided, must be a **finite integer > 0**. Invalid values (non-finite, non-integer, or ≤ 0) cause the mutation to throw and no job is created.
- **`model` selection:** optional `model` parameter accepts `"spark-1-mini"` or `"spark-1-pro"` to select the Firecrawl agent model.

## Type-safe integration (single source of truth)

```ts
import { exposeApi } from "@gitmaxd/convex-firecrawl-agent";
import { components } from "./_generated/api";

const agentApi = exposeApi(components.convexFirecrawlAgent, {
  auth: async (ctx, operation) => {
    const userId = (await ctx.auth.getUserIdentity())?.subject;
    if (!userId) throw new Error("Unauthorized");
    return userId;
  },
});

type AgentApi = typeof agentApi;
type AuthFn = Parameters<typeof exposeApi>[1]["auth"]; // (ctx, op) => Promise<string>
type CreateJobReturn = Awaited<ReturnType<AgentApi["createJob"]>>; // string jobId
```

## Large result handling

If a job result exceeds **750 KiB**, the component stores the full payload in **Convex file storage** and keeps an inline preview. Use `getResultDownloadUrl` to fetch a signed download URL when `resultStorageId` is present.

## Observability: job events

Enable `includeEvents: true` in `exposeApi` to record a per-job event timeline. The demo UI uses this to show state transitions and retry/backoff behavior.

## Troubleshooting

- **Missing `FIRECRAWL_API_KEY`:** set it with `npx convex env set FIRECRAWL_API_KEY "..."`.
- **Jobs stuck in `processing`:** check the job's event timeline and fields like `attempt`, `lastHttpStatus`, `firecrawlStatus`, and `error`; verify your API key is valid.

See the full troubleshooting guide: [`docs/10-troubleshooting.md`](./docs/10-troubleshooting.md).

## Documentation

- Start here: [`docs/02-getting-started.md`](./docs/02-getting-started.md)
- Full doc index: [`docs/README.md`](./docs/README.md)

## References

- Convex Components Authoring: https://docs.convex.dev/components/authoring
- Firecrawl Agent feature docs: https://docs.firecrawl.dev/features/agent

## License

Apache-2.0. See [`LICENSE`](./LICENSE).